Dell R310 with SAS drives very slow

[Tracy Di Marco White <netbsd-t%menelos.com@localhost>]

We have several new Dell R310 & one R410 with PERC 6/ir controllers
and SAS drives. Disk writes are incredibly slow running NetBSD, but
are fine under Ubuntu. We have the same machines, with the same
controllers, with SATA drives, and they're responding just fine. We've
used lsiutil on Ubuntu to enable the write cache, but if that's doing
anything at all, it doesn't stick when we boot into NetBSD.  We've
tested with Bonnie, using 16GB transfers.

Most of our NetBSD testing has been with i386 NetBSD 5.1_RC1, with "no
options ENHANCED_SPEEDSTEP" and "no options INTEL_CORETEMP" set, so
they boot. The disks are 2 146GB 10K (or 15K) RPM SAS drives
configured in a RAID1 array. The one configured with SATA disks is
160GB 7.2k RPM.

We expected disk write speeds to be critical on several of these,
which is why they are SAS. Ideas for improving disk access?

% cat r310.dmesg
Copyright (c) 1996, 1997, 1998, 1999, 2000, 2001, 2002, 2003, 2004, 2005,
    2006, 2007, 2008, 2009
    The NetBSD Foundation, Inc.  All rights reserved.
Copyright (c) 1982, 1986, 1989, 1991, 1993
    The Regents of the University of California.  All rights reserved.

NetBSD 5.1_RC1 (GENERIC) #0: Wed May 12 10:10:09 CDT 2010
        
root%six.its.iastate.edu@localhost:/usr/obj/5.1/usr/cvs/5.1RC1-i7/src/sys/arch/i386/compile/GENERIC
total memory = 3062 MB
avail memory = 2997 MB
timecounter: Timecounters tick every 10.000 msec
timecounter: Timecounter "i8254" frequency 1193182 Hz quality 100
Dell Inc. PowerEdge R310
mainbus0 (root)
cpu0 at mainbus0 apid 0: Intel 686-class, 2660MHz, id 0x106e5
cpu1 at mainbus0 apid 2: Intel 686-class, 2660MHz, id 0x106e5
cpu2 at mainbus0 apid 4: Intel 686-class, 2660MHz, id 0x106e5
cpu3 at mainbus0 apid 6: Intel 686-class, 2660MHz, id 0x106e5
cpu4 at mainbus0 apid 1: Intel 686-class, 2660MHz, id 0x106e5
cpu5 at mainbus0 apid 3: Intel 686-class, 2660MHz, id 0x106e5
cpu6 at mainbus0 apid 5: Intel 686-class, 2660MHz, id 0x106e5
cpu7 at mainbus0 apid 7: Intel 686-class, 2660MHz, id 0x106e5
ioapic0 at mainbus0 apid 0: pa 0xfec00000, version 20, 24 pins
acpi0 at mainbus0: Intel ACPICA 20080321
acpi0: X/RSDT: OemId <DELL  ,PE_SC3  ,00000001>, AslId <DELL,00000001>
acpi0: SCI interrupting at int 9
acpi0: fixed-feature power button present
timecounter: Timecounter "ACPI-Fast" frequency 3579545 Hz quality 1000
ACPI-Fast 24-bit timer
npx1 at acpi0 (FPU, PNP0C04): io 0xf0-0xff irq 13
npx1: reported by CPUID; using exception 16
attimer1 at acpi0 (TMR, PNP0100): io 0x40-0x5f irq 0
COMA (PNP0501) at acpi0 not configured
COMB (PNP0501) at acpi0 not configured
hpet0 at acpi0 (HPET, PNP0103-0): mem 0xfed00000-0xfed003ff
timecounter: Timecounter "hpet0" frequency 14318179 Hz quality 2000
apm0 at acpi0: Power Management spec V1.2
ipmi0 at mainbus0
pci0 at mainbus0 bus 0: configuration mode 1
pci0: i/o space, memory space enabled, rd/line, rd/mult, wr/inv ok
pchb0 at pci0 dev 0 function 0
pchb0: vendor 0x8086 product 0xd130 (rev. 0x11)
ppb0 at pci0 dev 3 function 0: vendor 0x8086 product 0xd138 (rev. 0x11)
ppb0: unsupported PCI Express version
pci1 at ppb0 bus 4
pci1: i/o space, memory space enabled, rd/line, wr/inv ok
ppb1 at pci0 dev 5 function 0: vendor 0x8086 product 0xd13a (rev. 0x11)
ppb1: unsupported PCI Express version
pci2 at ppb1 bus 5
pci2: i/o space, memory space enabled, rd/line, wr/inv ok
vendor 0x8086 product 0xd155 (miscellaneous system, revision 0x11) at
pci0 dev 8 function 0 not configured
vendor 0x8086 product 0xd156 (miscellaneous system, revision 0x11) at
pci0 dev 8 function 1 not configured
vendor 0x8086 product 0xd157 (miscellaneous system, revision 0x11) at
pci0 dev 8 function 2 not configured
vendor 0x8086 product 0xd158 (miscellaneous system, revision 0x11) at
pci0 dev 8 function 3 not configured
vendor 0x8086 product 0xd150 (miscellaneous system, revision 0x11) at
pci0 dev 16 function 0 not configured
vendor 0x8086 product 0xd151 (miscellaneous system, revision 0x11) at
pci0 dev 16 function 1 not configured
ehci0 at pci0 dev 26 function 0: vendor 0x8086 product 0x3b3c (rev. 0x05)
ehci0: interrupting at ioapic0 pin 22
ehci0: BIOS has given up ownership
ehci0: EHCI version 1.0
usb0 at ehci0: USB revision 2.0
ppb2 at pci0 dev 28 function 0: vendor 0x8086 product 0x3b42 (rev. 0x05)
ppb2: unsupported PCI Express version
pci3 at ppb2 bus 3
pci3: i/o space, memory space enabled, rd/line, wr/inv ok
mpt0 at pci3 dev 0 function 0: vendor 0x1000 product 0x0058
mpt0: interrupting at ioapic0 pin 16
mpt0: Phy 0: Link Rate 3.0 Gbps
mpt0: Phy 1: Link Rate 3.0 Gbps
mpt0: Unknown async event: 0x21
mpt0: Unknown async event: 0x21
mpt0: Unknown async event: 0xb
scsibus0 at mpt0: 112 targets, 8 luns per target
ppb3 at pci0 dev 28 function 4: vendor 0x8086 product 0x3b4a (rev. 0x05)
ppb3: unsupported PCI Express version
pci4 at ppb3 bus 2
pci4: i/o space, memory space enabled, rd/line, wr/inv ok
bnx0 at pci4 dev 0 function 0: Broadcom NetXtreme II BCM5716 1000Base-T
bnx0: Ethernet address a4:ba:db:35:95:6b
bnx0: interrupting at ioapic0 pin 16
brgphy0 at bnx0 phy 1: BCM5709 10/100/1000baseT PHY, rev. 8
brgphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT,
1000baseT-FDX, auto
bnx1 at pci4 dev 0 function 1: Broadcom NetXtreme II BCM5716 1000Base-T
bnx1: Ethernet address a4:ba:db:35:95:6c
bnx1: interrupting at ioapic0 pin 17
brgphy1 at bnx1 phy 1: BCM5709 10/100/1000baseT PHY, rev. 8
brgphy1: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT,
1000baseT-FDX, auto
ehci1 at pci0 dev 29 function 0: vendor 0x8086 product 0x3b34 (rev. 0x05)
ehci1: interrupting at ioapic0 pin 22
ehci1: EHCI version 1.0
usb1 at ehci1: USB revision 2.0
ppb4 at pci0 dev 30 function 0: vendor 0x8086 product 0x244e (rev. 0xa5)
pci5 at ppb4 bus 1
pci5: i/o space, memory space enabled
vga1 at pci5 dev 3 function 0: vendor 0x102b product 0x0532 (rev. 0x0a)
wsdisplay0 at vga1 kbdmux 1: console (80x25, vt100 emulation)
wsmux1: connecting to wsdisplay0
drm at vga1 not configured
pcib0 at pci0 dev 31 function 0
pcib0: vendor 0x8086 product 0x3b14 (rev. 0x05)
piixide0 at pci0 dev 31 function 2
piixide0: Intel 3400 Serial ATA Controller (rev. 0x05)
piixide0: bus-master DMA support present
piixide0: primary channel configured to native-PCI mode
piixide0: using ioapic0 pin 20 for native-PCI interrupt
atabus0 at piixide0 channel 0
piixide0: secondary channel configured to native-PCI mode
atabus1 at piixide0 channel 1
piixide1 at pci0 dev 31 function 5
piixide1: Intel 3400 Serial ATA Controller (rev. 0x05)
piixide1: bus-master DMA support present
piixide1: primary channel wired to native-PCI mode
piixide1: using ioapic0 pin 21 for native-PCI interrupt
atabus2 at piixide1 channel 0
piixide1: secondary channel wired to native-PCI mode
atabus3 at piixide1 channel 1
isa0 at pcib0
com0 at isa0 port 0x3f8-0x3ff irq 4: ns16550a, working fifo
com1 at isa0 port 0x2f8-0x2ff irq 3: ns16550a, working fifo
pckbc0 at isa0 port 0x60-0x64
pckbc: cmd word write error
pcppi0 at isa0 port 0x61
midi0 at pcppi0: PC speaker (CPU-intensive output)
sysbeep0 at pcppi0
attimer1: attached to pcppi0
timecounter: Timecounter "clockinterrupt" frequency 100 Hz quality 0
timecounter: Timecounter "TSC" frequency 2660103020 Hz quality 3000
scsibus0: waiting 2 seconds for devices to settle...
uhub0 at usb0: vendor 0x8086 EHCI root hub, class 9/0, rev 2.00/1.00, addr 1
uhub0: 2 ports with 2 removable, self powered
uhub1 at usb1: vendor 0x8086 EHCI root hub, class 9/0, rev 2.00/1.00, addr 1
uhub1: 2 ports with 2 removable, self powered
uhub2 at uhub1 port 1: vendor 0x8087 product 0x0020, class 9/0, rev
2.00/0.00, addr 2
uhub2: single transaction translator
uhub3 at uhub0 port 1: vendor 0x8087 product 0x0020, class 9/0, rev
2.00/0.00, addr 2
uhub3: single transaction translator
uhub2: 8 ports with 8 removable, self powered
uhub3: 6 ports with 6 removable, self powered
sd0 at scsibus0 target 0 lun 0: <Dell, VIRTUAL DISK, 1028> disk fixed
sd0: 136 GB, 139392 cyl, 16 head, 128 sec, 512 bytes/sect x 285474816 sectors
probe(mpt0:0:8:0): generic HBA error
uhub4 at uhub3 port 1: vendor 0x0424 product 0x2514, class 9/0, rev
2.00/0.00, addr 3
uhub4: multiple transaction translators
uhub4: 4 ports with 4 removable, self powered
uhidev0 at uhub4 port 1 configuration 1 interface 0
uhidev0: Dell Dell USB Keyboard, rev 1.10/3.01, addr 4, iclass 3/1
ukbd0 at uhidev0
wskbd0 at ukbd0 mux 1
wskbd0: connecting to wsdisplay0
uhub5 at uhub4 port 2: vendor 0x13fe USB 2.0 HUB, class 9/0, rev
2.00/7.02, addr 5
uhub5: single transaction translator
uhub5: 2 ports with 0 removable, self powered
umass0 at uhub5 port 1 configuration 1 interface 0
umass0: Aladdin NG Flash 4.27, rev 2.00/1.10, addr 6
umass0: using SCSI over Bulk-Only
scsibus1 at umass0: 2 targets, 1 lun per target
sd1 at scsibus1 target 0 lun 0: <Aladdin, NG Flash 4.27, PMAP> disk removable
sd1: 492 MB, 1968 cyl, 16 head, 32 sec, 512 bytes/sect x 1007616 sectors
ugen0 at uhub5 port 2
ugen0: Aladdin Knowledge Systems Ltd. Token 4.27.1.1 2.6.189, rev
1.10/1.00, addr 7
atapibus0 at atabus2: 2 targets
cd0 at atapibus0 drive 0: <TEAC DVD-ROM DV-28SW, 10040823233804, R.2A>
cdrom removable
cd0: 32-bit data port
cd0: drive supports PIO mode 4, DMA mode 2, Ultra-DMA mode 5 (Ultra/100)
cd0(piixide1:0:0): using PIO mode 4, Ultra-DMA mode 5 (Ultra/100) (using DMA)
ipmi0: version 2.0 interface KCS iobase 0xca8/8 spacing 4
Kernelized RAIDframe activated
pad0: outputs: 44100Hz, 16-bit, stereo
audio0 at pad0: half duplex, playback, capture
boot device: sd0
root on sd0a dumps on sd0b
root file system type: ffs
wsdisplay0: screen 1 added (80x25, vt100 emulation)
wsdisplay0: screen 2 added (80x25, vt100 emulation)
wsdisplay0: screen 3 added (80x25, vt100 emulation)
wsdisplay0: screen 4 added (80x25, vt100 emulation)