Subject: 3ware RAID controller slow?
To: NetBSD Users <netbsd-users@netbsd.org>
From: Jukka Marin <jmarin@embedtronics.fi>
List: netbsd-users
Date: 10/27/2007 16:56:08
--ikeVEW9yuYc//A+q
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline

Hi,

I'm testing this new system.  Extracting pkgsrc.tgz to a RAID5 disk:

tar: ustar vol 1, 111398 files, 261967650 bytes read, 0 bytes written in 378
secs (693036 bytes/sec)

The same operation on a single disk (connected to the same RAID controller):

tar: ustar vol 1, 111398 files, 261967650 bytes read, 0 bytes written in 23
secs (11389897 bytes/sec)

Both partitions are mounted using softdeps, both tests performed under the
same conditions and both results are repeatable (with some variation in
time).

Question: Why is RAID5 this slow?  Is it a NetBSD problem (in the twa driver,
for example) or is this how RAID5 performs?

I'm also having problems with NFS, but I don't know whether the problem is
the new system (with wm Ethernet) or the server (with nfe).  Both machines
are running 4.0RC3 SMP kernels.

dmesg output attached.

  -jm


--ikeVEW9yuYc//A+q
Content-Type: text/plain; charset=us-ascii
Content-Disposition: attachment; filename="dmesg.txt"

Copyright (c) 1996, 1997, 1998, 1999, 2000, 2001, 2002, 2003, 2004, 2005,
    2006, 2007
    The NetBSD Foundation, Inc.  All rights reserved.
Copyright (c) 1982, 1986, 1989, 1991, 1993
    The Regents of the University of California.  All rights reserved.

NetBSD 4.0_RC3 (GENERIC.MP) #9: Fri Oct 26 20:49:18 EEST 2007
	root@pyy:/tmp/nb4/obj/sys/arch/i386/compile/GENERIC.MP
total memory = 3582 MB
rbus: rbus_min_start set to 0xc0000000
avail memory = 3508 MB
timecounter: Timecounters tick every 10.000 msec
timecounter: Timecounter "i8254" frequency 1193182 Hz quality 100
BIOS32 rev. 0 found at 0xfd450
mainbus0 (root)
cpu0 at mainbus0: apid 0 (boot processor)
cpu0: Intel Core 2 (Merom) (686-class), 2394.14 MHz, id 0x6fb
cpu0: features bfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR>
cpu0: features bfebfbff<PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX>
cpu0: features bfebfbff<FXSR,SSE,SSE2,SS,HTT,TM,SBF>
cpu0: features2 e3bd<SSE3,MONITOR,DS-CPL,VMX,EST,TM2,xTPR>
cpu0: "Intel(R) Core(TM)2 Quad CPU    Q6600  @ 2.40GHz"
cpu0: I-cache 32 KB 64B/line 8-way, D-cache 32 KB 64B/line 8-way
cpu0: L2 cache 4 MB 64B/line 16-way
cpu0: using thermal monitor 1
cpu0: Enhanced SpeedStep disabled by BIOS
cpu0: calibrating local timer
cpu0: apic clock running at 265 MHz
cpu0: 64 page colors
cpu1 at mainbus0: apid 1 (application processor)
cpu1: starting
cpu1: Intel Core 2 (Merom) (686-class), 2394.00 MHz, id 0x6fb
cpu1: features bfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR>
cpu1: features bfebfbff<PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX>
cpu1: features bfebfbff<FXSR,SSE,SSE2,SS,HTT,TM,SBF>
cpu1: features2 e3bd<SSE3,MONITOR,DS-CPL,VMX,EST,TM2,xTPR>
cpu1: "Intel(R) Core(TM)2 Quad CPU    Q6600  @ 2.40GHz"
cpu1: I-cache 32 KB 64B/line 8-way, D-cache 32 KB 64B/line 8-way
cpu1: L2 cache 4 MB 64B/line 16-way
cpu1: using thermal monitor 1
cpu1: Enhanced SpeedStep disabled by BIOS
cpu2 at mainbus0: apid 2 (application processor)
cpu2: starting
cpu2: Intel Core 2 (Merom) (686-class), 2394.01 MHz, id 0x6fb
cpu2: features bfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR>
cpu2: features bfebfbff<PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX>
cpu2: features bfebfbff<FXSR,SSE,SSE2,SS,HTT,TM,SBF>
cpu2: features2 e3bd<SSE3,MONITOR,DS-CPL,VMX,EST,TM2,xTPR>
cpu2: "Intel(R) Core(TM)2 Quad CPU    Q6600  @ 2.40GHz"
cpu2: I-cache 32 KB 64B/line 8-way, D-cache 32 KB 64B/line 8-way
cpu2: L2 cache 4 MB 64B/line 16-way
cpu2: using thermal monitor 1
cpu2: Enhanced SpeedStep disabled by BIOS
cpu3 at mainbus0: apid 3 (application processor)
cpu3: starting
cpu3: Intel Core 2 (Merom) (686-class), 2394.00 MHz, id 0x6fb
cpu3: features bfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR>
cpu3: features bfebfbff<PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX>
cpu3: features bfebfbff<FXSR,SSE,SSE2,SS,HTT,TM,SBF>
cpu3: features2 e3bd<SSE3,MONITOR,DS-CPL,VMX,EST,TM2,xTPR>
cpu3: "Intel(R) Core(TM)2 Quad CPU    Q6600  @ 2.40GHz"
cpu3: I-cache 32 KB 64B/line 8-way, D-cache 32 KB 64B/line 8-way
cpu3: L2 cache 4 MB 64B/line 16-way
cpu3: using thermal monitor 1
cpu3: Enhanced SpeedStep disabled by BIOS
ioapic0 at mainbus0 apid 4 (I/O APIC)
ioapic0: pa 0xfec00000, version 20, 24 pins
ioapic1 at mainbus0 apid 5 (I/O APIC)
ioapic1: pa 0xfecc0000, version 20, 24 pins
ioapic2 at mainbus0 apid 6 (I/O APIC)
ioapic2: pa 0xfecc0400, version 20, 24 pins
acpi0 at mainbus0: Advanced Configuration and Power Interface
acpi0: using Intel ACPI CA subsystem version 20060217
acpi0: X/RSDT: OemId <PTLTD ,  RSDT  ,06040000>, AslId < LTP,00000000>
acpi0: SCI interrupting at int 9
acpi0: fixed-feature power button present
timecounter: Timecounter "ACPI-Fast" frequency 3579545 Hz quality 1000
ACPI-Fast 24-bit timer
ACPI Object Type 'Processor' (0x0c) at acpi0 not configured
ACPI Object Type 'Processor' (0x0c) at acpi0 not configured
ACPI Object Type 'Processor' (0x0c) at acpi0 not configured
ACPI Object Type 'Processor' (0x0c) at acpi0 not configured
ACPI Object Type 'Processor' (0x0c) at acpi0 not configured
ACPI Object Type 'Processor' (0x0c) at acpi0 not configured
ACPI Object Type 'Processor' (0x0c) at acpi0 not configured
ACPI Object Type 'Processor' (0x0c) at acpi0 not configured
PNP0A03 [PCI/PCI-X Host Bridge] at acpi0 not configured
PNP0C02 [Plug and Play motherboard register resources] at acpi0 not configured
PNP0200 [AT DMA Controller] at acpi0 not configured
npx1 at acpi0 (PNP0C04)
npx1: io 0xf0-0xfe irq 13
npx1: reported by CPUID; using exception 16
PNP0000 [AT Interrupt Controller] at acpi0 not configured
PNP0B00 [AT Real-Time Clock] at acpi0 not configured
pcppi1 at acpi0 (PNP0800)
pcppi1: io 0x61
pcppi1: children must have an explicit unit
midi0 at pcppi1: PC speaker (CPU-intensive output)
sysbeep0 at pcppi1
attimer1 at acpi0 (PNP0100): AT Timer
attimer1: io 0x40-0x43,0x50-0x53
PNP0103 [HPET Timer] at acpi0 not configured
PNP0C0F [PCI interrupt link device] at acpi0 not configured
PNP0C0F [PCI interrupt link device] at acpi0 not configured
PNP0C0F [PCI interrupt link device] at acpi0 not configured
PNP0C0F [PCI interrupt link device] at acpi0 not configured
PNP0C0F [PCI interrupt link device] at acpi0 not configured
INT0800 [Intel FWH Random Number Generator] at acpi0 not configured
PNP0A05 [Generic Container Device] at acpi0 not configured
pckbc1 at acpi0 (PNP0303): kbd port
pckbc1: io 0x60,0x64 irq 1
pckbc2 at acpi0 (PNP0F13): aux port
pckbc2: irq 12
com3 at acpi0 (PNP0501-1)
com3: io 0x3f8-0x3ff irq 4
com3: ns16550a, working fifo
com4 at acpi0 (PNP0501-2)
com4: io 0x2f8-0x2ff irq 3
com4: ns16550a, working fifo
PNP0700 [PC standard floppy disk controller] at acpi0 not configured
acpibut0 at acpi0 (PNP0C0C): ACPI Power Button
pcppi1: attached to attimer1
pckbd0 at pckbc1 (kbd slot)
pckbc1: using irq 1 for kbd slot
wskbd0 at pckbd0: console keyboard
pms0 at pckbc1 (aux slot)
pckbc1: using irq 12 for aux slot
wsmouse0 at pms0 mux 0
pci0 at mainbus0 bus 0: configuration mode 1
pci0: i/o space, memory space enabled, rd/line, rd/mult, wr/inv ok
pchb0 at pci0 dev 0 function 0
pchb0: Intel product 0x2778 (rev. 0xc0)
ppb0 at pci0 dev 1 function 0: Intel product 0x2779 (rev. 0xc0)
pci1 at ppb0 bus 1
pci1: i/o space, memory space enabled, rd/line, wr/inv ok
ppb1 at pci1 dev 0 function 0: Intel 6700PXH PCI Express-to-PCI Bridge #0 (rev. 0x09)
pci2 at ppb1 bus 2
pci2: i/o space, memory space enabled, rd/line, wr/inv ok
Intel product 0x0326 (interrupt system, interface 0x20, revision 0x09) at pci1 dev 0 function 1 not configured
ppb2 at pci1 dev 0 function 2: Intel 6700PXH PCI Express-to-PCI Bridge #1 (rev. 0x09)
pci3 at ppb2 bus 3
pci3: i/o space, memory space enabled, rd/line, wr/inv ok
3ware 9550SX series: (rev. 0x00)
twa0 at pci3 dev 3 function 0: 3ware Apache
twa0: interrupting at ioapic2 pin 4 (irq 7)
twa0: AEN 0x003C: INFO: Background initialize paused: unit=1
twa0: AEN 0x0053: INFO: Need to do a capacity test: 
twa0: 8 ports, Firmware FE9X 3.04.00.005, BIOS BE9X 3.04.00.002
twa0: Monitor BL9X 3.02.00.001, PCB Rev 032 , Achip 1.90    , Pchip 1.60    
twa0: port 0: WDC WD10EACS-00ZJB0                      953869 MB
twa0: port 1: WDC WD10EACS-00ZJB0                      953869 MB
twa0: port 2: WDC WD10EACS-00ZJB0                      953869 MB
twa0: port 3: WDC WD10EACS-00ZJB0                      953869 MB
twa0: port 4: WDC WD10EACS-00ZJB0                      953869 MB
twa0: port 5: WDC WD10EACS-00ZJB0                      953869 MB
twa0: port 7: SAMSUNG HD501LJ                          476940 MB
twa0: AMCC    9550SXU-8L DISK 3.04MP81912022FA9E000675
ld0 at twa0 unit 0
ld0: 465 GB, 60786 cyl, 255 head, 63 sec, 512 bytes/sect x 976541696 sectors
twa0: AMCC    9550SXU-8L DISK 3.04J015880522FA9E001892
ld1 at twa0 unit 1
ld1: 4656 GB, 607875 cyl, 255 head, 63 sec, 512 bytes/sect x 9765519360 sectors
Intel product 0x0327 (interrupt system, interface 0x20, revision 0x09) at pci1 dev 0 function 3 not configured
ppb3 at pci0 dev 28 function 0: Intel 82801GB/GR PCI Express Port #1 (rev. 0x01)
pci4 at ppb3 bus 14
pci4: no spaces enabled!
ppb4 at pci0 dev 28 function 4: Intel 82801GB/GR PCI Express Port #5 (rev. 0x01)
pci5 at ppb4 bus 18
pci5: i/o space, memory space enabled, rd/line, wr/inv ok
wm0 at pci5 dev 0 function 0: Intel i82573E IAMT, rev. 3
wm0: interrupting at ioapic0 pin 16 (irq 7)
wm0: PCI-Express bus
wm0: 256 word (8 address bits) SPI EEPROM
wm0: Ethernet address 00:30:48:92:2b:f6
makphy0 at wm0 phy 1: Marvell 88E1111 Gigabit PHY, rev. 2
makphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-FDX, auto
ppb5 at pci0 dev 28 function 5: Intel 82801GB/GR PCI Express Port #6 (rev. 0x01)
pci6 at ppb5 bus 19
pci6: i/o space, memory space enabled, rd/line, wr/inv ok
wm1 at pci6 dev 0 function 0: Intel i82573L Gigabit Ethernet, rev. 0
wm1: interrupting at ioapic0 pin 17 (irq 11)
wm1: PCI-Express bus
wm1: 256 word (8 address bits) SPI EEPROM
wm1: Ethernet address 00:30:48:92:2b:f7
makphy1 at wm1 phy 1: Marvell 88E1111 Gigabit PHY, rev. 2
makphy1: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-FDX, auto
uhci0 at pci0 dev 29 function 0: Intel 82801GB/GR USB UHCI Controller (rev. 0x01)
uhci0: interrupting at ioapic0 pin 23 (irq 10)
usb0 at uhci0: USB revision 1.0
uhub0 at usb0
uhub0: Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1
uhub0: 2 ports with 2 removable, self powered
uhci1 at pci0 dev 29 function 1: Intel 82801GB/GR USB UHCI Controller (rev. 0x01)
uhci1: interrupting at ioapic0 pin 19 (irq 11)
usb1 at uhci1: USB revision 1.0
uhub1 at usb1
uhub1: Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1
uhub1: 2 ports with 2 removable, self powered
uhci2 at pci0 dev 29 function 2: Intel 82801GB/GR USB UHCI Controller (rev. 0x01)
uhci2: interrupting at ioapic0 pin 18 (irq 5)
usb2 at uhci2: USB revision 1.0
uhub2 at usb2
uhub2: Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1
uhub2: 2 ports with 2 removable, self powered
uhci3 at pci0 dev 29 function 3: Intel 82801GB/GR USB UHCI Controller (rev. 0x01)
uhci3: interrupting at ioapic0 pin 16 (irq 7)
usb3 at uhci3: USB revision 1.0
uhub3 at usb3
uhub3: Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1
uhub3: 2 ports with 2 removable, self powered
ehci0 at pci0 dev 29 function 7: Intel 82801GB/GR USB EHCI Controller (rev. 0x01)
ehci0: interrupting at ioapic0 pin 23 (irq 10)
ehci0: BIOS has given up ownership
ehci0: EHCI version 1.0
ehci0: companion controllers, 2 ports each: uhci0 uhci1 uhci2 uhci3
usb4 at ehci0: USB revision 2.0
uhub4 at usb4
uhub4: Intel EHCI root hub, class 9/0, rev 2.00/1.00, addr 1
uhub4: 8 ports with 8 removable, self powered
ppb6 at pci0 dev 30 function 0: Intel 82801BA Hub-PCI Bridge (rev. 0xe1)
pci7 at ppb6 bus 20
pci7: i/o space, memory space enabled
vga1 at pci7 dev 4 function 0: ATI Technologies product 0x515e (rev. 0x02)
wsdisplay0 at vga1 kbdmux 1: console (80x25, vt100 emulation), using wskbd0
wsmux1: connecting to wsdisplay0
pcib0 at pci0 dev 31 function 0
pcib0: Intel 82801GB/GR LPC Interface Bridge (rev. 0x01)
piixide0 at pci0 dev 31 function 1
piixide0: Intel 82801GB/GR IDE Controller (ICH7) (rev. 0x01)
piixide0: bus-master DMA support present
piixide0: primary channel configured to compatibility mode
piixide0: primary channel interrupting at ioapic0 pin 14 (irq 14)
atabus0 at piixide0 channel 0
piixide0: secondary channel configured to compatibility mode
piixide0: secondary channel interrupting at ioapic0 pin 15 (irq 15)
atabus1 at piixide0 channel 1
ahcisata0 at pci0 dev 31 function 2: Intel 82801GB/GR AHCI SATA Controller
ahcisata0: interrupting at ioapic0 pin 19 (irq 11)
ahcisata0: AHCI revision 1.1, 4 ports, 32 command slots, features 0xc720e000
atabus2 at ahcisata0 channel 0
atabus3 at ahcisata0 channel 1
atabus4 at ahcisata0 channel 2
atabus5 at ahcisata0 channel 3
Intel 82801GB/GR SMBus Controller (SMBus serial bus, revision 0x01) at pci0 dev 31 function 3 not configured
isa0 at pcib0
isapnp0 at isa0 port 0x279: ISA Plug 'n Play device support
fdc0 at isa0 port 0x3f0-0x3f7 irq 6 drq 2
isapnp0: no ISA Plug 'n Play devices found
ioapic0: enabling
ioapic1: enabling
ioapic2: enabling
timecounter: Timecounter "clockinterrupt" frequency 100 Hz quality 0
fd0 at fdc0 drive 0: 1.44MB, 80 cyl, 2 head, 18 sec
Kernelized RAIDframe activated
atapibus0 at atabus0: 2 targets
cd0 at atapibus0 drive 0: <TSSTcorpDVD-ROM SH-D162D, , SB00> cdrom removable
cd0: 32-bit data port
cd0: drive supports PIO mode 4, DMA mode 2, Ultra-DMA mode 2 (Ultra/33)
cd0(piixide0:0:0): using PIO mode 4, Ultra-DMA mode 2 (Ultra/33) (using DMA)
umass0 at uhub4 port 7 configuration 1 interface 0
umass0: Peppercon AG Multidevice, rev 2.00/0.01, addr 2
umass0: using SCSI over Bulk-Only
scsibus0 at umass0: 2 targets, 1 lun per target
uhidev0 at uhub4 port 7 configuration 1 interface 1sd0 at scsibus0 target 0 lun 0: <PepperC, Virtual Disc 1, 0.01> disk removable

uhidev0: Peppercon AG Multidevice, rev 2.00/0.01, addr 2, iclass 3/0
ums0 at uhidev0
ums0: X report 0x0022 not supported
uhidev1 at uhub4 port 7 configuration 1 interface 2sd0: drive offline

uhidev1: Peppercon AG Multidevice, rev 2.00/0.01, addr 2, iclass 3/1
ukbd0 at uhidev1
wskbd1 at ukbd0 mux 1
wskbd1: connecting to wsdisplay0
sd0(umass0:0:0:0):  Check Condition on CDB: 0x00 00 00 00 00 00
    SENSE KEY:  Not Ready
     ASC/ASCQ:  Medium Not Present

sd0(umass0:0:0:0):  Check Condition on CDB: 0x00 00 00 00 00 00
    SENSE KEY:  Not Ready
     ASC/ASCQ:  Medium Not Present

sd0(umass0:0:0:0):  Check Condition on CDB: 0x00 00 00 00 00 00
    SENSE KEY:  Not Ready
     ASC/ASCQ:  Medium Not Present

sd0(umass0:0:0:0):  Check Condition on CDB: 0x00 00 00 00 00 00
    SENSE KEY:  Not Ready
     ASC/ASCQ:  Medium Not Present

boot device: ld0
root on ld0a dumps on ld0b
root file system type: ffs
cpu1: CPU 1 running
cpu2: CPU 2 running
cpu3: CPU 3 running
wsdisplay0: screen 1 added (80x25, vt100 emulation)
wsdisplay0: screen 2 added (80x25, vt100 emulation)
wsdisplay0: screen 3 added (80x25, vt100 emulation)
wsdisplay0: screen 4 added (80x25, vt100 emulation)
twa0: AEN 0x000C: INFO: Background initialize started: unit=1

--ikeVEW9yuYc//A+q--