Re: Dell R310 with SAS drives very slow

To: port-amd64%netbsd.org@localhost
Subject: Re: Dell R310 with SAS drives very slow
From: Tracy Di Marco White <netbsd-t%menelos.com@localhost>
Date: Wed, 26 May 2010 16:58:39 -0500
On Tue, May 25, 2010 at 6:30 AM, Edgar Fuß <ef%math.uni-bonn.de@localhost> 
wrote:
> EF> Looks like your controller fails to negotiate sync/wide transfers.
> MB> Which is not surprising for a RAID virtual disk.
> Ouch, I overlooked it ws a RAID.
>
> I'm unsure about the "probe(mpt0:0:8:0): generic HBA error", though.
>
> Moreover, what does "terribly slow" mean in terms of dd throughput?

Executive summary:
                RHEL     Ubuntu    NetBSD
       Read     145M/s   180M/s    140M/s
      Write     136M/s   157M/s     27M/s  <==OUCH

All tests run using a 16GB data set on a
dell r410 with 2 6-core Xeons, 6GB memory,
and 2 146GB 15k SAS drives in a Raid-1 config.


========================= RHEL 5 ==============================

Version  1.03       ------Sequential Output------ --Sequential Input- --Random-
                   -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
rhel5           16G           136498  18 67161  10           145423  11 652.5  1

time dd if=/dev/zero of=/tmp/16g bs=65536 count=262144
262144+0 records in
262144+0 records out
17179869184 bytes (17 GB) copied, 125.476 seconds, 137 MB/s


========================= Ubuntu 9.0.4 ==============================

Version 1.03c       ------Sequential Output------ --Sequential Input- --Random-
                   -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
ubuntu          16G           156777  12 72482   8           179692  11 578.7  0


time dd if=/dev/zero of=/tmp/16g bs=65536 count=262144
262144+0 records in
262144+0 records out
17179869184 bytes (17 GB) copied, 110.402 s, 156 MB/s

========================= NetBSD 5.1rc1 ==============================
Version 1.03e       ------Sequential Output------ --Sequential Input- --Random-
                   -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
netbsd          16G           26918   6 13233   3           140123  25 275.3   0

time dd if=/dev/zero of=/tmp/16g bs=65536 count=262144
262144+0 records in
262144+0 records out
17179869184 bytes transferred in 687.275 secs (24997081 bytes/sec)
0.233u 28.387s 11:27.27 4.1%    0+0k 18+835io 0pf+0w

This particular machine's dmesg:
% cat new-asw-1.dmesg
Copyright (c) 1996, 1997, 1998, 1999, 2000, 2001, 2002, 2003, 2004, 2005,
    2006, 2007, 2008, 2009
    The NetBSD Foundation, Inc.  All rights reserved.
Copyright (c) 1982, 1986, 1989, 1991, 1993
    The Regents of the University of California.  All rights reserved.

NetBSD 5.1_RC1 (GENERIC) #0: Wed May 12 10:10:09 CDT 2010
        
root%six.its.iastate.edu@localhost:/usr/obj/5.1/usr/cvs/5.1RC1-i7/src/sys/arch/i386/compile/GENERIC
total memory = 3318 MB
avail memory = 3248 MB
timecounter: Timecounters tick every 10.000 msec
timecounter: Timecounter "i8254" frequency 1193182 Hz quality 100
Dell Inc. PowerEdge R410
mainbus0 (root)
cpu0 at mainbus0 apid
32/usr/cvs/5.1RC1-i7/src/sys/arch/x86/x86/mtrr_i686.c: FIXME: more
than 8 MTRRs
: Intel 686-class, 2793MHz, id 0x206c2
cpu1 at mainbus0 apid 0: Intel 686-class, 2793MHz, id 0x206c2
cpu2 at mainbus0 apid 34: Intel 686-class, 2793MHz, id 0x206c2
cpu3 at mainbus0 apid 2: Intel 686-class, 2793MHz, id 0x206c2
cpu4 at mainbus0 apid 36: Intel 686-class, 2793MHz, id 0x206c2
cpu5 at mainbus0 apid 4: Intel 686-class, 2793MHz, id 0x206c2
cpu6 at mainbus0 apid 48: Intel 686-class, 2793MHz, id 0x206c2
cpu7 at mainbus0 apid 16: Intel 686-class, 2793MHz, id 0x206c2
cpu8 at mainbus0 apid 50: Intel 686-class, 2793MHz, id 0x206c2
cpu9 at mainbus0 apid 18: Intel 686-class, 2793MHz, id 0x206c2
cpu10 at mainbus0 apid 52: Intel 686-class, 2793MHz, id 0x206c2
cpu11 at mainbus0 apid 20: Intel 686-class, 2793MHz, id 0x206c2
cpu12 at mainbus0 apid 33: Intel 686-class, 2793MHz, id 0x206c2
cpu13 at mainbus0 apid 1: Intel 686-class, 2793MHz, id 0x206c2
cpu14 at mainbus0 apid 35: Intel 686-class, 2793MHz, id 0x206c2
cpu15 at mainbus0 apid 3: Intel 686-class, 2793MHz, id 0x206c2
cpu16 at mainbus0 apid 37: Intel 686-class, 2793MHz, id 0x206c2
cpu17 at mainbus0 apid 5: Intel 686-class, 2793MHz, id 0x206c2
cpu18 at mainbus0 apid 49: Intel 686-class, 2793MHz, id 0x206c2
cpu19 at mainbus0 apid 17: Intel 686-class, 2793MHz, id 0x206c2
cpu20 at mainbus0 apid 51: Intel 686-class, 2793MHz, id 0x206c2
cpu21 at mainbus0 apid 19: Intel 686-class, 2793MHz, id 0x206c2
cpu22 at mainbus0 apid 53: Intel 686-class, 2793MHz, id 0x206c2
cpu23 at mainbus0 apid 21: Intel 686-class, 2793MHz, id 0x206c2
ioapic0 at mainbus0 apid 0: pa 0xfec00000, version 20, 24 pins
ioapic1 at mainbus0 apid 1: pa 0xfec80000, version 20, 24 pins
acpi0 at mainbus0: Intel ACPICA 20080321
acpi0: X/RSDT: OemId <DELL  ,PE_SC3  ,00000001>, AslId <DELL,00000001>
acpi0: SCI interrupting at int 9
acpi0: fixed-feature power button present
timecounter: Timecounter "ACPI-Fast" frequency 3579545 Hz quality 1000
ACPI-Fast 24-bit timer
npx1 at acpi0 (FPU, PNP0C04): io 0xf0-0xff irq 13
npx1: reported by CPUID; using exception 16
attimer1 at acpi0 (TMR, PNP0100): io 0x40-0x5f irq 0
COMA (PNP0501) at acpi0 not configured
COMB (PNP0501) at acpi0 not configured
hpet0 at acpi0 (HPET, PNP0103-0): mem 0xfed00000-0xfed003ff
timecounter: Timecounter "hpet0" frequency 14318179 Hz quality 2000
apm0 at acpi0: Power Management spec V1.2
ipmi0 at mainbus0
pci0 at mainbus0 bus 0: configuration mode 1
pci0: i/o space, memory space enabled, rd/line, rd/mult, wr/inv ok
pchb0 at pci0 dev 0 function 0
pchb0: vendor 0x8086 product 0x3403 (rev. 0x13)
ppb0 at pci0 dev 1 function 0: vendor 0x8086 product 0x3408 (rev. 0x13)
ppb0: unsupported PCI Express version
pci1 at ppb0 bus 1
pci1: i/o space, memory space enabled, rd/line, wr/inv ok
bnx0 at pci1 dev 0 function 0: Broadcom NetXtreme II BCM5716 1000Base-T
bnx0: Ethernet address a4:ba:db:39:03:ec
bnx0: interrupting at ioapic1 pin 4
brgphy0 at bnx0 phy 1: BCM5709 10/100/1000baseT PHY, rev. 8
brgphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT,
1000baseT-FDX, auto
bnx1 at pci1 dev 0 function 1: Broadcom NetXtreme II BCM5716 1000Base-T
bnx1: Ethernet address a4:ba:db:39:03:ed
bnx1: interrupting at ioapic1 pin 16
brgphy1 at bnx1 phy 1: BCM5709 10/100/1000baseT PHY, rev. 8
brgphy1: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT,
1000baseT-FDX, auto
ppb1 at pci0 dev 3 function 0: vendor 0x8086 product 0x340a (rev. 0x13)
ppb1: unsupported PCI Express version
pci2 at ppb1 bus 2
pci2: i/o space, memory space enabled, rd/line, wr/inv ok
mpt0 at pci2 dev 0 function 0: vendor 0x1000 product 0x0058
mpt0: interrupting at ioapic1 pin 0
mpt0: Phy 0: Link Rate 3.0 Gbps
mpt0: Phy 1: Link Rate 3.0 Gbps
mpt0: Unknown async event: 0x21
mpt0: Unknown async event: 0x21
mpt0: Unknown async event: 0xb
scsibus0 at mpt0: 112 targets, 8 luns per target
ppb2 at pci0 dev 7 function 0: vendor 0x8086 product 0x340e (rev. 0x13)
ppb2: unsupported PCI Express version
pci3 at ppb2 bus 3
pci3: i/o space, memory space enabled, rd/line, wr/inv ok
vendor 0x8086 product 0x342e (interrupt system, revision 0x13) at pci0
dev 20 function 0 not configured
vendor 0x8086 product 0x3422 (interrupt system, revision 0x13) at pci0
dev 20 function 1 not configured
vendor 0x8086 product 0x3423 (interrupt system, revision 0x13) at pci0
dev 20 function 2 not configured
uhci0 at pci0 dev 26 function 0: vendor 0x8086 product 0x3a37 (rev. 0x00)
uhci0: interrupting at ioapic0 pin 17
usb0 at uhci0: USB revision 1.0
uhci1 at pci0 dev 26 function 1: vendor 0x8086 product 0x3a38 (rev. 0x00)
uhci1: interrupting at ioapic0 pin 18
usb1 at uhci1: USB revision 1.0
ehci0 at pci0 dev 26 function 7: vendor 0x8086 product 0x3a3c (rev. 0x00)
ehci0: interrupting at ioapic0 pin 19
ehci0: BIOS has given up ownership
ehci0: EHCI version 1.0
ehci0: companion controllers, 2 ports each: uhci0 uhci1
usb2 at ehci0: USB revision 2.0
uhci2 at pci0 dev 29 function 0: vendor 0x8086 product 0x3a34 (rev. 0x00)
uhci2: interrupting at ioapic0 pin 21
usb3 at uhci2: USB revision 1.0
uhci3 at pci0 dev 29 function 1: vendor 0x8086 product 0x3a35 (rev. 0x00)
uhci3: interrupting at ioapic0 pin 20
usb4 at uhci3: USB revision 1.0
uhci4 at pci0 dev 29 function 2: vendor 0x8086 product 0x3a36 (rev. 0x00)
uhci4: interrupting at ioapic0 pin 21
usb5 at uhci4: USB revision 1.0
uhci5 at pci0 dev 29 function 3: vendor 0x8086 product 0x3a39 (rev. 0x00)
uhci5: interrupting at ioapic0 pin 20
usb6 at uhci5: USB revision 1.0
ehci1 at pci0 dev 29 function 7: vendor 0x8086 product 0x3a3a (rev. 0x00)
ehci1: interrupting at ioapic0 pin 21
ehci1: EHCI version 1.0
ehci1: wrong number of companions (2 != 4)
ehci1: companion controllers, 2 ports each: uhci2 uhci3
usb7 at ehci1: USB revision 2.0
ppb3 at pci0 dev 30 function 0: vendor 0x8086 product 0x244e (rev. 0x90)
pci4 at ppb3 bus 4
pci4: i/o space, memory space enabled
vga1 at pci4 dev 3 function 0: vendor 0x102b product 0x0532 (rev. 0x0a)
wsdisplay0 at vga1 kbdmux 1: console (80x25, vt100 emulation)
wsmux1: connecting to wsdisplay0
drm at vga1 not configured
pcib0 at pci0 dev 31 function 0
pcib0: vendor 0x8086 product 0x3a16 (rev. 0x00)
piixide0 at pci0 dev 31 function 2
piixide0: Intel ICH10 Serial ATA 2 Controller 4x2 (rev. 0x00)
piixide0: bus-master DMA support present
piixide0: primary channel configured to native-PCI mode
piixide0: using ioapic0 pin 23 for native-PCI interrupt
atabus0 at piixide0 channel 0
piixide0: secondary channel configured to native-PCI mode
atabus1 at piixide0 channel 1
piixide1 at pci0 dev 31 function 5
piixide1: Intel ICH10 Serial ATA 2 Controller 2x2 (rev. 0x00)
piixide1: bus-master DMA support present
piixide1: primary channel wired to native-PCI mode
piixide1: using ioapic0 pin 22 for native-PCI interrupt
atabus2 at piixide1 channel 0
piixide1: secondary channel wired to native-PCI mode
atabus3 at piixide1 channel 1
isa0 at pcib0
com0 at isa0 port 0x3f8-0x3ff irq 4: ns16550a, working fifo
com1 at isa0 port 0x2f8-0x2ff irq 3: ns16550a, working fifo
pckbc0 at isa0 port 0x60-0x64
pckbd0 at pckbc0 (kbd slot)
pckbc0: using irq 1 for kbd slot
wskbd0 at pckbd0: console keyboard, using wsdisplay0
pcppi0 at isa0 port 0x61
midi0 at pcppi0: PC speaker (CPU-intensive output)
sysbeep0 at pcppi0
attimer1: attached to pcppi0
timecounter: Timecounter "clockinterrupt" frequency 100 Hz quality 0
scsibus0: waiting 2 seconds for devices to settle...
uhub0 at usb0: vendor 0x8086 UHCI root hub, class 9/0, rev 1.00/1.00, addr 1
uhub0: 2 ports with 2 removable, self powered
uhub1 at usb1: vendor 0x8086 UHCI root hub, class 9/0, rev 1.00/1.00, addr 1
uhub1: 2 ports with 2 removable, self powered
uhub2 at usb2: vendor 0x8086 EHCI root hub, class 9/0, rev 2.00/1.00, addr 1
uhub2: 4 ports with 4 removable, self powered
uhub3 at usb4: vendor 0x8086 UHCI root hub, class 9/0, rev 1.00/1.00, addr 1
uhub3: 2 ports with 2 removable, self powered
uhub4 at usb3: vendor 0x8086 UHCI root hub, class 9/0, rev 1.00/1.00, addr 1
uhub4: 2 ports with 2 removable, self powered
uhub5 at usb5: vendor 0x8086 UHCI root hub, class 9/0, rev 1.00/1.00, addr 1
uhub5: 2 ports with 2 removable, self powered
uhub6 at usb6: vendor 0x8086 UHCI root hub, class 9/0, rev 1.00/1.00, addr 1
uhub6: 2 ports with 2 removable, self powered
uhub7 at usb7: vendor 0x8086 EHCI root hub, class 9/0, rev 2.00/1.00, addr 1
uhub7: 4 ports with 4 removable, self powered
uhub8 at uhub2 port 3: vendor 0x0424 product 0x2514, class 9/0, rev
2.00/0.00, addr 2
uhub8: multiple transaction translators
uhub8: 4 ports with 4 removable, self powered
sd0 at scsibus0 target 0 lun 0: <Dell, VIRTUAL DISK, 1028> disk fixed
sd0: 136 GB, 139392 cyl, 16 head, 128 sec, 512 bytes/sect x 285474816 sectors
probe(mpt0:0:8:0): generic HBA error
uhub9 at uhub8 port 1: vendor 0x05e3 USB2.0 Hub, class 9/0, rev
2.00/77.60, addr 3
uhub9: single transaction translator
uhub9: 4 ports with 4 removable, self powered
uhidev0 at uhub9 port 1 configuration 1 interface 0
uhidev0: Dell Dell USB Keyboard, rev 1.10/3.01, addr 4, iclass 3/1
ukbd0 at uhidev0
wskbd1 at ukbd0 mux 1
wskbd1: connecting to wsdisplay0
uhidev1 at uhub9 port 2 configuration 1 interface 0
uhidev1: Logitech Optical USB Mouse, rev 2.00/3.40, addr 5, iclass 3/1
ums0 at uhidev1: 3 buttons and Z dir
wsmouse0 at ums0 mux 0
atapibus0 at atabus2: 2 targets
cd0 at atapibus0 drive 0: <TEAC DVD-ROM DV-28SW, 10040811130540, R.2A>
cdrom removable
cd0: 32-bit data port
cd0: drive supports PIO mode 4, DMA mode 2, Ultra-DMA mode 5 (Ultra/100)
cd0(piixide1:0:0): using PIO mode 4, Ultra-DMA mode 5 (Ultra/100) (using DMA)
ipmi0: version 2.0 interface KCS iobase 0xca8/8 spacing 4
Kernelized RAIDframe activated
pad0: outputs: 44100Hz, 16-bit, stereo
audio0 at pad0: half duplex, playback, capture
boot device: sd0
root on sd0a dumps on sd0b
root file system type: ffs
wsdisplay0: screen 1 added (80x25, vt100 emulation)
wsdisplay0: screen 2 added (80x25, vt100 emulation)
wsdisplay0: screen 3 added (80x25, vt100 emulation)
wsdisplay0: screen 4 added (80x25, vt100 emulation)
References:
- Dell R310 with SAS drives very slow
  - From: Tracy Di Marco White
- Re: Dell R310 with SAS drives very slow
  - From: Edgar Fuß
- Re: Dell R310 with SAS drives very slow
  - From: Manuel Bouyer
- Re: Dell R310 with SAS drives very slow
  - From: Edgar Fuß
Prev by Date: Re: Abit KV8Pro (VIA 8237) and viaide lost interrupts @ Sata HD
Next by Date: Re: TLS/lwp_private support
Previous by Thread: Re: Dell R310 with SAS drives very slow
Next by Thread: Re: Dell R310 with SAS drives very slow
Indexes:
Home | Main Index | Thread Index | Old Index