On Fri 28 Nov 2003 at 19:25:05 -0800, Bill Studenmund wrote:
> > So it's not like the card is not capable of proper transfer counts and
> > speeds, it seems it is held back somehow by the file system or buffer
> > cache.
> 
> What is the setup of your raid volume?

We have a RAID-5 of 5 U-320 SCSI disks of about 35 G each, and one hot
spare. I have also attached the dmesg output and the disklabel.

For some reason, the tracks/cylinder value keeps getting changed back to
255, even though I changed it to 51 (and multiplied  the sectors/track
by 5).  I had to do this in order to newfs the partitions without
resorting to a larger bsize. Sysinst fails on this, still, which is
quite annoying.

I have been playing a bit with the trickle sync. In
sys/miscfs/syncfs/sync_subr.c I changed the delay in the syncer task
from one second to 1/4 second, and it had some effect, but only in a
strange way. During the run of bonnie++, the xfers/sec increases in the
beginning, but only until all memory has been used for the file cache -
then it drops back to 60 again, as before.

I also noticed that the syncer task tries to obtain a vnode lock, so I
made it print some output to indicate whether that succeeds. My
reasoning was that it won't sync much if it can never lock the vnode.
But it does get it almost all the time; only when I decrease the waits
in the syncer it starts to fail to get the lock occasionally.

Anyway, if this is a problem with the trickle sync it must still depend
on this particular hardware, since runs of bonnie++ on older, completely
different hardware (but also with NetBSD 1.6.1) gives much higher
values for xfers/sec.

I also thought about dropping the FreeBSD aac driver into NetBSD, since
it seems to be changing there, while it has been essentially static in
NetBSD for ages, but I saw so many differences that are apparently due
to difference in environment, that I gave up on that idea (at least as a
quick solution).

> Bill

Output from rawio:

bash-2.05b# ./rawio -s 1g 1g -a /dev/rld0b
           Random read  Sequential read    Random write Sequential write
ID          K/sec  /sec    K/sec  /sec     K/sec  /sec     K/sec  /sec
ld0b      15587.8   950  32273.5  1970   15127.1   933   31824.6  1942  

Output from bonnie++:

bash-2.05b# bonnie++ -d . -s 1024 -n 0 -u nobody

Version  1.03       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
rode             1G  3889   3  3888   0  3389   1 41281  44 48774   8  1847   3
rode,1G,3889,3,3888,0,3389,1,41281,44,48774,8,1846.9,3,,,,,,,,,,,,,

Typical values from systat vmstat:

Discs  ld0  cd0  fd0  md0                              64 fmin
seeks                                                  85 ftarg
xfers   61                                          77595 itarg
Kbyte 3837                                            472 wired
%busy100.0                                                pdfre

A second test with a bigger file (2G) seemed to stall almost completely
for a (long) while, while I went out shopping. It dropped to about 5
xfers/sec. (This is even worse than I thought!!) This shows in the
rewrite performance figure. Still nothing obviously wrong with the
reading performance.

Version  1.03       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
rode             2G  3851   3  3880   0   373   0 44067  48 55674   9 544.2   1

Discs  ld0  cd0  fd0  md0                              64 fmin
seeks                                                  85 ftarg
xfers    5                                          77589 itarg
Kbyte  311                                            474 wired
%busy109.8                                                pdfre

Disklabel:

# /dev/rld0d:
type: unknown
disk: klei
label: 
flags:
bytes/sector: 512
sectors/track: 63
tracks/cylinder: 255
sectors/cylinder: 16065
cylinders: 17846
total sectors: 286709760
rpm: 3600
interleave: 1
trackskew: 0
cylinderskew: 0
headswitch: 0		# microseconds
track-to-track seek: 0	# microseconds
drivedata: 0 

16 partitions:
#        size    offset     fstype  [fsize bsize cpg/sgs]
 a:    273042        63     4.2BSD   1024  8192     0   # (Cyl.    0*- 16)
 b:   4209030    273105       swap                      # (Cyl.   17 - 278)
 c: 286709697        63     unused      0     0         # (Cyl.    0*- 17846*)
 d: 286709760         0     unused      0     0         # (Cyl.    0 - 17846*)
 e:   6152895   4482135     4.2BSD   1024  8192     0   # (Cyl.  279 - 661)
 f:   1060290  10635030     4.2BSD   1024  8192     0   # (Cyl.  662 - 727)
 g:   1060290  11695320     4.2BSD   1024  8192     0   # (Cyl.  728 - 793)
 h: 136970190  12755610     4.2BSD   1024  8192     0   # (Cyl.  794 - 9319)
 i: 136983960 149725800     4.2BSD   1024  8192     0   # (Cyl. 9320 - 17846*)

NetBSD 1.6.1 (RODE) #4: Fri Nov 28 18:47:51 CET 2003
    root@rode:/usr/src/sys/arch/i386/compile/RODE
cpu0: Intel Pentium 4 (686-class), 2392.17 MHz
cpu0: D-cache 8 KB 64b/line 4-way
cpu0: L2 cache 512 KB 64b/line 8-way
cpu0: features ffffffffbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR>
cpu0: features ffffffffbfebfbff<PGE,MCA,CMOV,FGPAT,PSE36,CFLUSH,DS,ACPI,MMX>
cpu0: features ffffffffbfebfbff<FXSR,SSE,SSE2,SS,HTT,TM,B31>
total memory = 1023 MB
avail memory = 946 MB
using 6144 buffers containing 52480 KB of memory
BIOS32 rev. 0 found at 0xfd6e0
mainbus0 (root)
pci0 at mainbus0 bus 0: configuration mode 1
pci0: i/o space, memory space enabled, rd/line, rd/mult, wr/inv ok
pchb0 at pci0 dev 0 function 0
pchb0: Intel product 0x254c (rev. 0x01)
Intel E7500 MCH DRAM Controller (undefined subclass 0x00, revision 0x01) at pci0 dev 0 function 1 not configured
ppb0 at pci0 dev 2 function 0: Intel E7500 MCH HI_B vppb 1 (rev. 0x01)
pci1 at ppb0 bus 1
pci1: i/o space, memory space enabled
Intel 82870P2 P64H2 IOxAPIC (interrupt system, interface 0x20, revision 0x04) at pci1 dev 28 function 0 not configured
ppb1 at pci1 dev 29 function 0: Intel 82870P2 P64H2 PCI-to-PCI Bridge (rev. 0x04)
pci2 at ppb1 bus 2
pci2: i/o space, memory space enabled
Intel 82870P2 P64H2 IOxAPIC (interrupt system, interface 0x20, revision 0x04) at pci1 dev 30 function 0 not configured
ppb2 at pci1 dev 31 function 0: Intel 82870P2 P64H2 PCI-to-PCI Bridge (rev. 0x04)
pci3 at ppb2 bus 3
pci3: i/o space, memory space enabled
aac0 at pci3 dev 1 function 0: Adaptec ASR-2120S
aac0: interrupting at irq 10
aac0: i960RX at 100MHz, 48MB cache, optional battery present, kernel 3.4-0
ld0 at aac0 unit 0: RAID 5
ld0: 136 GB, 17846 cyl, 255 head, 63 sec, 512 bytes/sect x 286709760 sectors
Intel E7500 MCH HI_B vppb 2 (undefined subclass 0x00, revision 0x01) at pci0 dev 2 function 1 not configured
uhci0 at pci0 dev 29 function 0: Intel 82801CA/CAM USB Controller (rev. 0x02)
uhci0: interrupting at irq 10
usb0 at uhci0: USB revision 1.0
uhub0 at usb0
uhub0: Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1
uhub0: 2 ports with 2 removable, self powered
uhci1 at pci0 dev 29 function 1: Intel 82801CA/CAM USB Controller (rev. 0x02)
uhci1: interrupting at irq 5
usb1 at uhci1: USB revision 1.0
uhub1 at usb1
uhub1: Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1
uhub1: 2 ports with 2 removable, self powered
uhci2 at pci0 dev 29 function 2: Intel 82801CA/CAM USB Controller (rev. 0x02)
uhci2: interrupting at irq 10
usb2 at uhci2: USB revision 1.0
uhub2 at usb2
uhub2: Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1
uhub2: 2 ports with 2 removable, self powered
ppb3 at pci0 dev 30 function 0: Intel 82801BA Hub-to-PCI Bridge (rev. 0x42)
pci4 at ppb3 bus 4
pci4: i/o space, memory space enabled
vga0 at pci4 dev 3 function 0: ATI Technologies Rage XL (rev. 0x27)
wsdisplay0 at vga0 kbdmux 1: console (80x25, vt100 emulation)
wsmux1: connecting to wsdisplay0
fxp0 at pci4 dev 4 function 0: i82550 Ethernet, rev 13
fxp0: interrupting at irq 11
fxp0: Ethernet address 00:02:b3:d8:db:27
inphy0 at fxp0 phy 1: i82555 10/100 media interface, rev. 4
inphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto
wm0 at pci4 dev 5 function 0: Intel i82540EM 1000BASE-T Ethernet, rev. 2
wm0: interrupting at irq 10
wm0: Ethernet address 00:02:b3:d8:dc:37
makphy0 at wm0 phy 1: Marvell 88E1011 Gigabit PHY, rev. 3
makphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-FDX, auto
pcib0 at pci0 dev 31 function 0
pcib0: Intel 82801CA LPC Interface (rev. 0x02)
pciide0 at pci0 dev 31 function 1: Intel 82801CA IDE Controller (rev. 0x02)
pciide0: bus-master DMA support present
pciide0: primary channel wired to compatibility mode
atapibus0 at pciide0 channel 0: 2 targets
cd0 at atapibus0 drive 0: <CD-224E, , 1.9A> type 5 cdrom removable
cd0: 32-bit data port
cd0: drive supports PIO mode 4, DMA mode 2, Ultra-DMA mode 2 (Ultra/33)
pciide0: primary channel interrupting at irq 14
cd0(pciide0:0:0): using PIO mode 4, Ultra-DMA mode 2 (Ultra/33) (using DMA data transfers)
pciide0: secondary channel wired to compatibility mode
pciide0: disabling secondary channel (no drives)
Intel 82801CA/CAM SMBus Controller (SMBus serial bus, revision 0x02) at pci0 dev 31 function 3 not configured
isa0 at pcib0
com0 at isa0 port 0x3f8-0x3ff irq 4: ns16550a, working fifo
com1 at isa0 port 0x2f8-0x2ff irq 3: ns16550a, working fifo
pckbc0 at isa0 port 0x60-0x64
pckbd0 at pckbc0 (kbd slot)
pckbc0: using irq 1 for kbd slot
wskbd0 at pckbd0: console keyboard, using wsdisplay0
pms0 at pckbc0 (aux slot)
pckbc0: using irq 12 for aux slot
wsmouse0 at pms0 mux 0
lpt0 at isa0 port 0x378-0x37b irq 7
pcppi0 at isa0 port 0x61
sysbeep0 at pcppi0
npx0 at isa0 port 0xf0-0xff: using exception 16
fdc0 at isa0 port 0x3f0-0x3f7 irq 6 drq 2
fd0 at fdc0 drive 0: 1.44MB, 80 cyl, 2 head, 18 sec
biomask e765 netmask ef65 ttymask ffe7
boot device: ld0
root on ld0a dumps on ld0b
root file system type: ffs
wsdisplay0: screen 1 added (80x25, vt100 emulation)
wsdisplay0: screen 2 added (80x25, vt100 emulation)
wsdisplay0: screen 3 added (80x25, vt100 emulation)
wsdisplay0: screen 4 added (80x25, vt100 emulation)

-Olaf.
-- 
___ Olaf 'Rhialto' Seibert - rhialto@       -- "What good is a Ring of Power
\X/ polderland.nl            -- if you're unable...to Speak." - Agent Elrond