Re: kern/39686: NetBSD 4.x has I/O problems on HP Compaq DL38[0|5] w/ Smart Array 6i controller

["Michael L. Hitch" <mhitch%lightning.msu.montana.edu@localhost>]

The following reply was made to PR kern/39686; it has been noted by GNATS.

From: "Michael L. Hitch" <mhitch%lightning.msu.montana.edu@localhost>
To: gnats-bugs%NetBSD.org@localhost
Cc: 
Subject: Re: kern/39686: NetBSD 4.x has I/O problems on HP Compaq DL38[0|5]
 w/ Smart Array 6i controller
Date: Thu, 15 Apr 2010 15:11:30 -0600 (MDT)

 On Fri, 3 Oct 2008, pettai%nordu.net@localhost wrote:
 
 > This problem has been found on both HP Compaq DL 380 and DL 385 with the 
 > Smart Array 6i controller. We first tried NetBSD 4.0 RELEASE i386/amd64, 
 > which gave the same symptoms.
 >
 > Boot up the installation of NetBSD 4.99.72 (w/ "no APCI" enabled or 
 > disable), and during the installation (with progress) of the NetBSD base 
 > system the installer will show "- stalled -" instead of "xyz MiB/s". It 
 > usually happens after 79%+ of progress while installing (extracting via tar) 
 > the comp.tgz and xcomp.tgz packages.
 >
 >
 >
 >> How-To-Repeat:
 > The general I/O performance against the disk(controller) is extremely slow. 
 > Extracting pkgsrc.tgz takes about an hour...
 
    This will occur on any of the Compaq/HP servers with Smart Array 
 controllers that do not have a battery-backed cache with write caching 
 enabled.  Without battery backup, the controller will do no write caching 
 at all, and I presume turns off any write caching on the disk drives. 
 This causes any writes to the logical drives to be quite slow, since the 
 controller will not indicate the write has completed until all the data 
 has been written to all the associated disk drives.  I noticed this when I 
 was working on getting ciss(4) running on a DL360 G4 with a 5i or 6i 
 controller.  The DL360 that I requested for a later project had the 
 battery-backed cache option, and write performance was much better.
 
    The netbsd-users thread that lead to this PR, 
 http://mail-index.netbsd.org/netbsd-users/2008/10/01/msg002083.html , 
 mentions that FreeBSD works "fine".  I was never able to find anything in 
 the FreeBSD driver, nor the Linux driver, nor the OpenBSD driver that 
 could have helped in this case.  I think I did do a simple test on Linux, 
 and found that if I wrote a very large amount of data using dd, it would 
 eventually get quite a bit slower.  My guess at the time was that Linux 
 was buffering even the dd output in memory making it appear to be much 
 faster that the disk writes actually were.
 
    With the mention of this problem recently, I started digging into it 
 again, using a ML370 G3.  This machine didn't have the integrated Smart 
 Array, but did have a Smart Array 642 without a battery-backed cache (I 
 had also added a Smart Array 5300, which does have battery-backed cache).
 
    I did verify that extracting pkgsrc.tar.gz onto a RAID1 volume did take 
 just over an hour, with the disks on the SA 642.  With the SA 5300, the 
 same extract took less than 3 minutes.  Then I disabled array acceleration 
 on that RAID1 volume, and was back to the 1 hour time to extract 
 pkgsrc.tar.gz.
 
    I had a FreeBSD livecd (FreeSBIE, based on FreeBSD 6.2), and found that 
 the pkgsrc.tar.gz extract took about 32 minutes.  While much better than 
 the 1 hour that NetBSD took, I wouldn't consider it good performance. 
 Wondering what the difference might be, I took a look at the number of 
 disk transfers for both NetBSD and FreeBSD, and found that NetBSD did 
 almost 2 times the number of transfer - which accounts for the difference 
 in speed.
 
    The next think I tried was to use a log mount (WAPBL) on NetBSD [I'm 
 running with NetBSD-5.0_STABLE by the way].  Wow, what a difference!  The 
 pkgsrc.tar.gz extract time was 17 minutes.
 
    I'll be running some more tests with FreeBSD with the array acceleration 
 enabled to compare with the non-accelerated case, and will then probably 
 try the same variations with a Linux live CD I have.
 
    So, for the best performance with ciss(4), you really need to have a 
 battery-backed cache option.  Failing that, running NetBSD 5.0 (or later) 
 and using a log mounted file system should help (although the 
 pkgsrc.tar.gz extract is probably an extreme case where WAPBL helps 
 significantly).
 
 --
 Michael L. Hitch                       mhitch%montana.edu@localhost
 Computer Consultant
 Information Technology Center
 Montana State University       Bozeman, MT     USA