Re: kern/41579: siista driver interrupt and command completion handling broken

[Wolfgang.Stukenbrock%nagler-company.com@localhost]

The following reply was made to PR kern/41579; it has been noted by GNATS.

From: Wolfgang.Stukenbrock%nagler-company.com@localhost
To: gnats-bugs%NetBSD.org@localhost
Cc: jakllsch%kollasch.net@localhost
Subject: Re: kern/41579: siista driver interrupt and command completion
        handling broken
Date: Mon, 22 Jun 2009 13:23:51 +0200

 Hi,
 
 some background information from our setup:
 (and some information related to the debugging process ...)
 
 We have had this problem on all server-systems using a PCI-X 133 MHz =20
 (4-port) or PCIe (2 port) controler with a chip from this family.
 We never try such an IO-card this on a 32-bit bus.
 An IO-Card with 2 or 4 3-GBit Ports makes no sence in a 32-bit slot, =20
 because you will never get the data stream though it ... (4 * 66MHz =3D =20
 264 MByte/sec - 3Gbit/10 =3D 300MByte/sec - so either 600MByte/sec or =20
 1,2GByte/sec disc rate versus 264MByte/sec (or only 132MByte/sec on =20
 33MHz) theoretical bus bandwith)
 For such a card you need at least PCI-X 133MHz -> 8 * 133 =3D 1,064 GByte/se=
 c.
 This sounds like the reason for me why you cannot reproduce the problem.
 
 Our systems:
 2x Intel S3000AHxx/S3200SHxx with Xeon 3xxx dual-Core-CPU (1x PCI-X, 1x PCIe=
 )
 1x Supermicro (type see bug-report) with Xeon 3xxx quad-core-CPU (PCI-X)
 An 4 port card resides on my desc, because it makes no sence to install it .=
 ..
 All systems running with serial console at 38400 baud.
 
 "Originally" these system run an 4.x here with some drivers imported =20
 from "-current" due to the need of the HW ...
 Before I've reported the problem, I've setup the Supermicro system =20
 with a 5.0 and run the test again - same problem. So it was not =20
 related to the adaption of the driver to 4.x.
 
 All system run fine for a wile and suddenly looses a disc on the =20
 SATA-Card. It was very hard to debug, because I've found no load =20
 scenario that will reproduce the problem after a short time reliable. =20
 And the select style of the disc looks like random ...
 We need more SATA-disc in the tree systems as the motherboards =20
 supports, so I've to find the problem in an other way.
 I've added some print statements to the driver to get the famous last =20
 words before looseing a disc. (not realy successfull ...)
 It looks like that the system was loosing interrupts, so I've started =20
 debugging of the interrupt routines and dumped some state information =20
 from the chip - again with printf statements.
 An suddenly all interrupts are lost! The timeout for the transfer =20
 happens, but no timeout processing takes place ...
 I've located the main problem in the "wrong order" of register access =20
 and operation mode of the chip. - The debug dump of the chip register =20
 has cleared the interrupt cause. (And of cause in the missing timeout =20
 handling code ...)
 
 I'm not shure why the interrupt processing takes too much time on our =20
 systems sometimes.
 Due to the fact, that it was not realy reproducable with access to =20
 large files or the raw-device with larger transfers (-> 64k-transfers, =20
 the outdated limit from the ATA-stuff ...), the cause of the problem =20
 may be short writes to disc as done by the filesystem when updating =20
 some inode information.
 To transfer 4k to the disc over a 3 Gbit-link will take something =20
 around 13 us.
 The driver uses only one slot and I'm not what will happen in the =20
 upper layer of the wd-driver if a transfer completes. (The reason for =20
 useing only one slot is the ATA abstraction in netbsd that doesn't =20
 allow more than one pending transfer ... This should be corrected for =20
 performance reason as soon as possible - but I've no time to fix it =20
 for "all" drivers - sorry ...
 The first try of a port for this chip family (without any usable =20
 eror-handling :-( ...) to netbsd was mine, and it would have used all =20
 slots - as done in the OpenBSD-version, but the ATA-layer cannot realy =20
 handle it. So in fact only two slots are uses.
 By the way: my port had been 10% faster as the driver for the =20
 ICH-series - this one is something about 8% slower ...)
 
 Back to the main thing ..
 13 us is a short time - even on a fast CPU. And if some processing of =20
 a completed transfer is done after starting the next one, it is =20
 possible that this make take longer than 13 us. (I've haven't looked =20
 in the wd-driver to verify this, but I hope the next transfer will be =20
 started before dooing a wakeup for the completed. Otherwise the driver =20
 would be very very inefficent !!!)
 Due to the missing code fragment for timeout-processing, the timeout =20
 will happen, but is ignored -> still a completed transfer on the chip, =20
 but driver will never stop it.
 remark: on a 32-Bit 33MHz bus this will take at least 30 usec due to =20
 the limited bandwith of the system bus.
 
 This would be a very good explanation for the problem.
 If there are only "some" transfer for the discs, all processing has =20
 finished before the next transfer is generated and started.
 But if there are more transfers pending at a time and one of the =20
 waiting is e.g. an inode-block update an interrupt is cleared in the =20
 chip before interrupt processing takes place.
 As faster as a CPU is, it will be able to process everything before =20
 the chip generates the next interrupt and no disc is lost. And on a =20
 system with only 132MByte/sec bandwith, the interrupt routine will be =20
 done before the DMA was able to transfer the data to the chip.
 
 best regards
 
 W. Stukenbrock
 
 Zitat von "Jonathan A. Kollasch" <jakllsch%kollasch.net@localhost>:
 
 > The following reply was made to PR kern/41579; it has been noted by GNATS.
 >
 > From: "Jonathan A. Kollasch" <jakllsch%kollasch.net@localhost>
 > To: gnats-bugs%NetBSD.org@localhost
 > Cc:
 > Subject: Re: kern/41579: siista driver interrupt and command completion
 > =09handling broken
 > Date: Wed, 17 Jun 2009 18:57:38 +0000
 >
 >  I'm unable to reproduce this in either 5.0 or -current on my hardware
 >  (SiI3124 on 32-bit/33MHz PCI and SiI3132, WD5000AA[KJ]S and other
 >  older non-Flash drives).  So, testing a forward-port of your patch
 >  myself will be hard to do.
 >
 >  That said, this mostly looks like stuff that could use fixing.
 >  Thanks for pointing it out.
 >
 >  I'd be interested in knowing more about this system and/or why the
 >  commands complete so fast.
 >
 >