[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index][Old Index]
Re: kern/41579: siista driver interrupt and command completion handling broken
The following reply was made to PR kern/41579; it has been noted by GNATS.
Subject: Re: kern/41579: siista driver interrupt and command completion
Date: Mon, 22 Jun 2009 13:23:51 +0200
some background information from our setup:
(and some information related to the debugging process ...)
We have had this problem on all server-systems using a PCI-X 133 MHz =20
(4-port) or PCIe (2 port) controler with a chip from this family.
We never try such an IO-card this on a 32-bit bus.
An IO-Card with 2 or 4 3-GBit Ports makes no sence in a 32-bit slot, =20
because you will never get the data stream though it ... (4 * 66MHz =3D =20
264 MByte/sec - 3Gbit/10 =3D 300MByte/sec - so either 600MByte/sec or =20
1,2GByte/sec disc rate versus 264MByte/sec (or only 132MByte/sec on =20
33MHz) theoretical bus bandwith)
For such a card you need at least PCI-X 133MHz -> 8 * 133 =3D 1,064 GByte/se=
This sounds like the reason for me why you cannot reproduce the problem.
2x Intel S3000AHxx/S3200SHxx with Xeon 3xxx dual-Core-CPU (1x PCI-X, 1x PCIe=
1x Supermicro (type see bug-report) with Xeon 3xxx quad-core-CPU (PCI-X)
An 4 port card resides on my desc, because it makes no sence to install it .=
All systems running with serial console at 38400 baud.
"Originally" these system run an 4.x here with some drivers imported =20
from "-current" due to the need of the HW ...
Before I've reported the problem, I've setup the Supermicro system =20
with a 5.0 and run the test again - same problem. So it was not =20
related to the adaption of the driver to 4.x.
All system run fine for a wile and suddenly looses a disc on the =20
SATA-Card. It was very hard to debug, because I've found no load =20
scenario that will reproduce the problem after a short time reliable. =20
And the select style of the disc looks like random ...
We need more SATA-disc in the tree systems as the motherboards =20
supports, so I've to find the problem in an other way.
I've added some print statements to the driver to get the famous last =20
words before looseing a disc. (not realy successfull ...)
It looks like that the system was loosing interrupts, so I've started =20
debugging of the interrupt routines and dumped some state information =20
from the chip - again with printf statements.
An suddenly all interrupts are lost! The timeout for the transfer =20
happens, but no timeout processing takes place ...
I've located the main problem in the "wrong order" of register access =20
and operation mode of the chip. - The debug dump of the chip register =20
has cleared the interrupt cause. (And of cause in the missing timeout =20
handling code ...)
I'm not shure why the interrupt processing takes too much time on our =20
Due to the fact, that it was not realy reproducable with access to =20
large files or the raw-device with larger transfers (-> 64k-transfers, =20
the outdated limit from the ATA-stuff ...), the cause of the problem =20
may be short writes to disc as done by the filesystem when updating =20
some inode information.
To transfer 4k to the disc over a 3 Gbit-link will take something =20
around 13 us.
The driver uses only one slot and I'm not what will happen in the =20
upper layer of the wd-driver if a transfer completes. (The reason for =20
useing only one slot is the ATA abstraction in netbsd that doesn't =20
allow more than one pending transfer ... This should be corrected for =20
performance reason as soon as possible - but I've no time to fix it =20
for "all" drivers - sorry ...
The first try of a port for this chip family (without any usable =20
eror-handling :-( ...) to netbsd was mine, and it would have used all =20
slots - as done in the OpenBSD-version, but the ATA-layer cannot realy =20
handle it. So in fact only two slots are uses.
By the way: my port had been 10% faster as the driver for the =20
ICH-series - this one is something about 8% slower ...)
Back to the main thing ..
13 us is a short time - even on a fast CPU. And if some processing of =20
a completed transfer is done after starting the next one, it is =20
possible that this make take longer than 13 us. (I've haven't looked =20
in the wd-driver to verify this, but I hope the next transfer will be =20
started before dooing a wakeup for the completed. Otherwise the driver =20
would be very very inefficent !!!)
Due to the missing code fragment for timeout-processing, the timeout =20
will happen, but is ignored -> still a completed transfer on the chip, =20
but driver will never stop it.
remark: on a 32-Bit 33MHz bus this will take at least 30 usec due to =20
the limited bandwith of the system bus.
This would be a very good explanation for the problem.
If there are only "some" transfer for the discs, all processing has =20
finished before the next transfer is generated and started.
But if there are more transfers pending at a time and one of the =20
waiting is e.g. an inode-block update an interrupt is cleared in the =20
chip before interrupt processing takes place.
As faster as a CPU is, it will be able to process everything before =20
the chip generates the next interrupt and no disc is lost. And on a =20
system with only 132MByte/sec bandwith, the interrupt routine will be =20
done before the DMA was able to transfer the data to the chip.
Zitat von "Jonathan A. Kollasch" <jakllsch%kollasch.net@localhost>:
> The following reply was made to PR kern/41579; it has been noted by GNATS.
> From: "Jonathan A. Kollasch" <jakllsch%kollasch.net@localhost>
> To: gnats-bugs%NetBSD.org@localhost
> Subject: Re: kern/41579: siista driver interrupt and command completion
> =09handling broken
> Date: Wed, 17 Jun 2009 18:57:38 +0000
> I'm unable to reproduce this in either 5.0 or -current on my hardware
> (SiI3124 on 32-bit/33MHz PCI and SiI3132, WD5000AA[KJ]S and other
> older non-Flash drives). So, testing a forward-port of your patch
> myself will be hard to do.
> That said, this mostly looks like stuff that could use fixing.
> Thanks for pointing it out.
> I'd be interested in knowing more about this system and/or why the
> commands complete so fast.
Main Index |
Thread Index |