Subject: Re: very strange "temporary" filesytem lockups on netbsd-4 (i386)
To: NetBSD Kernel Technical Discussion List <tech-kern@NetBSD.org>
From: Greg A. Woods <woods@planix.ca>
List: netbsd-help
Date: 02/20/2007 14:44:38
OK, here's what's been happening since my first report....
I built a new kernel with the recent changes on netbsd-4 done to fix
many deadlock problems in NetBSD's nullfs and unionfs virtual
filesystems, hoping without much expectation that this would change the
behaviour of the temporary freezes I was seeing. Indeed there were no
improvements and no change in behaviour either.
I've worked around the problem for now with the following line added to
the bottom of /etc/rc.local:
nohup sh -c 'while : ; do dd if=/dev/rld0d of=/dev/null count=1; sleep 1; done' &
This does an IO on the stuck device once per second, restarting it, and
it seems to keep everything moving along reasonably smoothly.
I did capture some not-terribly-interesting kernel info from DDB though
and all you basically see is "ld" stuck in getblk state:
[Mon Feb 19 22:50:28 2007]login: [-- break #1 sent -- `\z' -- Mon Feb 19 22:54:51 2007]
Stopped at netbsd:cpu_Debugger+0x4: popl %ebp
[Mon Feb 19 22:54:51 2007]db{0}> ps
[Mon Feb 19 22:54:53 2007] PID PPID PGRP UID S FLAGS LWPS COMMAND WAIT
[Mon Feb 19 22:54:53 2007] 3920 18420 10164 1000 2 0x4002 1 ld getblk
[Mon Feb 19 22:54:53 2007] 18420 16402 10164 1000 2 0x4002 1 collect2 wait
[Mon Feb 19 22:54:53 2007] 16402 3137 10164 1000 2 0x4002 1 i386--netbsdelf- wait
[Mon Feb 19 22:54:53 2007] 3137 9416 10164 1000 2 0x4002 1 sh wait
[Mon Feb 19 22:54:53 2007] 9416 13265 10164 1000 2 0x4002 1 nbmake wait
[Mon Feb 19 22:54:53 2007] 13265 3034 10164 1000 2 0x4002 1 sh wait
[Mon Feb 19 22:54:53 2007] 3034 15061 10164 1000 2 0x4002 1 nbmake wait
[Mon Feb 19 22:54:53 2007] 15061 2593 10164 1000 2 0x4002 1 sh wait
[Mon Feb 19 22:54:53 2007] 2593 27295 10164 1000 2 0x4002 1 nbmake wait
[Mon Feb 19 22:54:53 2007] 27295 15819 10164 1000 2 0x4002 1 sh wait
[Mon Feb 19 22:54:53 2007] 18596 6309 18596 1000 2 0x4002 1 tail kqread
[Mon Feb 19 22:54:53 2007] 6309 20011 6309 1000 2 0x4002 1 ksh pause
[Mon Feb 19 22:54:53 2007] 20011 763 763 0 2 0x4100 1 rlogind poll
[Mon Feb 19 22:54:53 2007] 15819 10280 10164 1000 2 0x4002 1 nbmake wait
[Mon Feb 19 22:54:53 2007] 10280 28521 10164 1000 2 0x4002 1 sh wait
[Mon Feb 19 22:54:53 2007] 28521 27102 10164 1000 2 0x4002 1 nbmake wait
[Mon Feb 19 22:54:53 2007] 27102 2473 10164 1000 2 0x4002 1 sh wait
[Mon Feb 19 22:54:53 2007] 12119 621 12119 1000 2 0x4102 1 systat ttyin
[Mon Feb 19 22:54:53 2007] 2473 9639 10164 1000 2 0x4002 1 nbmake wait
[Mon Feb 19 22:54:53 2007] 9639 10936 10164 1000 2 0x4002 1 sh wait
[Mon Feb 19 22:54:53 2007] 10936 10384 10164 1000 2 0x4002 1 nbmake wait
[Mon Feb 19 22:54:53 2007] 10384 10164 10164 1000 2 0x4002 1 sh wait
[Mon Feb 19 22:54:53 2007] 10164 621 10164 1000 2 0x4002 1 ksh pause
[Mon Feb 19 22:54:53 2007] 621 620 621 1000 2 0x4003 1 ksh pause
[Mon Feb 19 22:54:54 2007] 620 763 763 0 2 0x4100 1 rlogind poll
[Mon Feb 19 22:54:54 2007] 695 1 695 0 2 0x4002 1 getty ttyin
[Mon Feb 19 22:54:54 2007] 826 1 826 0 2 0x4002 1 getty ttyin
[Mon Feb 19 22:54:54 2007] 782 1 782 0 2 0x4002 1 getty ttyin
[Mon Feb 19 22:54:54 2007]--db_more--
[Mon Feb 19 22:55:01 2007]db{0}> bt
[Mon Feb 19 22:55:08 2007]cpu_Debugger(c5145200,0,cea02f38,c0334f70,c5145260) at netbsd:cpu_Debugger+0x4
[Mon Feb 19 22:55:08 2007]comintr(c5145200,0,cea00010,30,10) at netbsd:comintr+0x6e0
[Mon Feb 19 22:55:08 2007]Xintr_ioapic_edge4() at netbsd:Xintr_ioapic_edge4+0x9c
[Mon Feb 19 22:55:08 2007]--- interrupt ---
[Mon Feb 19 22:55:08 2007]netbsd:cpu_switch+0xab:
[Mon Feb 19 22:55:46 2007]db{0}> machine cpu 1
[Mon Feb 19 22:55:55 2007]using CPU 1
[Mon Feb 19 22:55:55 2007]db{0}> bt
[Mon Feb 19 22:55:58 2007]netbsd:cpu_switch+0xab:
[Mon Feb 19 22:55:58 2007]db{0}> machine cpu 2
[Mon Feb 19 22:56:04 2007]using CPU 2
[Mon Feb 19 22:56:04 2007]db{0}> bt
[Mon Feb 19 22:56:05 2007]netbsd:cpu_switch+0xab:
[Mon Feb 19 22:56:05 2007]db{0}> machine cpu 3
[Mon Feb 19 22:56:08 2007]using CPU 3
[Mon Feb 19 22:56:08 2007]db{0}> bt
[Mon Feb 19 22:56:09 2007]netbsd:cpu_switch+0xab:
[Mon Feb 19 22:56:09 2007]db{0}> machine cpu 0
[Mon Feb 19 22:56:14 2007]using CPU 0
[Mon Feb 19 22:56:14 2007]db{0}> bt
[Mon Feb 19 22:56:17 2007]cpu_Debugger(c5145200,0,cea02f38,c0334f70,c5145260) at netbsd:cpu_Debugger+0x4
[Mon Feb 19 22:56:17 2007]comintr(c5145200,0,cea00010,30,10) at netbsd:comintr+0x6e0
[Mon Feb 19 22:56:17 2007]Xintr_ioapic_edge4() at netbsd:Xintr_ioapic_edge4+0x9c
[Mon Feb 19 22:56:17 2007]--- interrupt ---
[Mon Feb 19 22:56:17 2007]netbsd:cpu_switch+0xab:
[Mon Feb 19 22:56:17 2007]db{0}> cont
[Mon Feb 19 22:57:03 2007]
[Mon Feb 19 22:57:08 2007]
[Mon Feb 19 22:57:08 2007]NetBSD/i386 (once) (constty)
[Mon Feb 19 22:57:08 2007]
[Mon Feb 19 22:57:08 2007]login:
It looks more and more that the big culprit is the extreme age and
bit-rot in the aac(4) driver is at fault. There have been many
bug-fixes, improvements, and updates to the original FreeBSD sources
since it was ported to NetBSD, including this interesting one with a
possible fix and explanation for lost interrupts which would explain why
the I/O requests pending at the time of the hang never complete until
another I/O is started.
Unfortunately this fix was done after the driver's interrupt handling
was threaded:
----------------------------
revision 1.46
date: 2003/01/11 01:59:21; author: scottl; state: Exp; lines: +50 -39
Major bugfixes for large memory and fast systems.
aac.c:
Re-arrange the interrupt handler to optimize the common case of
the adapter interrupting us because one or more commands are complete,
and do a read across the pci bus to ensure that all posted status
writes are flushed. This should close a race that could cause command
completion interrupts to be lost.
Follow the spec a bit closer when filling out command structures.
Enable the Fast Response feature to eliminate the need for the card
to DMA successfull command completions back into host memory.
Tell the controller how much physical memory we have. Without this
there was a chance that our DMA regions would collide with the
memory window used by the cache on the controller. The result would
be massive data corruption. This seemed to mainly affect systems with
>2GB of memory.
Fix a few whitespace problems.
aac_debug.c:
Add an extra diagnostic when printing out commands.
aac_disk.c:
Add extra sanity checks.
aacreg.h:
Prepare for making this 64-bit clean by reducing the use of enumeration
types in structures.
Many thanks to Justin Gibbs for helping track these down.
----------------------------
Re-porting the driver from FreeBSD seems quite daunting, given the
major divergences growing between FreeBSD and NetBSD, particularly in
storage device drivers.
However much/all of the good work put into the FreeBSD driver has been
picked up by OpenBSD in their re-write of the aac(4) driver. It would
seem on first glance to be much easier to port the new OpenBSD driver
back over to NetBSD. Unfortunately I don't have time to do this all on
my own at the moment, though I would be able to help test, at least on
the netbsd-4 branch, as I have several HP NetRAID cards as well as this
Dell PE2650 with the embedded PERC controller.
At minimum it sure would be nice if someone could figure some way to fix
the race condition that seems to be causing interrupts to be lost.
Should I open a PR with the info in this and the last e-mail?
--
Greg A. Woods
Planix, Inc.
<woods@planix.ca> +1 416 489-5852 x122 http://www.planix.com/