Subject: Re: very strange "temporary" filesytem lockups on netbsd-4 (i386)
To: NetBSD Kernel Technical Discussion List <tech-kern@NetBSD.org>
From: Brian Buhrow <buhrow@lothlorien.nfbcal.org>
List: tech-kern
Date: 02/20/2007 12:06:05
You know, this reminds me of when I worked on the twe(4) driver some
years ago. We had a problem where we'd issue a command to the raid card,
and, sometimes, the card would just black hole it. The driver would wait
for the card to respond, and it never would. At the time, I wrote a patch
which implemented a watchdog timer for the raid card, and if we didn't see
any activity after we issued a command for a couple of seconds, we'd kick
the card, and that would usually get things going again until the next time
it happened. My guess is that the version of the aac(4) driver you're
using isn't using a watchdog type mechanism to make sure the card behaves.
Your work around seems to confirm this theory since issuing another i/o
request to the card gets it going again. I just looked at the aac(4)
driver in 3.0, and it doesn't appear to implement any sort of watchdog
timer mechanism. Even more interesting, if the card generates an interrupt
the driver doesn't recognize, it clears the interrupt, and does nothing.
One of the interrupts it appears not to recognize, is an incomplete response. So,
if the card is generating interrupts that are considered invalid by the
driver, they're not logged or acknowledged at the upper levels. They're
just cleared. I'd say there's a good chance this is happening to you for
some reason. Putting some instrumentation in /usr/src/sys/dev/ic/aac.c
should shed some light on this problem.
Hope that helps.
-Brian
On Feb 20, 2:44pm, "Greg A. Woods" wrote:
} Subject: Re: very strange "temporary" filesytem lockups on netbsd-4 (i386)
} OK, here's what's been happening since my first report....
}
} I built a new kernel with the recent changes on netbsd-4 done to fix
} many deadlock problems in NetBSD's nullfs and unionfs virtual
} filesystems, hoping without much expectation that this would change the
} behaviour of the temporary freezes I was seeing. Indeed there were no
} improvements and no change in behaviour either.
}
} I've worked around the problem for now with the following line added to
} the bottom of /etc/rc.local:
}
} nohup sh -c 'while : ; do dd if=/dev/rld0d of=/dev/null count=1; sleep 1; done' &
}
} This does an IO on the stuck device once per second, restarting it, and
} it seems to keep everything moving along reasonably smoothly.
}
} I did capture some not-terribly-interesting kernel info from DDB though
} and all you basically see is "ld" stuck in getblk state:
}
} [Mon Feb 19 22:50:28 2007]login: [-- break #1 sent -- `\z' -- Mon Feb 19 22:54:51 2007]
} Stopped at netbsd:cpu_Debugger+0x4: popl %ebp
} [Mon Feb 19 22:54:51 2007]db{0}> ps
} [Mon Feb 19 22:54:53 2007] PID PPID PGRP UID S FLAGS LWPS COMMAND WAIT
} [Mon Feb 19 22:54:53 2007] 3920 18420 10164 1000 2 0x4002 1 ld getblk
} [Mon Feb 19 22:54:53 2007] 18420 16402 10164 1000 2 0x4002 1 collect2 wait
} [Mon Feb 19 22:54:53 2007] 16402 3137 10164 1000 2 0x4002 1 i386--netbsdelf- wait
} [Mon Feb 19 22:54:53 2007] 3137 9416 10164 1000 2 0x4002 1 sh wait
} [Mon Feb 19 22:54:53 2007] 9416 13265 10164 1000 2 0x4002 1 nbmake wait
} [Mon Feb 19 22:54:53 2007] 13265 3034 10164 1000 2 0x4002 1 sh wait
} [Mon Feb 19 22:54:53 2007] 3034 15061 10164 1000 2 0x4002 1 nbmake wait
} [Mon Feb 19 22:54:53 2007] 15061 2593 10164 1000 2 0x4002 1 sh wait
} [Mon Feb 19 22:54:53 2007] 2593 27295 10164 1000 2 0x4002 1 nbmake wait
} [Mon Feb 19 22:54:53 2007] 27295 15819 10164 1000 2 0x4002 1 sh wait
} [Mon Feb 19 22:54:53 2007] 18596 6309 18596 1000 2 0x4002 1 tail kqread
} [Mon Feb 19 22:54:53 2007] 6309 20011 6309 1000 2 0x4002 1 ksh pause
} [Mon Feb 19 22:54:53 2007] 20011 763 763 0 2 0x4100 1 rlogind poll
} [Mon Feb 19 22:54:53 2007] 15819 10280 10164 1000 2 0x4002 1 nbmake wait
} [Mon Feb 19 22:54:53 2007] 10280 28521 10164 1000 2 0x4002 1 sh wait
} [Mon Feb 19 22:54:53 2007] 28521 27102 10164 1000 2 0x4002 1 nbmake wait
} [Mon Feb 19 22:54:53 2007] 27102 2473 10164 1000 2 0x4002 1 sh wait
} [Mon Feb 19 22:54:53 2007] 12119 621 12119 1000 2 0x4102 1 systat ttyin
} [Mon Feb 19 22:54:53 2007] 2473 9639 10164 1000 2 0x4002 1 nbmake wait
} [Mon Feb 19 22:54:53 2007] 9639 10936 10164 1000 2 0x4002 1 sh wait
} [Mon Feb 19 22:54:53 2007] 10936 10384 10164 1000 2 0x4002 1 nbmake wait
} [Mon Feb 19 22:54:53 2007] 10384 10164 10164 1000 2 0x4002 1 sh wait
} [Mon Feb 19 22:54:53 2007] 10164 621 10164 1000 2 0x4002 1 ksh pause
} [Mon Feb 19 22:54:53 2007] 621 620 621 1000 2 0x4003 1 ksh pause
} [Mon Feb 19 22:54:54 2007] 620 763 763 0 2 0x4100 1 rlogind poll
} [Mon Feb 19 22:54:54 2007] 695 1 695 0 2 0x4002 1 getty ttyin
} [Mon Feb 19 22:54:54 2007] 826 1 826 0 2 0x4002 1 getty ttyin
} [Mon Feb 19 22:54:54 2007] 782 1 782 0 2 0x4002 1 getty ttyin
} [Mon Feb 19 22:54:54 2007]--db_more--
} [Mon Feb 19 22:55:01 2007]db{0}> bt
} [Mon Feb 19 22:55:08 2007]cpu_Debugger(c5145200,0,cea02f38,c0334f70,c5145260) at netbsd:cpu_Debugger+0x4
} [Mon Feb 19 22:55:08 2007]comintr(c5145200,0,cea00010,30,10) at netbsd:comintr+0x6e0
} [Mon Feb 19 22:55:08 2007]Xintr_ioapic_edge4() at netbsd:Xintr_ioapic_edge4+0x9c
} [Mon Feb 19 22:55:08 2007]--- interrupt ---
} [Mon Feb 19 22:55:08 2007]netbsd:cpu_switch+0xab:
} [Mon Feb 19 22:55:46 2007]db{0}> machine cpu 1
} [Mon Feb 19 22:55:55 2007]using CPU 1
} [Mon Feb 19 22:55:55 2007]db{0}> bt
} [Mon Feb 19 22:55:58 2007]netbsd:cpu_switch+0xab:
} [Mon Feb 19 22:55:58 2007]db{0}> machine cpu 2
} [Mon Feb 19 22:56:04 2007]using CPU 2
} [Mon Feb 19 22:56:04 2007]db{0}> bt
} [Mon Feb 19 22:56:05 2007]netbsd:cpu_switch+0xab:
} [Mon Feb 19 22:56:05 2007]db{0}> machine cpu 3
} [Mon Feb 19 22:56:08 2007]using CPU 3
} [Mon Feb 19 22:56:08 2007]db{0}> bt
} [Mon Feb 19 22:56:09 2007]netbsd:cpu_switch+0xab:
} [Mon Feb 19 22:56:09 2007]db{0}> machine cpu 0
} [Mon Feb 19 22:56:14 2007]using CPU 0
} [Mon Feb 19 22:56:14 2007]db{0}> bt
} [Mon Feb 19 22:56:17 2007]cpu_Debugger(c5145200,0,cea02f38,c0334f70,c5145260) at netbsd:cpu_Debugger+0x4
} [Mon Feb 19 22:56:17 2007]comintr(c5145200,0,cea00010,30,10) at netbsd:comintr+0x6e0
} [Mon Feb 19 22:56:17 2007]Xintr_ioapic_edge4() at netbsd:Xintr_ioapic_edge4+0x9c
} [Mon Feb 19 22:56:17 2007]--- interrupt ---
} [Mon Feb 19 22:56:17 2007]netbsd:cpu_switch+0xab:
} [Mon Feb 19 22:56:17 2007]db{0}> cont
} [Mon Feb 19 22:57:03 2007]
} [Mon Feb 19 22:57:08 2007]
} [Mon Feb 19 22:57:08 2007]NetBSD/i386 (once) (constty)
} [Mon Feb 19 22:57:08 2007]
} [Mon Feb 19 22:57:08 2007]login:
}
}
} It looks more and more that the big culprit is the extreme age and
} bit-rot in the aac(4) driver is at fault. There have been many
} bug-fixes, improvements, and updates to the original FreeBSD sources
} since it was ported to NetBSD, including this interesting one with a
} possible fix and explanation for lost interrupts which would explain why
} the I/O requests pending at the time of the hang never complete until
} another I/O is started.
}
} Unfortunately this fix was done after the driver's interrupt handling
} was threaded:
}
} ----------------------------
} revision 1.46
} date: 2003/01/11 01:59:21; author: scottl; state: Exp; lines: +50 -39
} Major bugfixes for large memory and fast systems.
}
} aac.c:
} Re-arrange the interrupt handler to optimize the common case of
} the adapter interrupting us because one or more commands are complete,
} and do a read across the pci bus to ensure that all posted status
} writes are flushed. This should close a race that could cause command
} completion interrupts to be lost.
} Follow the spec a bit closer when filling out command structures.
} Enable the Fast Response feature to eliminate the need for the card
} to DMA successfull command completions back into host memory.
} Tell the controller how much physical memory we have. Without this
} there was a chance that our DMA regions would collide with the
} memory window used by the cache on the controller. The result would
} be massive data corruption. This seemed to mainly affect systems with
} >2GB of memory.
} Fix a few whitespace problems.
}
} aac_debug.c:
} Add an extra diagnostic when printing out commands.
}
} aac_disk.c:
} Add extra sanity checks.
}
} aacreg.h:
} Prepare for making this 64-bit clean by reducing the use of enumeration
} types in structures.
}
} Many thanks to Justin Gibbs for helping track these down.
} ----------------------------
}
}
} Re-porting the driver from FreeBSD seems quite daunting, given the
} major divergences growing between FreeBSD and NetBSD, particularly in
} storage device drivers.
}
} However much/all of the good work put into the FreeBSD driver has been
} picked up by OpenBSD in their re-write of the aac(4) driver. It would
} seem on first glance to be much easier to port the new OpenBSD driver
} back over to NetBSD. Unfortunately I don't have time to do this all on
} my own at the moment, though I would be able to help test, at least on
} the netbsd-4 branch, as I have several HP NetRAID cards as well as this
} Dell PE2650 with the embedded PERC controller.
}
} At minimum it sure would be nice if someone could figure some way to fix
} the race condition that seems to be causing interrupts to be lost.
}
} Should I open a PR with the info in this and the last e-mail?
}
} --
} Greg A. Woods
} Planix, Inc.
}
} <woods@planix.ca> +1 416 489-5852 x122 http://www.planix.com/
>-- End of excerpt from "Greg A. Woods"