Subject: Re: very strange "temporary" filesytem lockups on netbsd-4 (i386)
To: NetBSD Kernel Technical Discussion List <tech-kern@NetBSD.org>
From: Brian Buhrow <buhrow@lothlorien.nfbcal.org>
List: netbsd-help
Date: 02/20/2007 12:06:05
	You know, this reminds me of when I worked on the twe(4) driver some
years ago.  We had a problem where we'd issue a command to the raid card,
and, sometimes, the card would just black hole it.  The driver would wait
for the card to respond, and it never would.  At the time, I wrote a patch
which implemented a watchdog timer for the raid card, and if we didn't see
any activity after we issued a command for a couple of seconds, we'd kick
the card, and that would usually get things going again until the next time
it happened.  My guess is that the version of the aac(4) driver you're
using isn't using a watchdog type mechanism to make sure the card behaves.
Your work around seems to confirm this theory since issuing another i/o
request to the card gets it going again.  I just looked at the aac(4)
driver in 3.0, and it doesn't appear to implement  any sort of watchdog
timer mechanism.  Even more interesting, if the card generates an interrupt
the driver doesn't recognize, it clears the interrupt, and does nothing. 
One of the interrupts it appears not to recognize, is an incomplete response.  So,
if the card is generating interrupts that are considered invalid by the
driver, they're not logged or acknowledged at the upper levels.  They're
just cleared.  I'd say there's a good  chance this is happening to you for
some reason.  Putting some instrumentation in /usr/src/sys/dev/ic/aac.c
should shed some light on this problem.

Hope that helps.
-Brian
On Feb 20,  2:44pm, "Greg A. Woods" wrote:
} Subject: Re: very strange "temporary" filesytem lockups on netbsd-4 (i386)
} OK, here's what's been happening since my first report....
} 
} I built a new kernel with the recent changes on netbsd-4 done to fix
} many deadlock problems in NetBSD's nullfs and unionfs virtual
} filesystems, hoping without much expectation that this would change the
} behaviour of the temporary freezes I was seeing.  Indeed there were no
} improvements and no change in behaviour either.
} 
} I've worked around the problem for now with the following line added to
} the bottom of /etc/rc.local:
} 
} 	nohup sh -c 'while : ; do dd if=/dev/rld0d of=/dev/null count=1; sleep 1; done' &
} 
} This does an IO on the stuck device once per second, restarting it, and
} it seems to keep everything moving along reasonably smoothly.
} 
} I did capture some not-terribly-interesting kernel info from DDB though
} and all you basically see is "ld" stuck in getblk state:
} 
} [Mon Feb 19 22:50:28 2007]login: [-- break #1 sent -- `\z' -- Mon Feb 19 22:54:51 2007]
} Stopped at      netbsd:cpu_Debugger+0x4:        popl    %ebp
} [Mon Feb 19 22:54:51 2007]db{0}> ps
} [Mon Feb 19 22:54:53 2007] PID           PPID     PGRP        UID S   FLAGS LWPS          COMMAND    WAIT
} [Mon Feb 19 22:54:53 2007] 3920         18420    10164       1000 2  0x4002    1               ld  getblk
} [Mon Feb 19 22:54:53 2007] 18420        16402    10164       1000 2  0x4002    1         collect2    wait
} [Mon Feb 19 22:54:53 2007] 16402         3137    10164       1000 2  0x4002    1 i386--netbsdelf-    wait
} [Mon Feb 19 22:54:53 2007] 3137          9416    10164       1000 2  0x4002    1               sh    wait
} [Mon Feb 19 22:54:53 2007] 9416         13265    10164       1000 2  0x4002    1           nbmake    wait
} [Mon Feb 19 22:54:53 2007] 13265         3034    10164       1000 2  0x4002    1               sh    wait
} [Mon Feb 19 22:54:53 2007] 3034         15061    10164       1000 2  0x4002    1           nbmake    wait
} [Mon Feb 19 22:54:53 2007] 15061         2593    10164       1000 2  0x4002    1               sh    wait
} [Mon Feb 19 22:54:53 2007] 2593         27295    10164       1000 2  0x4002    1           nbmake    wait
} [Mon Feb 19 22:54:53 2007] 27295        15819    10164       1000 2  0x4002    1               sh    wait
} [Mon Feb 19 22:54:53 2007] 18596         6309    18596       1000 2  0x4002    1             tail  kqread
} [Mon Feb 19 22:54:53 2007] 6309         20011     6309       1000 2  0x4002    1              ksh   pause
} [Mon Feb 19 22:54:53 2007] 20011          763      763          0 2  0x4100    1          rlogind    poll
} [Mon Feb 19 22:54:53 2007] 15819        10280    10164       1000 2  0x4002    1           nbmake    wait
} [Mon Feb 19 22:54:53 2007] 10280        28521    10164       1000 2  0x4002    1               sh    wait
} [Mon Feb 19 22:54:53 2007] 28521        27102    10164       1000 2  0x4002    1           nbmake    wait
} [Mon Feb 19 22:54:53 2007] 27102         2473    10164       1000 2  0x4002    1               sh    wait
} [Mon Feb 19 22:54:53 2007] 12119          621    12119       1000 2  0x4102    1           systat   ttyin
} [Mon Feb 19 22:54:53 2007] 2473          9639    10164       1000 2  0x4002    1           nbmake    wait
} [Mon Feb 19 22:54:53 2007] 9639         10936    10164       1000 2  0x4002    1               sh    wait
} [Mon Feb 19 22:54:53 2007] 10936        10384    10164       1000 2  0x4002    1           nbmake    wait
} [Mon Feb 19 22:54:53 2007] 10384        10164    10164       1000 2  0x4002    1               sh    wait
} [Mon Feb 19 22:54:53 2007] 10164          621    10164       1000 2  0x4002    1              ksh   pause
} [Mon Feb 19 22:54:53 2007] 621            620      621       1000 2  0x4003    1              ksh   pause
} [Mon Feb 19 22:54:54 2007] 620            763      763          0 2  0x4100    1          rlogind    poll
} [Mon Feb 19 22:54:54 2007] 695              1      695          0 2  0x4002    1            getty   ttyin
} [Mon Feb 19 22:54:54 2007] 826              1      826          0 2  0x4002    1            getty   ttyin
} [Mon Feb 19 22:54:54 2007] 782              1      782          0 2  0x4002    1            getty   ttyin
} [Mon Feb 19 22:54:54 2007]--db_more--
} [Mon Feb 19 22:55:01 2007]db{0}> bt
} [Mon Feb 19 22:55:08 2007]cpu_Debugger(c5145200,0,cea02f38,c0334f70,c5145260) at netbsd:cpu_Debugger+0x4
} [Mon Feb 19 22:55:08 2007]comintr(c5145200,0,cea00010,30,10) at netbsd:comintr+0x6e0
} [Mon Feb 19 22:55:08 2007]Xintr_ioapic_edge4() at netbsd:Xintr_ioapic_edge4+0x9c
} [Mon Feb 19 22:55:08 2007]--- interrupt ---
} [Mon Feb 19 22:55:08 2007]netbsd:cpu_switch+0xab:
} [Mon Feb 19 22:55:46 2007]db{0}> machine cpu 1
} [Mon Feb 19 22:55:55 2007]using CPU 1
} [Mon Feb 19 22:55:55 2007]db{0}> bt
} [Mon Feb 19 22:55:58 2007]netbsd:cpu_switch+0xab:
} [Mon Feb 19 22:55:58 2007]db{0}> machine cpu 2
} [Mon Feb 19 22:56:04 2007]using CPU 2
} [Mon Feb 19 22:56:04 2007]db{0}> bt
} [Mon Feb 19 22:56:05 2007]netbsd:cpu_switch+0xab:
} [Mon Feb 19 22:56:05 2007]db{0}> machine cpu 3
} [Mon Feb 19 22:56:08 2007]using CPU 3
} [Mon Feb 19 22:56:08 2007]db{0}> bt
} [Mon Feb 19 22:56:09 2007]netbsd:cpu_switch+0xab:
} [Mon Feb 19 22:56:09 2007]db{0}> machine cpu 0
} [Mon Feb 19 22:56:14 2007]using CPU 0
} [Mon Feb 19 22:56:14 2007]db{0}> bt
} [Mon Feb 19 22:56:17 2007]cpu_Debugger(c5145200,0,cea02f38,c0334f70,c5145260) at netbsd:cpu_Debugger+0x4
} [Mon Feb 19 22:56:17 2007]comintr(c5145200,0,cea00010,30,10) at netbsd:comintr+0x6e0
} [Mon Feb 19 22:56:17 2007]Xintr_ioapic_edge4() at netbsd:Xintr_ioapic_edge4+0x9c
} [Mon Feb 19 22:56:17 2007]--- interrupt ---
} [Mon Feb 19 22:56:17 2007]netbsd:cpu_switch+0xab:
} [Mon Feb 19 22:56:17 2007]db{0}> cont
} [Mon Feb 19 22:57:03 2007]
} [Mon Feb 19 22:57:08 2007]
} [Mon Feb 19 22:57:08 2007]NetBSD/i386 (once) (constty)
} [Mon Feb 19 22:57:08 2007]
} [Mon Feb 19 22:57:08 2007]login: 
} 
} 
} It looks more and more that the big culprit is the extreme age and
} bit-rot in the aac(4) driver is at fault.  There have been many
} bug-fixes, improvements, and updates to the original FreeBSD sources
} since it was ported to NetBSD, including this interesting one with a
} possible fix and explanation for lost interrupts which would explain why
} the I/O requests pending at the time of the hang never complete until
} another I/O is started.
} 
} Unfortunately this fix was done after the driver's interrupt handling
} was threaded:
} 
} ----------------------------
} revision 1.46
} date: 2003/01/11 01:59:21;  author: scottl;  state: Exp;  lines: +50 -39
} Major bugfixes for large memory and fast systems.
} 
} aac.c:
}         Re-arrange the interrupt handler to optimize the common case of
}         the adapter interrupting us because one or more commands are complete,
}         and do a read across the pci bus to ensure that all posted status
}         writes are flushed.  This should close a race that could cause command
}         completion interrupts to be lost.
}         Follow the spec a bit closer when filling out command structures.
}         Enable the Fast Response feature to eliminate the need for the card
}         to DMA successfull command completions back into host memory.
}         Tell the controller how much physical memory we have.  Without this
}         there was a chance that our DMA regions would collide with the
}         memory window used by the cache on the controller.  The result would
}         be massive data corruption.  This seemed to mainly affect systems with
}         >2GB of memory.
}         Fix a few whitespace problems.
} 
} aac_debug.c:
}         Add an extra diagnostic when printing out commands.
} 
} aac_disk.c:
}         Add extra sanity checks.
} 
} aacreg.h:
}         Prepare for making this 64-bit clean by reducing the use of enumeration
}         types in structures.
} 
} Many thanks to Justin Gibbs for helping track these down.
} ----------------------------
} 
} 
} Re-porting the driver from FreeBSD seems quite daunting, given the
} major divergences growing between FreeBSD and NetBSD, particularly in
} storage device drivers.
} 
} However much/all of the good work put into the FreeBSD driver has been
} picked up by OpenBSD in their re-write of the aac(4) driver.  It would
} seem on first glance to be much easier to port the new OpenBSD driver
} back over to NetBSD.  Unfortunately I don't have time to do this all on
} my own at the moment, though I would be able to help test, at least on
} the netbsd-4 branch, as I have several HP NetRAID cards as well as this
} Dell PE2650 with the embedded PERC controller.
} 
} At minimum it sure would be nice if someone could figure some way to fix
} the race condition that seems to be causing interrupts to be lost.
} 
} Should I open a PR with the info in this and the last e-mail?
} 
} -- 
} 						Greg A. Woods
} 						Planix, Inc.
} 
} <woods@planix.ca>     +1 416 489-5852 x122      http://www.planix.com/
>-- End of excerpt from "Greg A. Woods"