One other possibility strikes me: rigging the machines for serial
console, and putting the consoles onto something that will save
everything that spills out the RS-232...

Christos Zoulas writes:
> On Oct 9,  1:09pm, R.S.Brooks@liverpool.ac.uk (Roger Brooks) wrote:
> -- Subject: Re: Kernal panic woes
> 
> | On Fri, 9 Oct 1998, Christos Zoulas wrote:
> | 
> | >In article <Pine.SOL.3.96.981009121304.23059A-100000@uxa.liv.ac.uk> Dave.T
yson@liverpool.ac.uk (Dave Tyson) writes:
> | >
> | >>panic: trap
> | >>#0  0x6 in ?? ()
> | >>(gdb) where
> | >>#0  0x6 in ?? ()
> | >>#1  0xf01e5d3b in cpu_reboot ()
> | >>#2  0xf013a949 in panic ()
> | >>#3  0xf01ec5d6 in trap ()
> | >>(gdb) q
> | >>
> | >>I have had a quick look at the source code around this this area, but it
> | >>is not very enlightening. Any suggestions as to how to pin down this 
> | >>problem would be appreciated. 
> | >
> | >
> | >You are dying because of a panic, which is good. Unfortunately you are
> | >rebooting immediately after a panic, so that you are not seeing the panic
> | >message. In addition, your stack is trashed, so you don't see where panic
> | >is being called from. I suggest that you compile a kernel with ddb, and
> | >when it panic's you'll see the message in the console, and you'll be able
> | >to print a complete backtrace. My suspicion is that you are running out
> | >of some resource and you need to boost some parameter.
> | 
> | The problem is these machines are in user service, providing our site's
> | web cache.  Having the system drop into ddb on panic is fine during the
> | day when there's someone here to notice it, but if it happens overnight
> | or at weekend we want the machine to reboot without intervention.  Is
> | there any way we can build a kernel with ddb and then toggle the behaviour
> | on panic (enter ddb or reboot) on the running system?  We can't really
> | build two kernels (with and without ddb) and reboot twice daily!
> 
> Yes, man sysctl will tell you :-)
> 
> In the morning you can:
> 
> 	sysctl -w ddb.onpanic=1
> 
> and just before you leave you can:
> 
> 	sysctl -w ddb.onpanic=0
> 
> Of course, the problem is that you'll enter ddb if you crash for another
> reason (not a panic, but an bus error or segmentation fault in the kernel
> for example). 
> 
> So I suggest that you do this on both machines, hopefully you'll get 
> a panic during the day, then you can go back to running kernels without
> ddb. Maybe we should add an option so that you enter ddb only on panic.
> 
> christos
>