Re: panic in netbsd-5 (out of space in kmem_map)

[Andrew Doran <ad%netbsd.org@localhost>]

On Thu, Dec 04, 2008 at 10:42:00PM +0100, Manuel Bouyer wrote:

> On Thu, Dec 04, 2008 at 08:49:46AM -0800, Jeff Rizzo wrote:
> > I got a familiar-seeming panic in a netbsd-5 system that was recently
> > upgraded from 4.99.72.   It's an amd64 box, if that matters:
> > 
> > panic: malloc: out of space in kmem_map
> > fatal breakpoint trap in supervisor mode
> > trap type 1 code 0 rip ffffffff804ea125 cs 8 rflags 246 cr2 
> > 7f7ffdb13064 cpl 0 rsp ffff80009fff9750
> > Stopped in pid 13828.1 (raidctl) at     netbsd:breakpoint+0x5:  leave
> > db{0}> bt 
> > breakpoint() at netbsd:breakpoint+0x5
> > panic() at netbsd:panic+0x260
> > malloc() at netbsd:malloc+0x3a7
> > raidioctl() at netbsd:raidioctl+0x478
> > cdev_ioctl() at netbsd:cdev_ioctl+0x91
> > VOP_IOCTL() at netbsd:VOP_IOCTL+0x6e
> > vn_ioctl() at netbsd:vn_ioctl+0x6d
> > sys_ioctl() at netbsd:sys_ioctl+0x134
> > syscall() at netbsd:syscall+0xb6
> > db{0}>
> > 
> > This is netbsd-5 from a couple weeks ago - does this look familiar to
> > anyone?  It happened (I assume, due to timing) during the daily
> > housekeeping (from /etc/daily).  Anything else I should be getting from
> > ddb next time?
> > 
> > 
> > Here's a dmesg in case it helps:
> > booting hd0a:netbsd - starting in 0
> > 9819544+478496+781792 [702096+455719]=0xcacd78
> > Copyright (c) 1996, 1997, 1998, 1999, 2000, 2001, 2002, 2003, 2004, 2005,
> >     2006, 2007, 2008
> >     The NetBSD Foundation, Inc.  All rights reserved.
> > Copyright (c) 1982, 1986, 1989, 1991, 1993
> >     The Regents of the University of California.  All rights reserved.
> > 
> > NetBSD 5.0_BETA (GENERIC) #1: Tue Nov 18 11:59:20 PST 2008
> >    
> > riz%colossus.corp.revision3.com@localhost:/home/riz/netbsd/src5/sys/arch/amd64/compile/obj.amd64/GENERIC
> > total memory = 16382 MB
> > avail memory = 15869 MB
> 
> You have too much RAM in this system :) More specifically, when the
> file cache grows, it eats kmem_map entries, and the default computed
> size is likely too low.
> See
> options NKMEMPAGES
> options NKMEMPAGES_MIN
> options NKMEMPAGES_MAX
> in options(4). sysctl vm.nkmempages will show you the current value (I'm
> not sure it's writable, unfortunably).
> You may also be able to patch nkmempages with gdb without rebuilding a
> new kernel.

This is suspicious because not a whole lot comes from kmem_map any more. The
only vfs cacheing bits allocated from it should be 'struct buf', but not the
buffer contents, which now come from kernel_map. Can you try enabling malloc
stats in the kernel, make it crash again, 'call dumpsys' in ddb and and post
output from 'vmstat -m -M netbsd.X.core -N netbsd.X'?

Andrew