On Thu, Dec 01, 2005 at 11:25:31AM -0600, John D. Baker wrote:
> I've been pondering this problem since I've now been bitten by it more
> than a few times, and others have recently run up against it for the
> first time and are voicing some confusion over it.

Count me as another one.  I ran into panics, hangs, and scsi bus errors
repeatedly on an SS20 w/512MB, 2 original CPUs (70MHz?) such that I could
not keep the system running longer than about 24 hours.  After increasing
NKMEMPAGES to double the standard value, I have THUS FAR been able to run
for over 53 hours and running.  Before reading about this issue on this
list, I had thought my SCSI disks/bus were having issues, or that NetBSD
simply wasn't up to the task for running in multiprocessor mode.

> I'd like to get it straight in my mind just what the issue is.  It seems
> to be this:  kmem_map is a block of descriptors that malloc() and friends
> use to point to chunks of memory that get allocated.  The problem comes
> about on large-memory systems where memory is available, but there are
> no more descriptors to point to it and malloc() fails.
> 
> Is this an accurate description?
> 
> I recall reading that the size of kmem_map, as set by
> NKMEMPAGES{,_MIN,_MAX}{,_DEFAULT} is supposed to be only a starting point
> and that some dynamic sizing is supposed to take place.  But this seems
> not to be the case, at least on sparc.  Kernels that I've built with
> NKMEMPAGES_MAX increased to any value always result in the maximum size
> in vm.nkmempages.
> 
> In the most pessimistic case, you'd need one map entry (descriptor) for
> each page of allocable memory.  I believe the formula for that has been
> mentioned here in the past.
> 
> I recall reading something indicating that NKMEMPAGES is kept small in
> the GENERIC kernels because making it too big prevents low-memory systems
> from booting a kernel.  I'm a bit confused by this.  Is kmem_map a fixed-
> size structure in the kernel image itself, or can it not be set up at
> kernel initialization time, sized appropriately for the available memory?
> 
> Is there any chance of implementing a more graceful failure mode than
> a panic?  Can a lack of descriptors be treated similarly to a lack of
> physical pages and trigger paging to virtual memory (with a message to
> the console/syslog to indicate that a kernel with a bigger kmem_map
> should be built)?  Or does paging/swapping also depend on descriptors
> to map their pages too?  Might there be a way to dynamically extend
> kmem_map (the last available descriptor points to another map block)?
> 
> I'm curious about other ports having problems like this.  I can't really
> investigate directly as my SPARC systems are the only ones with sufficient
> memory to exhibit the problem.
> 
> -- 
> John D. Baker, KN5UKS                    NetBSD     Darwin/MacOS X
> jdbaker(at)mylinuxisp(dot)com                 OpenBSD            FreeBSD
> BSD -- It just sits there and _works_!
> GPG fingerprint:  D703 4A7E 479F 63F8 D3F4  BD99 9572 8F23 E4AD 1645

-- 

Steven F. Siirila			Office: Lind Hall, Room 130B
Internet Services			E-mail: sfs@umn.edu
Office of Information Technology	Voice: (612) 626-0244
University of Minnesota			Fax: (612) 626-7593