I've been pondering this problem since I've now been bitten by it more
than a few times, and others have recently run up against it for the
first time and are voicing some confusion over it.

I'd like to get it straight in my mind just what the issue is.  It seems
to be this:  kmem_map is a block of descriptors that malloc() and friends
use to point to chunks of memory that get allocated.  The problem comes
about on large-memory systems where memory is available, but there are
no more descriptors to point to it and malloc() fails.

Is this an accurate description?

I recall reading that the size of kmem_map, as set by
NKMEMPAGES{,_MIN,_MAX}{,_DEFAULT} is supposed to be only a starting point
and that some dynamic sizing is supposed to take place.  But this seems
not to be the case, at least on sparc.  Kernels that I've built with
NKMEMPAGES_MAX increased to any value always result in the maximum size
in vm.nkmempages.

In the most pessimistic case, you'd need one map entry (descriptor) for
each page of allocable memory.  I believe the formula for that has been
mentioned here in the past.

I recall reading something indicating that NKMEMPAGES is kept small in
the GENERIC kernels because making it too big prevents low-memory systems
from booting a kernel.  I'm a bit confused by this.  Is kmem_map a fixed-
size structure in the kernel image itself, or can it not be set up at
kernel initialization time, sized appropriately for the available memory?

Is there any chance of implementing a more graceful failure mode than
a panic?  Can a lack of descriptors be treated similarly to a lack of
physical pages and trigger paging to virtual memory (with a message to
the console/syslog to indicate that a kernel with a bigger kmem_map
should be built)?  Or does paging/swapping also depend on descriptors
to map their pages too?  Might there be a way to dynamically extend
kmem_map (the last available descriptor points to another map block)?

I'm curious about other ports having problems like this.  I can't really
investigate directly as my SPARC systems are the only ones with sufficient
memory to exhibit the problem.

-- 
John D. Baker, KN5UKS                    NetBSD     Darwin/MacOS X
jdbaker(at)mylinuxisp(dot)com                 OpenBSD            FreeBSD
BSD -- It just sits there and _works_!
GPG fingerprint:  D703 4A7E 479F 63F8 D3F4  BD99 9572 8F23 E4AD 1645