While trying to tune a kernel for an old 1.3.3/i386 machine, that while
very near the end of it's planned production life, was in dire need of
more capacity without wiggling too much, I too encountered this problem.

Unfortunately the answer suggested in this still-open PR is very
sub-optimal from a performance perspective.

Here are the notes from my final working kernel configuration showing
how I arrived at a much much better answer.

Note that text resembling the first paragraph from the second section
should probably be edited into options(4) before this PR is closed as
it's still apparently relevant with UVM.  The rest of course is for
historical purposes and in case anyone else is still running a pre-UVM
machine that might need tuning to handle more processes, mbufs, etc.

FYI I also discovered coincidentally that "netstat -m" only produces
meaningful output if "options KMEMSTATS" is enabled too....

# note CLBYTES is 4096 on i386 so 128MB of RAM is 32768 MBCLUSTERS.
options 	NMBCLUSTERS="(128*1024*1024/CLBYTES)" # get ready to network!

# default NKMEMCLUSTERS is (6 * 1024 * 1024 / CLBYTES)
#
# for ages on a 192MB machine we had 24MB's worth, which was always
# enough, but on a 320MB machine with relatively small process
# footprints it's not nearly enough to make full use of the system.
# vm_fork panics from a failed kmem_alloc() as soon as it's under any
# realistic load.
#
# note CLBYTES is 4096 so 64MB of RAM is 16384 KMEMCLUSTERS.
# 
# increase this if you get any kmem_alloc panics (eg. from vm_fork)
# (or decrease MAXUSERS and/or MAXPROC), but watch for clashes in the
# maximum available kernel memory between what NBUF requires and what
# NMBCLUSTERS requires.
# 
options 	NKMEMCLUSTERS="(48*1024*1024/CLBYTES)"

# Static buffer cache is defined by two kernel parameters:  NBUF and
# BUFPAGES.
# 
# NBUF defines the number of buffer headers, i.e. the
# number of files that may have a buffer cache entry.  Each buffer
# header includes space for an I/O buffer of MAXBSIZE (which defaults
# to 65536) so take care not to allocate too many since this space
# must be shared with other kernel memory, including that used by
# KMEMCLUSTERS and NMBCLUSTERS and on i386 there is a limit.
# 
# The size of the actual buffer data cache is defined by the kernel
# parameter bufpages.  This size is given in 4kB pages (CLBYTES).
# 
# to fix "panic: kmem_suballoc: bad status return of 3" it was
# suggested in PR#2998 that we adjust MAXBSIZE to 16384, but that's
# 1/4 of the default value of 64K and so it's not the best thing to
# do if you want any performance from the machine....
# 
# The real problem causing the panic is that nbuf gets too big when
# auto-calculated on large machines (and once KMEMCLUSTERS is
# increased enough there's no room left for the buffer map).  So the
# solution is to lock down NBUF to a reasonable value.
# 
# Nbuf was auto-sized to 2822 on a machine with 320MB RAM, which is
# just a bit over 16M for buffers when nbuf==bufpages.  However before
# with only 192MB RAM NBUF was just 1659, which is just under 6.5MB
# for buffers when nbuf==bufpages.  This worked OK with NKMEMCLUSTERS
# sized for 24MB.
# 
# However with NKMEMCLUSTERS up at 48MB we have to keep NBUF lower
# than 2000 so that the buffer_map (MAXBSIZE*nbuf) is less than the
# remaining KMEM space available for it.  However with lots of RAM we
# can increase BUFPAGES to keep in line with the 10% factor.
# 
options 	NBUF=1800				# buffer headers
options 	BUFPAGES="(32*1024*1024/CLBYTES)"	# 10% of RAM

-- 
							Greg A. Woods

+1 416 218-0098      VE3TCP      <gwoods@acm.org>     <woods@robohack.ca>
Planix, Inc. <woods@planix.com>;   Secrets of the Weird <woods@weird.com>