>Actually, the biggest problem is that upping max-users (given the way
>various things in the pmap works) causes a _GIGANTIC_ amount of kernel
>virtual space to be used...
>
>as i recall (i looked at it a while ago), it increases linearly with
>maxusers, and with maxusers == 16 it's ~1G and maxusers == 32 it's
>~2G.  (something like that; i know that i was running into 'int'
>signedness problems, and i think it was while trying maxusers == 32.)
>
>the kernel only allows 8G of KV-space, max, and i've not tried to boot
>with anything larger than '32'.  as i recall, '32' worked, though.

Hmm... yes, I think I see.  From what we (the CCed gentleman and I) can 
tell, the crux of the issue is the following code in 
src/alpha/alpha/pmap.c:

	/*
	 * Figure out how many PTE's are necessary to map the kernel.
	 * The '512' comes from PAGER_MAP_SIZE in vm_pager_init().
	 * This should be kept in sync.
	 * We also reserve space for kmem_alloc_pageable() for vm_fork().
	 */
	Sysmapsize = (VM_KMEM_SIZE + VM_MBUF_SIZE + VM_PHYS_SIZE +
	       	nbuf * MAXBSIZE + 16 * NCARGS) / NBPG + 512 + 256;
 Sysmapsize += maxproc * (btoc(ALPHA_STSIZE) + btoc(ALPHA_MAX_PTSIZE));

and later...

	/*
	 * Allocate the segment table map
	 */
	s = maxproc * ALPHA_STSIZE;
	st_map = kmem_suballoc(kernel_map, &addr, &addr2, s, TRUE);

	/*
	 * Allocate the page table map
	 */
	s = maxproc * ALPHA_MAX_PTSIZE;			/* XXX limit it */
	pt_map = kmem_suballoc(kernel_map, &addr, &addr2, s, TRUE);

With a MAXUSERS of 256, maxproc ends up being 4116 (per conf/param.c), 
and with ALPHA_MAX_PTSIZE being 8MB, kmem_suballoc() is trying to map out 
about 32GB of memory.  Yikes, I don't even have a tenth of that in swap 
space :-)  It appears that the calculation is worst case -- map out the 
maximum amount of page table info per maximum number of processes.  
However, given that the result is far beyond actual resources, it would 
seem that the calculation could be made more lifelike.  So the question 
is: does it make sense to do something like the following instead?

#define SWAP_MULT 8 /* max ratio of swap to real memory */
                    /* or "options SWAP_MULT=8" in the config file */

 Sysmapsize += ((totalphysmem * SWAP_MULT) >> (PGSHIFT-PTESHIFT))
            + (totalphysmem * SWAP_MULT);
 
	s = (ctob(totalphysmem) * SWAP_MULT) >> (PGSHIFT-PTESHIFT);
	st_map = kmem_suballoc(kernel_map, &addr, &addr2, s, TRUE);

 s = ctob(totalphysmem) * SWAP_MULT;
 pt_map = kmem_suballoc(kernel_map, &addr, &addr2, s, TRUE);

That way, assuming that SWAP_MULT is set correctly, I think we're 
guaranteed to allocate space for at least as many PT entries as we have 
pages to table, but not drastically more so, and can easily configure for 
a better fit.

Thoughts?  We've tested a hardcoded version of this which seems to boot 
successfully with a MAXUSERS of 250, so treat the stuff above as a 
pseudocode attempt to generalize it.

dp