boot fail with large RAM

[Manuel Bouyer <bouyer%antioche.eu.org@localhost>]

Hi,
today I tried to boot a recent netbsd-5 on a server with 48GB RAM.
This fails when devices tries to allocate bus_dma(9) memory in
the lower 4GB RAM.
Here's the boot message (with some debug code turned on):

> boot netbsd64 -s
booting hd0a:netbsd64 (howto 0x2)
10584312+572992+823424 [731328+479128]=0xd95f18
kernel text is mapped with 6 large pages and 25 normal pages
BIOS MEMORY MAP (8 ENTRIES):
    addr 0x0  size 0xa0000  type 0x1
    addr 0x100000  size 0x6f599000  type 0x1
    addr 0x6f699000  size 0x16000  type 0x2
    addr 0x6f6af000  size 0x1f000  type 0x3
    addr 0x6f6ce000  size 0x932000  type 0x2
    addr 0xe0000000  size 0x10000000  type 0x2
    addr 0xfe000000  size 0x2000000  type 0x2
    addr 0x100000000  size 0xb88000000  type 0x1
loading 0xe000-0xa0000 (0xe-0xa0)
loading 0xf09000-0x1000000 (0xf09-0x1000)
loading 0x1000000-0x6f699000 (0x1000-0x6f699)
loading 0x100000000-0xc88000000 (0x100000-0xc88000)
Loaded initial symtab at 0xffffffff80d6dec0, strtab at 0xffffffff80e20f80, # en0
Copyright (c) 1996, 1997, 1998, 1999, 2000, 2001, 2002, 2003, 2004, 2005,
    2006, 2007, 2008, 2009
    The NetBSD Foundation, Inc.  All rights reserved.
Copyright (c) 1982, 1986, 1989, 1991, 1993
    The Regents of the University of California.  All rights reserved.

NetBSD 5.0_STABLE (GENERIC) #0: Wed Nov 18 19:04:59 CET 2009
        bouyer@roll:/dsk/l1/misc/bouyer/tmp/amd64/obj/dsk/l1/misc/bouyer/netbsdC
total memory = 49014 MB
avail memory = 47509 MB
SMBIOS rev. 2.6 @ 0x6f79c000 (83 entries)
Dell Inc. PowerEdge R710
mainbus0 (root)

As you can see this system has an unusual memory map, in the
sense that there's only 1782MB availaable in the 32bit range (we're
used to see about 3GB in this range for large-memory machines).
This is confirmed by a 32bit NetBSD kernel, and a 32bit Xen kernel.

My guess is that the kernel eats up all the memory below 4GB for its
internal structures and there's no more low-memory free when PCI
discovery is started. But I didn't find yet where that much memory is
allocated.

I also noticed that amd64 has only 2 free lists, one for the low 16MB
memory and one for anything else. Would it help to move to a third
free list; one for low 16MB, a second for the low 4GB and a third one for
anything else ? uvm(9) doens't say much about free lists ...

-- 
Manuel Bouyer, LIP6, Universite Paris VI.           
Manuel.Bouyer%lip6.fr@localhost
     NetBSD: 26 ans d'experience feront toujours la difference
--