Re: boot fail with large RAM

[Christoph Egger <Christoph_Egger%gmx.de@localhost>]

Manuel Bouyer wrote:
> Hi,
> today I tried to boot a recent netbsd-5 on a server with 48GB RAM.
> This fails when devices tries to allocate bus_dma(9) memory in
> the lower 4GB RAM.
> Here's the boot message (with some debug code turned on):
> 
>> boot netbsd64 -s
> booting hd0a:netbsd64 (howto 0x2)
> 10584312+572992+823424 [731328+479128]=0xd95f18
> kernel text is mapped with 6 large pages and 25 normal pages
> BIOS MEMORY MAP (8 ENTRIES):
>     addr 0x0  size 0xa0000  type 0x1
>     addr 0x100000  size 0x6f599000  type 0x1
>     addr 0x6f699000  size 0x16000  type 0x2
>     addr 0x6f6af000  size 0x1f000  type 0x3
>     addr 0x6f6ce000  size 0x932000  type 0x2
>     addr 0xe0000000  size 0x10000000  type 0x2
>     addr 0xfe000000  size 0x2000000  type 0x2
>     addr 0x100000000  size 0xb88000000  type 0x1
> loading 0xe000-0xa0000 (0xe-0xa0)
> loading 0xf09000-0x1000000 (0xf09-0x1000)
> loading 0x1000000-0x6f699000 (0x1000-0x6f699)
> loading 0x100000000-0xc88000000 (0x100000-0xc88000)
> Loaded initial symtab at 0xffffffff80d6dec0, strtab at 0xffffffff80e20f80, # 
> en0
> Copyright (c) 1996, 1997, 1998, 1999, 2000, 2001, 2002, 2003, 2004, 2005,
>     2006, 2007, 2008, 2009
>     The NetBSD Foundation, Inc.  All rights reserved.
> Copyright (c) 1982, 1986, 1989, 1991, 1993
>     The Regents of the University of California.  All rights reserved.
> 
> NetBSD 5.0_STABLE (GENERIC) #0: Wed Nov 18 19:04:59 CET 2009
>         
> bouyer@roll:/dsk/l1/misc/bouyer/tmp/amd64/obj/dsk/l1/misc/bouyer/netbsdC
> total memory = 49014 MB
> avail memory = 47509 MB
> SMBIOS rev. 2.6 @ 0x6f79c000 (83 entries)
> Dell Inc. PowerEdge R710
> mainbus0 (root)
> 
> As you can see this system has an unusual memory map, in the
> sense that there's only 1782MB availaable in the 32bit range (we're
> used to see about 3GB in this range for large-memory machines).
> This is confirmed by a 32bit NetBSD kernel, and a 32bit Xen kernel.
> 
> My guess is that the kernel eats up all the memory below 4GB for its
> internal structures and there's no more low-memory free when PCI
> discovery is started. But I didn't find yet where that much memory is
> allocated.
> 
> I also noticed that amd64 has only 2 free lists, one for the low 16MB
> memory and one for anything else. Would it help to move to a third
> free list; one for low 16MB, a second for the low 4GB and a third one for
> anything else ? uvm(9) doens't say much about free lists ...

Can you dump the memory usages of the various pools we have
in the kernel with 1GB, 2GB, 4GB, 8GB, 16GB, 32GB and 48GB RAM
and plot graphs ?

Christoph