Re: Max VM process usage

[Jean-Yves Migeon <jeanyves.migeon%free.fr@localhost>]

On Mon, 21 Jun 2010 19:25:06 +1000, Ray Phillips 
<r.phillips%uq.edu.au@localhost>
wrote:
> I'm running Roaring Penguin's CanIt-PRO on a NetBSD/i386 5.0.2 
> machine (although it isn't a supported platform) which has 2 GB of 
> RAM and 5GB of swap space.  One of it's processes frequently exits 
> unexpectedly.  ktrace has revealed the probable cause:
> 
>    3716      1 perl     CALL  break(0x1e910000)
>    3716      1 perl     RET   break -1 errno 12 Cannot allocate memory
> 
> and the developers have asked "what is the maximum virtual memory 
> size of a process on NetBSD on your architecture?"  Would the correct 
> answer be "there is no limit" based on the following sysctl output, 
> or am I looking in the wrong place?

Nope, correct place:

proc.curproc.rlimit.memoryuse.soft = 2094112768
proc.curproc.rlimit.memoryuse.hard = 2094112768

=> 2GB for resident set size (the program cannot allocate more memory when
its RSS reaches 2GB).

FWIW, for i386, maximum virtual memory space for userland is 3GB (kernel
reserves the upper 1GB).

> It's usual for top to say about 400 MB of swap space are being 
> used--way below the free amount--so maybe I should increase the 
> process' maximum data segment size.

IIUC, if the swapped out data is not demand paged, it will stay in swap.
swapper will not map it back into memory. There is no need to swap data
back in if it is not required.

> Changing subject slightly... I've noticed in top's output on a 
> NetBSD/i386 5.0.2 PC which has 1 GB RAM and 641 MB of swap space 
> which is running squid, that the file cache sometimes uses more than 
> 700 MB of RAM and at the same time more than 300 MB of swap space is 
> used.  I wondered if system performance might be better if less RAM 
> was used by the file cache so as to be able to use less slow swap 
> space.  I guess that's not right though since the memory management 
> routines know what they're doing.

Well, the routines are limited in what they "know". They cannot guess what
you want, or how the system is going to be solicited later :)

If your squid caches lots of file content, then the page cache matters a
lot. If you reduce it, you will cache fewer files, hence, have slower
performance.

Instead, if you only cache a small amount of files, reducing the file
cache size won't affect perfomance much.

>  Could you give me a rough idea of 
> how the kernel (?) decides to balance the competing needs of file 
> caching and swap usage?

Might be of interest:

http://www.selonen.org/arto/netbsd/

-- 
Jean-Yves Migeon
jeanyves.migeon%free.fr@localhost