Subject: Attempting to stop lock up when out of memory
To: None <tech-kern@NetBSD.org>
From: Jeremy C. Reed <reed@reedmedia.net>
List: tech-kern
Date: 07/21/2005 15:36:39
(Please carbon copy me on replies.)

For a few years, my i386 system running 1.6.x and then 2.0.2 often locks 
up when I am out of memory. pinging still works for awhile and ls listing 
from nfs works extremely slow. But waiting a lot longer even can't ping 
system. Log files show nothing.

It would be nice if there were some sane defaults for memory resource 
limits and also if the system could be kept available.

My system has:

   total memory = 127 MB
   avail memory = 117 MB

And it has 192 MB of swap.

The following are all defaults.

As a regular user my limits are:

proc.curproc.rlimit.datasize.soft = 134217728   (128 MB)
proc.curproc.rlimit.datasize.hard = 1073741824  (1 GB)
proc.curproc.rlimit.stacksize.soft = 2097152    (2 MB)
proc.curproc.rlimit.stacksize.hard = 33554432   (32 MB)
proc.curproc.rlimit.memoryuse.soft = 122421248  (116 MB)
proc.curproc.rlimit.memoryuse.hard = 122421248
proc.curproc.rlimit.memorylocked.soft = 40807082 (38 MB)
proc.curproc.rlimit.memorylocked.hard = 122421248 (116 MB)

Any my shell tells me my same limits:

data seg size         (kbytes, -d) 131072   -- soft
data seg size         (kbytes, -d) 1048576
max locked memory     (kbytes, -l) 39850    -- soft
max locked memory     (kbytes, -l) 119552
max memory size       (kbytes, -m) 119552   -- soft
max memory size       (kbytes, -m) 119552
stack size            (kbytes, -s) 2048     -- soft
stack size            (kbytes, -s) 32768
virtual memory        (kbytes, -v) 133120   -- soft
virtual memory        (kbytes, -v) 1081344    (1056 MB)

As root these limits were identical.

On a second system with:
   total memory = 93820 KB
   avail memory = 84012 KB
with a 25704 KBytes swap, I have the following:

proc.curproc.rlimit.datasize.soft = 134217728
proc.curproc.rlimit.datasize.hard = 1073741824
proc.curproc.rlimit.stacksize.soft = 2097152
proc.curproc.rlimit.stacksize.hard = 33554432
proc.curproc.rlimit.memoryuse.soft = 85405696  (81 MB)
proc.curproc.rlimit.memoryuse.hard = 85405696
proc.curproc.rlimit.memorylocked.soft = 28468565  (27 MB)
proc.curproc.rlimit.memorylocked.hard = 85405696

data(kbytes)         131072
stack(kbytes)        2048
lockedmem(kbytes)    27801
memory(kbytes)       83404


I didn't know what some of this meant, but I found that login.conf 
describes memorylocked as "Maximum locked in core memory size limit" and 
memoryuse as "Maximum in core memoryuse size limit". csh documents 
memoryuse as "the maximum size (in bytes) to which a process's resident 
set size (RSS) may grow" and memorylocked as "The maximum size (in bytes) 
which a process may lock into memory using the mlock(2) function."

What can be done so the default out-of-the-box configurations won't allow 
the system to become locked up or unavailable when out of memory?

It seems like default syslog.conf is configured fine, but I don't see any 
messages logged from kernel when running out of memory. I am running X, so 
don't see on console either. And maybe syslogd is already dead -- I can't 
tell. And I don't see any "killed: out of swap" either.

(Yes, I know I need more swap and/or more physical memory, but I am hoping 
it not become unavailable when I hit these problems.)

Maybe sys/arch/i386/i386/trap.c (or for other platforms) can have the 
kernel tsleep for a moment and retest for the ENOMEM before it kills the 
process when out of swap. Any thoughts about that?

Or maybe it can be made to kill off latest processes first?

But even the "out of swap" does not explain to me why pinging finally 
stops working too. Any ideas?

(Please CC me on replies.)

  Jeremy C. Reed

  	  	 	 BSD News, BSD tutorials, BSD links
 	  	 	 http://www.bsdnewsletter.com/