[ On Saturday, November 27, 2004 at 21:55:55 (+0100), theo borm wrote: ]
> Subject: Re: memory (re-)allocation woes
>
> >14:29 [702] $ ./trealloc 1048576 1048576 
> >[[ .... ]]
> >step 342: reallocating 359661568 bytes... trealloc: realloc() failed: Cannot allocate memory
> >  
> >
> I note that the program could not (re-) allocate 342 Mbyte, which
> is about a third of the memory available to user processes:
> 
> data(kbytes)         1048576

Remember that it's trying to realloc() -- i.e. it already has a 341 MB
block allocated and now it's trying to allocate an additional 342 MB
block into which the first block will be copied.  That`s a total of
683MB.  Now in the worst case a simplistic realloc() algorithm is
"stuck" with leaving the previous 340 MB block in place and trying to
extend the heap by the entire 342 MB because the previous 340 MB chunk
is at the start and cannot be extended.

	+--------------------------------------------+
	| 340 MB chunk | 341 MB chunk | 342 MB attempt -- oops! too far
	+--------------------------------------------+

In theory the realloc() algorithm could be fixed to first copy the 341MB
chunk down to the start of the heap then it wouldn't have to extend the
heap at all until the next realloc(343MB), but apparently it doesn't do
that.

So in fact the process has grown to the full 1GB limit and is now trying
to grow past that limit.  Your "only 1/3 of memory" statment threw me
off but now that I think about it again that's exaclty right and it fits
in perfectly with my observation in "top" of the size going up and down
as the third block would be freed every other time and the nice
allocator was giving back memory to the system when that happened!  :-)

> You still had 512MB of physical memory to play with.

Indeed -- which is why I suggested the 1GB default RLIMIT_DATA was just
way too big for your environment.  :-)

Any time your system's operation, and especially that of some CPU hungry
process that regularly touches most of the pages in its heap, depends on
swap space in order to grow big enough then your system is simply too
small.  VM is great for lazy progrmmers who want to grab great big gobs
of memory just to make their algorithms simpler (maybe they only
reference most of it just once, or maybe just twice in some neat order
that doesn't fight against the pager, and maybe there are big holes that
are never referenced), but as soon as you try to hammer away on all that
primary storage back and forth constantly you either need to have enough
real RAM for the vast majority of it, or else you need to be prepared to
wait perhaps many _thousands_ of times longer as it'll all have to be
moved on and off the secondary swap storage many times over and over
again.


> In my case "sluggish"
> was not the word. I waited for more than 8 hours for the system to come back
> up, and it didn't. I've also had sudden reboots because of this.

I still wonder sometimes if maybe there's a deadlock of some kind in the
pager when things get that tight.  I'm assuming your disk was constantly
busy during that time....  I know I've seen NetBSD systems thrash
themselves silly with no apparent progress ever being made.  This isn't
necessarily the over-commit of resources either -- it seems to be some
kind of "big loop" deadlock....

In theory with our new VM it should be be easier to tune it so that more
pages of the right kinds can be kept in memory so that this kind of
deadlock doesn't happen so easily, but doing the experiments to find
that tuning could be very tedious and time consuming.


> well, setting the resource limits lower may help, but will not prevent
> processes naively allocating memomry from being killed.

The process wasn't killed was it?  realloc() returned NULL and it exited
of its own accord by design of its code.

AIX has/had a SIGDANGER which will kill any large process that hasn't
caught the signal (e.g. Xserver catches it! ;-) so that when big
processes allocate lots of memory and then eventually does reference all
those pages (as this realloc() experiment will), then if there's not
enough swap space available the kenel will start sending SIGDANGERs
until enough things die and give up enough resources for the system to
continue on working as a whole.

-- 
						Greg A. Woods

+1 416 218-0098                  VE3TCP            RoboHack <woods@robohack.ca>
Planix, Inc. <woods@planix.com>          Secrets of the Weird <woods@weird.com>