Re: track down why malloc fails

["James K. Lowden" <jklowden%schemamania.org@localhost>]

Michael van Elst wrote:
> The trace says that there is insufficient memory.

You are of course correct, see below.  Did you rely on

 28999 svn      CALL  mmap(0,0x21000,3,0x1002,0xffffffff,0,0,0)
 28999 svn      RET   mmap -1 errno 12 Cannot allocate memory

or was there something else besides that?  

> >data(kbytes)         131072
> 
> >That tells me the process can allocate 256 MB of memory.
> 
> 128MB
... 
> Search for the first break() call to see roughly where it starts.

Bingo!  

        $ grep break kdump.out.2008.06.26 | grep -v 0$ | sed -n '1p; $p;' 
         28999 svn      CALL  break(0x8066dac)
         28999 svn      CALL  break(0x10059000)

        $ grep break kdump.out.2008.06.26 \
        | grep -v 0$ \
        | awk -F'[()]' '{print $2}' \
        | sed -n '1p; $p;' \
        | xargs printf 'echo memory used: $(( (-%d + %d) / 1024 ))' \
        | sh; \
        echo "ulimit -d:   $(ulimit -d)"

        memory used: 131016
        ulimit -d:   131072

And, as you said, the overnight job uses a lot less memory when I run it
myself(!)

        $ kdump ktrace.out \
        | grep break \
        | grep -v 0$ \
        | awk -F'[()]' '{print $2}' \
        | sed -n '1p; $p;' \
        | xargs printf 'echo memory used: $(( (-%d + %d) / 1024 ))' \
        | sh ; \
        echo "ulimit -d:  $(ulimit -d)"

        memory used: 17504
        ulimit -d:  131072

Thanks very much, Michael.  Next time you're in my neck of the woods,
remind me it's my turn to buy.  

Regards, 

--jkl