Re: Linux compat and swap

[Paul Ripke <stix%stix.id.au@localhost>]

On Thu, Apr 23, 2020 at 06:14:00PM +0200, tlaronde%polynum.com@localhost wrote:
> On Thu, Apr 23, 2020 at 01:30:58PM +0100, Mike Pumford wrote:
> > 
> > 
> > On 23/04/2020 11:48, tlaronde%polynum.com@localhost wrote:
> > >[Please forgive me to give supplementary information gathered for all
> > >answers and "before". And thank you to all for answering.]
> > >
> > >To be more precise:
> > >
> > >The node is not a desktop node but a server one (so there is no X11
> > >running, so no firefox or whatever), with two main functions:
> > >
> > >1) A file server;
> > >2) A processing node for data (GIS) manipulations, since there can be
> > >huge amounts of data to process, so it is more efficient to process
> > >where it is saved due to the network capabilities.
> > >
> > In my case the build machines main purpose is as a build server (hence the
> > jenkins server which is a java process). I use it as a desktop when
> > developing things as its my most powerful BSD build machine and its easier
> > to develop when you can use a browser to check API references etc so its
> > desktop use is intermittent. So they were just examples of memory hungry
> > processes that got swapped out unexpectedly rather than the only ones.
> > 
> > >The node' uptime is 225 days (since some disk replacements) serving
> > >during office time files to 2 to 6 nodes. And it is processing data
> > >(sometimes during office time; generally outside office time).
> > >
> > >It has 3862 MB of avail memory.
> > >
> > >When using a program to convert coordinates, this program being a Linux
> > >binary, the program bailed out, 3 times in a row, after processing 20M
> > >records. Splitting the input data under this amount, I managed to
> > >process everything.
> > >
> > Could you be running into the data size limits for the process?
> 
> No I think that you gave the answer in your previous mail. This is the
> sysctl variable vm.execmax.
> 
> It is set to 30 (30%) and it was reaching the limit. Since there was
> till now no need for more, the swap space is limited (only 128MB).
> 
> So the solution is indeed to play with the vm.* variables and here to
> increase vm.execmax (and I will too add some swap since I have spare
> partitions on some disks).

To the best of my knowledge, vm.execmax just defines the maximum amount
of memory for mapped executable pages, and won't cause processes to be
killed. Since mapped executable pages are generally read-only, they can
just be dropped from RAM and paged in again from the executables as
needed.

You probably just need to decrease vm.filemax, so that the file cache
doesn't put so much pressure on anonymous memory, and allows processes
to grow larger.

> > ulimit -d is
> > data size, -s is stack size. If a process hits those limits it may be killed
> > even if there IS memory available to protect the rest of the system.
> > 
> > Here's a comparison from the bash ulimit on BSD and linux:
> > 
> > BSD system has 2GB of RAM:
> > $ ulimit -a
> > number of threads               (-T) 1024
> > socket buffer size       (bytes, -b) unlimited
> > core file size          (blocks, -c) unlimited
> > data seg size           (kbytes, -d) 524288
> > file size               (blocks, -f) unlimited
> > max locked memory       (kbytes, -l) 669921
> > max memory size         (kbytes, -m) 2009764
> > open files                      (-n) 2048
> > pipe size            (512 bytes, -p) 1
> > stack size              (kbytes, -s) 4096
> > cpu time               (seconds, -t) unlimited
> > max user processes              (-u) 300
> > virtual memory          (kbytes, -v) unlimited
> > 
> > Ubuntu 18 Linux (VM with 8GB of RAM):
> > $ ulimit -a
> > core file size          (blocks, -c) 0
> > data seg size           (kbytes, -d) unlimited
> > scheduling priority             (-e) 0
> > file size               (blocks, -f) unlimited
> > pending signals                 (-i) 31721
> > max locked memory       (kbytes, -l) 16384
> > max memory size         (kbytes, -m) unlimited
> > open files                      (-n) 1024
> > pipe size            (512 bytes, -p) 8
> > POSIX message queues     (bytes, -q) 819200
> > real-time priority              (-r) 0
> > stack size              (kbytes, -s) 8192
> > cpu time               (seconds, -t) unlimited
> > max user processes              (-u) 31721
> > virtual memory          (kbytes, -v) unlimited
> > file locks                      (-x) unlimited
> > 
> > Note the fact that linux has datasize unlimited by default. That BSD
> > datasize does not scale with system memory. Its still 512MB even on my
> > system with 16GB of RAM. All my build wrappers for pkgsrc do ulimit -d to
> > allow for the compile processes to grow really large (which they do :( )
> > 
> > >=> This is why I jumped to the conclusion that the program had memory
> > >leak because processing should be one line of input at a time so
> > >whatever the number of lines, I fail to see why it would need more
> > >memory for more records.
> > >
> > Perhaps its badly written. ;) As I said in an earlier e-mail I had a log
> > crunching program that leaked in the same sort of way. It did store small
> > amounts of data for each line processed but the leak was actually caused by
> > a logic error in a core loop than inadvertently caused memory to be kept
> > around for every line processed.
> > >I found, afterwards, in /var/log/messages, that this program was killed
> > >because "out of swap".
> > >
> > >
> > >Concerning my sentence "RAM is not exhausted", my assumption (another
> > >one...) was that when memory for processing was getting scarced, file
> > >caching was reduced since there seems to be a basic "page" device for
> > >files: the file on disk. So for me, when there was still a huge amount
> > >of memory for file caching there was still a huge amount of memory as
> > >spare for programs at the detriment of file caching hence speed.
> > >
> > See above about process limits. you might find ulimit -d unlimited will let
> > the process use more RAM and possibly even run to completion.
> 
> As said above, I think your hints about the sysctl vm variables is the
> main solution to my problem.
> 
> Thanks again!
> -- 
>         Thierry Laronde <tlaronde +AT+ polynum +dot+ com>
>                      http://www.kergis.com/
>                        http://www.sbfa.fr/
> Key fingerprint = 0FF7 E906 FBAF FE95 FD89  250D 52B1 AE95 6006 F40C

-- 
Paul Ripke
"Great minds discuss ideas, average minds discuss events, small minds
 discuss people."
-- Disputed: Often attributed to Eleanor Roosevelt. 1948.