NetBSD-Users archive

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index][Old Index]

Re: Linux compat and swap

On 23/04/2020 11:48, wrote:
[Please forgive me to give supplementary information gathered for all
answers and "before". And thank you to all for answering.]

To be more precise:

The node is not a desktop node but a server one (so there is no X11
running, so no firefox or whatever), with two main functions:

1) A file server;
2) A processing node for data (GIS) manipulations, since there can be
huge amounts of data to process, so it is more efficient to process
where it is saved due to the network capabilities.

In my case the build machines main purpose is as a build server (hence the jenkins server which is a java process). I use it as a desktop when developing things as its my most powerful BSD build machine and its easier to develop when you can use a browser to check API references etc so its desktop use is intermittent. So they were just examples of memory hungry processes that got swapped out unexpectedly rather than the only ones.

The node' uptime is 225 days (since some disk replacements) serving
during office time files to 2 to 6 nodes. And it is processing data
(sometimes during office time; generally outside office time).

It has 3862 MB of avail memory.

When using a program to convert coordinates, this program being a Linux
binary, the program bailed out, 3 times in a row, after processing 20M
records. Splitting the input data under this amount, I managed to
process everything.

Could you be running into the data size limits for the process? ulimit -d is data size, -s is stack size. If a process hits those limits it may be killed even if there IS memory available to protect the rest of the system.

Here's a comparison from the bash ulimit on BSD and linux:

BSD system has 2GB of RAM:
$ ulimit -a
number of threads               (-T) 1024
socket buffer size       (bytes, -b) unlimited
core file size          (blocks, -c) unlimited
data seg size           (kbytes, -d) 524288
file size               (blocks, -f) unlimited
max locked memory       (kbytes, -l) 669921
max memory size         (kbytes, -m) 2009764
open files                      (-n) 2048
pipe size            (512 bytes, -p) 1
stack size              (kbytes, -s) 4096
cpu time               (seconds, -t) unlimited
max user processes              (-u) 300
virtual memory          (kbytes, -v) unlimited

Ubuntu 18 Linux (VM with 8GB of RAM):
$ ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 31721
max locked memory       (kbytes, -l) 16384
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 31721
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

Note the fact that linux has datasize unlimited by default. That BSD datasize does not scale with system memory. Its still 512MB even on my system with 16GB of RAM. All my build wrappers for pkgsrc do ulimit -d to allow for the compile processes to grow really large (which they do :( )

=> This is why I jumped to the conclusion that the program had memory
leak because processing should be one line of input at a time so
whatever the number of lines, I fail to see why it would need more
memory for more records.

Perhaps its badly written. ;) As I said in an earlier e-mail I had a log crunching program that leaked in the same sort of way. It did store small amounts of data for each line processed but the leak was actually caused by a logic error in a core loop than inadvertently caused memory to be kept around for every line processed.
I found, afterwards, in /var/log/messages, that this program was killed
because "out of swap".

Concerning my sentence "RAM is not exhausted", my assumption (another
one...) was that when memory for processing was getting scarced, file
caching was reduced since there seems to be a basic "page" device for
files: the file on disk. So for me, when there was still a huge amount
of memory for file caching there was still a huge amount of memory as
spare for programs at the detriment of file caching hence speed.

See above about process limits. you might find ulimit -d unlimited will let the process use more RAM and possibly even run to completion.


Home | Main Index | Thread Index | Old Index