Re: ulimit -d/-m do not actually work

[Greg Troxel <gdt%lexort.com@localhost>]

Michael van Elst <mlelstv%serpens.de@localhost> writes:

> On Wed, Oct 23, 2024 at 11:59:17PM +0700, Robert Elz wrote:
>
>> Everything should be counted in one of those three categories for resource
>> limit purposes.
>
> The three categories no longer match the real world when people started
> to use shared memory, and you can not pretend that these are still
> sufficient as a model.
>
> But more important, the limits are also no longer fit for their purpose.
> What does a memory limit mean, when you can still allocate address space
> and what does an address space limit mean, when it doesn't affect
> memory usage ?

I am coming into this discussion wanting to set limits that do what I
want: to control the amount of memory in use by a process, in order to:

  a) protect my system from piggy processes, both because they slow
  things down, and because of other bugs running out of memory leads to
  lockups

  b) understand whether a workload fits in a limit, to help debug a
  program that should use less memory than it does


POSIX has some words, and one can argue about how that should be mapped
to the modern world.

My take is that when I run a program, the "text" size is the program's
size(1) plus that of the shlibs.  There's a minor complexity with
dlopening further libraries, but generally, this is not where the pain
is.   POSIX doesn't have a label/limit for this and that's a clue that
it is not mysterious, hard to debug, and a source of woe.

And, the "data" size I see as the initial data plus anything that smells
like heap or allocation.  I intend to limit the total amount of memory
used to store data.  So anon mmap, is really the same thing as sbrk,
from the user/sysadmin viewpoint of "did the program get some memory to
use" and "does the system now have less available".  The fact that
malloc is using anon mmap instead of sbrk -- even though they are
basically the same thing operationally -- is just an implementation
detail.

I can see that mmap of a file is different, as that is letting you see
data whose storage is part of the buffer cache (loosely speaking as I am
intending to, because I'm talking about unix in general, not just
netbsd).

The virtual limit is awkward as there is a lot involved and it's really
not obvious what anything means.   And, it's not what I want, which is
"address space allocated that is going to be backed by allocated pages
if I use it".

The rss limit seems conceptually broken as all the rest are "shall not"
and are limits.  rss "limit" is like vm.filemax which is "sort of
protected up to x".  The setrlimit man page describes it incorrectly:
"The maximum size (in bytes) to which a process's resident set size may
grow."

The real problem is that if I want to say "this program is allowed 8192
MB of data, and I want malloc to fail after that", I can't.  And I can't
do it portably.

I found that on Linux, "ulimit -d" did what I expected (and what I
want).  I think kre@ is basically saying that's how NetBSD should behave
also.

My impression is that it wasn't all that long ago that I had to bump up
'datasize' in login.conf to get programs to run.