Subject: Re: Replacement for grep(1) (part 2)
To: Matthew Dillon <firstname.lastname@example.org>
From: Chris G. Demetriou <email@example.com>
Date: 07/13/1999 12:24:35
[cc list trimmed because it was getting ... insane, and it's not like
this is a critical point. It's just beating up a topic which has been
beaten up by many others.]
Matthew Dillon <firstname.lastname@example.org> writes:
> Thus it makes little sense to try to disallow overcommit. It gains you
> absolutely nothing, and forces you to waste huge amounts of disk space.
> To anyone who is paranoid about it: Fine, then just allocate an insane
> amount of swap and forget about it. It would be no more then you would
> have to allocate anyway if we were to actually disallow overcommits! But
> with overcommits allowed, your box will never come close to using that
> much swap.
This may be a decent answer for the workstation world, but it's not so
good for more restricted systems. Further, your claim that
disallowing overcommit gains you absolutely nothing is simply false.
It gains you two things (which are at least immediately obvious to me):
* Certain knowledge that (if the system is implemented correctly) the
system will never have to kill a process (or take similar corrective
action) due to overcommit, and that attempts to allocate more backing
store resources than are present will fail.
If the programs you're using do reasonable error handling when out-of
memory situations crop up, then this can be a very useful propery to
have if you're building a reliable system. It's not necessarily a
matter of gigs of swap space. It means that you know that you can
safely run in, say, 64MB or memory, with _no_ swap, if your programs
don't overallocate ridiculously.
The only alternative to avoiding overcommit that seems plausibly
enforceable by the OS/system designer is using resource limits and
then planning based on the maximum resource limits and number of
processes you allow. However, that results in a ridiculously loose
upper bound and a fair bit of extra work tuning the limits.
The obvious workstation example is a bit grotesque: 1GB max datasize
by default, maxproc 532, and you need > .5TB to be _sure_ that no
process will get killed for lack of resources.
A more realistic embedded system might allow 64 processes, two of
which need 10MB each and the rest of which need 300K-1MB each. If you
do this naively, you end up "needing" 640MB to guarantee safely. If
you tweak a bunch of things with some special-purpose resource
limiting code, you can get it down to 96MB or so to guarantee safety.
Only by going to a _great_ deal of extra effort -- which could be
avoided by just disabling overcommit -- you can't be sure the system
will operate safely in, say, 48MB or even 64MB, both of which may well
Using resource limits to solve this problem is ... not the right
answer (even if various systems implemented resource limits
* protection against bogosity.
I may run a system in which all of the processes are effectively
unlimited (i.e. have huge resource limits), but I know within a
tolerance what the actual expected usage of the system is.
If a program mallocs (or equivalently allocates with mmap) <a huge
amount> it should be told up front that that's not possible (or
that it is), and filling that memory in later simply should not
result in its death or in the death of another process. This
becomes especially important when the process making the bogus
request is a "critical" process or when the other process that
happens to be killed is a critical process.
Certain knowledge about system operation, and deterministinc system
behaviour in the face of programming bugs is exactly what the desire
for overcommit is about.
Overcommit avoidance may not be useful for your particular uses of
these UNIX-like systems. However, if you think that it's not useful
to anybody who uses them (or that people who think it's useful are
deluding themselves 8-), then you're sorely mistaken and have a
... very wrong-headed attitude about why people find such features
Chris Demetriou - email@example.com - http://www.netbsd.org/People/Pages/cgd.html
Disclaimer: Not speaking for NetBSD, just expressing my own opinion.