Subject: Re: Replacement for grep(1) (part 2)
To: Chris G. Demetriou <email@example.com>
From: Matthew Dillon <firstname.lastname@example.org>
Date: 07/13/1999 15:12:14
:Matthew Dillon <email@example.com> writes:
:> If you don't have the disk necessary for a standard overcommit model to
:> work, you definitely do not have the disk necessary for a non-overcommit
:> model to work.
:I'd _really_ like to know how you figure this.
:text data bss dec hex filename
:45024 4096 392 49512 c168 /bin/cat
:311264 12288 9900 333452 5168c /bin/sh
:212960 4096 28492 245548 3bf2c inetd.static
:458720 12288 73716 544724 84fd4 sendmail.static
:None of these are particularly huge. Dynamically linked binaries
:inflate things somewhat, of course, but even there:
:442336 12288 35780 490404 77ba4 /usr/lib/libc.so.12.40
:the data and bss per library just aren't that huge.
:Of course, in addition to the sizes of the data and bss sections, you
:need to make sure you've got enough extra for dynamically allocated
:data, and for stack pages, and for a few other tidbits of dynamically
:allocated storage, but the end result is a system which, even without
:overcommit, can fit into a reasonable amount of backing store.
The text size of a program is irrelevant, because swap is never
allocated for it. The data and BSS are only relevant when they
The only thing swap is ever used for is the dynamic allocation of memory.
There are three ways to do it: sbrk(), mmap(... MAP_ANON), or
Dynamic allocation of memory can occur under a huge number of
conditions. The actual physical allocation of dynamic memory - what is
known as a copy-on-write - cannot be predicted from the potential
allocation of memory. The most obvious example of this is a fork().
There is a lot of hidden 'potential' VM that you haven't considered.
For example, if the resource limit for a process's stack is 8MB, then
the process can potentially allocate 8MB of stack even though it may
actually only allocate 32K of stack. When a process forks, the child
process can potentially modify just about every single non-text page that
was owned by the parent process, causing a copy-on-write to occur.
The dynamic potential can run into the megabytes but most child processes
only modify a small percentage of those pages.
:So, I just went back and looked at my mail folder for this NetBSD
:mailing list (tech-userlevel), which has about a week and a half's
:worth of messages.
:Nowhere did I see what amounts to anything other than hand-waving
:claims that you'll have to allocate much, much more backing store than
:you currently need to, and claims that that's unacceptable for general
:purpose computing environments. If you have a more specific analysis
:that you'd like me/us to read, please point us at it more specifically.
You are welcome to bring up real-life situations as examples. That
is what I do. But look who is doing the hand-waving now?
:* not all the world's a general purpose computing environment,
Which is meaningless handwaving. Again, you are welcome to point out
your own real-life situations.
:* while you certainly need to allocate more backing store than you
:would with overcommit, it's _not_ ridiculously more for most
:applications(+), and, finally,
Based on what? I am basing my information on the VM reservation made
by programs, and I am demonstrating specific points showing how those
reservations occur. For example, the default 8MB resource limit for
the stack segment.
I am also demonstrating the level of control that is possible with
resource limits. If you are doing an embedded system you can easily
guarentee memory utilization even in an overcommit model by properly
setting the memory resource limits for each process. If the system
did not have any swap, I would do this for every single program I
run on that system.
If you are going to argue the point, work out the numbers and present
I had to deal with a reservation model on our old SGI's running 5.3
for almost a year. I know what I'm talking about and I can point to
real-life cases that demonstrate it. Certainly there are many different
situations... you are welcome to bring up other real-life situations
:* even if you are not willing to pay that price, there _are_ people
:who are quite willing to pay that price to get the benefits that they
:see (whether it's a matter of perception or not, from their
:perspective they may as well be real) of such a scheme.
Quite true. In the embedded world we preallocate memory and shape
the programs to what is available in the system. But if we run out
of memory we usually panic and reboot - because the code is designed
to NOT run out of memory and thus running out of memory is a catastrophic
It is not appropriate to test every single allocation for a failure...
that results in bulky code and provides no real benefit since the
code is already designed to not overcommit the embedded system's memory.
If it does, you reboot. The most critical embedded systems, such as
those used in satellites, avoid dynamic allocation as much as possible
in order to guarentee that no memory failures occur. This works under
UNIX as well. If you run your programs and touch all your potentially
modifiable pages you have effectively reserved the memory.
:page from the system, it's almost certainly going write something to
:it, and, while there are undoubtedly a few pages that aren't written
:to, they are by far the majority. And, of course, once the page has
:been written, it's no longer reserved, it's committed. 8-)
Swap is used solely for dynamically modified pages of data that is not
otherwise backed (e.g. not backed by a file, for example).
:I would honestly love to know: what do you see huge numbers of
:reserved pages being reserved for, if they're not actually being
:committed, by 'average' UNIX applications (for any definition of
:average that excludes applications which do memory based computation
:on sparse dasta).
:Chris Demetriou - firstname.lastname@example.org - http://www.netbsd.org/People/Pages/cgd.html
Stack, hidden VM objects after a multi-way fork, MAP_PRIVATE memory maps,
private memory pools (e.g. for web servers, java VM's, and so forth).