Subject: Re: Replacement for grep(1) (part 2)
To: Nate Williams <nate@mt.sri.com>
From: Matthew Dillon <dillon@apollo.backplane.com>
List: tech-userlevel
Date: 07/14/1999 16:02:43
:
:Most of the work we've done wouldn't allow this, especially if we were
:using an OS like FreeBSD with a fairly long bootup time.  Especially if
:it can be avoided.
:
:Yes, we could (and did) do our own memory management, but it seems to me
:that the kernel has alot more information available to it and would do
:it better than I could.  Then again, maybe I'm totally confused about
:how the VM system 'does its thing', and in reality it's no better at it
:than our code, just alot more complex for no reason. :) :) :)
:
:Nate

    The kernel just isn't the best place to do this.   The level of 
    sophistication required to satisfy the larger set of programming
    is much greater then anything discussed here.  The last thing that the
    kernel should be doing is returning NULL from malloc() to a program
    which is operating within its specifications.  That doesn't help anyone,
    least of all the programmer who now has to deal not only with the
    complexity of the project he is working on, but must also deal with
    the potential that the OS will step in at any time and give him an 
    error that he must deal with in a sophisticated fashion even when his
    software is operating properly.

    The same goes for non-embedded work.  Why is it that programs generally
    exit when they encounter a memory allocation failure?  Because it is too
    damn hard to ensure proper operation at every instance where a program's
    allocation may fail, especially if the program getting the failure is not
    responsible for the problem.  The glib statement that "programs should deal
    with memory allocation errors" provides no solution.  98% of the source
    code in the BSD code base will exit if a memory allocation fails, and
    I don't know anyone who wants to go through and somehow "fix" it all 
    (and one could argue what the "fix" should be when something like grep or
    a shell has a memory allocation failure).  To require that this code be
    made twice as sophisticated as it is already in order to deal with a
    non-overcommit model reliably is a fantasy.

    Due to the side effects it causes, to put the capability in the kernel 
    in this situation will make things even more unstable then the kernel's
    current operation.  A situation made much worse by the magnitude of the
    resources necessary to bring such a system up to the same performance
    levels attainable by a standard overcommit system.  The only case where
    this does not occur is also the case where the programmer can fairly
    easily manage the memory to avoid the overcommit from occuring in the
    first place.

					-Matt
					Matthew Dillon 
					<dillon@backplane.com>