Subject: Re: Replacement for grep(1) (part 2)
To: Noriyuki Soda <soda@sra.co.jp>
From: Matthew Dillon <dillon@apollo.backplane.com>
List: tech-userlevel
Date: 07/13/1999 15:29:37
:>     a system out of swap, it is fairly easy to do whether the system
:>     uses an overcommit model or not.  The user has any number of
:>     ways of blowing the server up too - for example, by making
:>     thousands of connections to it or running many huge queries in
:>     parallel.
:
:If the kernel and the application behave properly, critical
:application doesn't lose it's data in such situation on
:non-overcommiting systems.
:Your example doesn't make sense.
:--
:soda

    Give me a shell and I can crash any machine.  If you are assuming
    hostile users, you cannot assume that your magic overcommit protection
    will save your server.  Saying that the kernel and application behave
    properly is a cop-out, because it's virtually impossible to guarentee
    that for every situation.  The chance of a user blowing up the server
    by finding a bug or a hole somewhere is much, much greater then the chance
    of a user running the system out of swap.  Concentrating on the memory
    reservation aspect of the situation means that you miss the more likely
    scenario and, in the end, you do not make the end result any more 
    reliable - in fact, less reliable then the person who doesn't worry
    about the 'kernel overcommit problem' and instead engineers the code 
    such that overcommits cannot occur in the first place.

    Saying that the kernel cannot support certain classes of applications
    because it does not handle a memory/swap overcommit the way you want
    does not change the fact that such critical applications should not allow
    themselves to even attempt an overcommit anyway.

    A good example of this is sendmail.  Before the MaxDaemonChildren and
    MaxArticleSize options, it was possible for sendmail to overcommit a 
    machine.  In this case the overcommit that can occur is with I/O, not 
    swap.  As a general performance rule, you have to set MaxDaemonChildren
    and MaxArticleSize to prevent the overcommit from occuring.  This is a
    function of sendmail, not a function of the kernel.

    Another good example is a web server.  A web server must have specific
    limitations on the number of simultanious connections it is allowed
    to handle at once and on the number of CGI's or other auxillary programs
    that are allowed to be running at any given time.  The overcommit issue
    here has nothing to do with swap and everything to do with performance.
    Specifically, these limitations exist to avoid cascade failures.

    In the same manner any truely critical system server must handle the
    resource management itself to deal with all sorts of problem situations,
    including memory.  You do not need to build any of this control into the
    kernel. 

    To say that FreeBSD does not support a certain class of system because
    it uses an overcommit model is not correct, because you can trivially
    solve the problem by implementing your own management of memory rather
    then use the UNIX libc builtins.  The UNIX libc bulitins properly assume
    a more general machine configuration and it would not be appropriate to
    use them for embedded work if memory use is an issue.

					-Matt
					Matthew Dillon 
					<dillon@backplane.com>