I'm really happy to have sparked off such a discussion. A lot of this
is very useful to me and I like corresponding with you folks...

> > #2 must not exceed #1-x% where x is some small single-digit number. If
> > it exceeds #1-x%, things choke and I need to reboot the machine. (This
> > load is driven mostly by users clicking and clicking and hitting
> > refresh etc.)
> >
> 	If it chokes can you pull the network cable and get control of
> 	the machine back, or has it really gone out to lunch?

Nah, when it gets so bad I can't even type at the console I *have*
seen it pull itself out of it, but only after about four or five hours
of churning. This was in my own testbed machinery which I use to try
(and usually fail) to simulate the real environment.

> > If #3 (and corresponding perl cgi children) exceeds #2, apache stalls
> > and is no longer capable of handling cgi. It must be SIGHUP'd to be
> > corrected.
> >
> 	Hmm - have you ha a chance to try to debug what apache is doing
> 	in this case?
>
> 	It would be nice if apache could be told to 'tune down' the
> 	max processes when hitting a fork failed condition (presumably
> 	to another 'MinMaxProcesses' limit).

True enough, that sort of behaviour would be very nice. However it's
looking like Apache rather than tuning down its forking, stops it
completely. I think it would be pretty simpleto fix; however I haven't
thus far been successful in simulating the failures in a non-live
environment. :) I've tracked down the error message itself in Apache
that it logs into its error log; just the "Resource temporarily
unavailable". I think dealing with the problems at this juncture would
be a good idea and something probably worthwhile pursuing (for me).

Either that or I'm just setting up the environment incorrectly and I'm
a goon. :)

> > It's a juggling act once the kernel is tuned to be capable of dealing
> > with the huge limits and doesn't panic any more under the loads.
> >
> 	Were the panics purely down to tuning the kernel params? Are
> 	there PRs open on them?

I must apologize, I couldn't find any open PRs but I figured you guys
had enough work to do without worrying about something as seemingly
trivial as this--all I had to do was increase MKMEMPAGES and
MAXUSERs..  and NMBCLUSTERS, etc to reasonable limits within the
currently available memory and the problems haven't resurfaced.

> 	NetBSD -current supports a unified buffer cache, which will
> 	hopefully eliminate the need to tune this at boottime, though
> 	I'd imagine it will have some parameters that can be tweaked.

Ah, my personal holy grail. :) A system that tunes itself. Scratch
that. A NetBSD kernel that tunes itself. :)

> > This is so complicated, I was hoping another user here could offer
> > some common server configurations based on available RAM and disk type
> > (IDE vs SCSI) that they've been able to work with and not have to
> > worry about things getting out of control.
>
> 	Are you using mod_perl or fastcgi on the server? Switching
> 	a heavy perl/postgresql cgi client to fastcgi increased its
> 	apachebench top rate from 1.8 hits per second to over 80.
>
> 	It would be really nice at the end to have a doc to put up on
> 	www.netbsd.org - would you be willing to put one together? :)

I'd be honoured to help throw something together--really. :) If I do
get something written who should I be submitting a draft to?

The modperl/modcgi is an excellent idea. I think I'm going to compile
those in as my next step. Totally forgot about them in my focus on
the rest of the system. :)

Marc Tooley