Re: netbsd-4 vs -current web server performance

[Jeff Rizzo <riz%NetBSD.org@localhost>]

matthew sporleder wrote:
> On Wed, Sep 3, 2008 at 3:08 PM, Jeff Rizzo <riz%netbsd.org@localhost> wrote:
>   
>> Thor Lancelot Simon wrote:
>>     
>>> On Tue, Sep 02, 2008 at 02:52:33PM -0700, Jeff Rizzo wrote:
>>>       
>>>>   PID USERNAME PRI NICE   SIZE   RES STATE      TIME   WCPU    CPU COMMAND
>>>>  4742 www       42    0    17M  314M parked/2 233:14 99.02% 99.02% httpd
>>>>  3265 www       39    0    17M  266M parked/0 164:34 99.02% 99.02% httpd
>>>>  8844 www       40    0    17M  298M parked/0 158:27 99.02% 99.02% httpd
>>>>
>>>>         
>>> Parked *and* 100% of a CPU?
>>>
>>> That seems just wrong.
>>>
>>> Thor
>>>
>>>       
>> I would tend to agree - but I don't know all that much about the new
>> states.  This issue is fairly reproducible (within a few hours) - and
>> also seems to go away (for a few hours) after restarting apache.  I have
>> a couple weeks to get this fixed, since even at it's worst it's not too
>> bad, and in any event I have multiple webservers.
>>
>>     
>
>
> Do the httpd's slowly increase until they get parked at 99% or do they
> suddenly spike?
>
>   

Almost impossible to tell - we have enough web traffic to keep a couple
servers busy at our slow times, so the line between "noticably, but
normally busy" and "wedged" is hard to demarcate - until they get into
the 'parked' state.

> If it's a slow increase, could you start monitoring various stats like
> the number of threads/PID, memory growth, etc?
>   

It hadn't occurred to me to look at specific threads - I am currently
seeing one thread on one machine that's using about 90% CPU, and it's
always on one of the CPUs.  That *process* is showing in 'parked' and is
using 90% of a CPU as well.  (I've reset everything in the last couple
hours, so this looks like the "bad state" starting to happen)  I don't
see any single thread on the other -current webserver getting more than
12-13% of a CPU for more than one 5-sec interval.

Interestingly, memory usage on the -current boxes seems lower than the
-4 box, if top is to be believed:

netbsd-4:
Memory: 7770M Act, 35M Inact, 9188K Wired, 19M Exec, 7062M File, 6924M Free
Swap: 1024M Total, 1024M Free

current (1):
Memory: 1286M Act, 502M Inact, 12M Wired, 14M Exec, 106M File, 14G Free
Swap: 4096M Total, 4096M Free

current(2):
Memory: 1289M Act, 747M Inact, 11M Wired, 14M Exec, 750M File, 13G Free
Swap: 4096M Total, 4096M Free

> If it's a sudden spike, is there a way you can run ktrace or something
> until you can start to isolate the trigger?
>   

Kinda hard.  Real busy.  And it's definitely _not_ sudden - looks like
problem threads just accumulate until the whole server slows down.

+j