Subject: Re: Profiled NetBSD slowing down applications considerably
To: None <tech-perform@netbsd.org>
From: Thor Lancelot Simon <tls@rek.tjls.com>
List: tech-perform
Date: 01/22/2003 13:21:23
On Wed, Jan 22, 2003 at 09:43:21AM -0800, Dheeraj Pandey wrote:
> I was trying to track down the reason where most of the CPU is being
> consumed in my application on NetBSD 1.5W. I therefore, profiled my
> application, and realized that 60% of the time, the application was
> on-the-cpu while executing _sys_write(), _sys_select() and _sys_read(). I
> realized that write() and read() could be because of the TCP stack the
> application uses. What was surprising that the sampling profiler found the
> application taking 13% on-the-cpu during select()?!!

This is probably a pretty good sign that you've got a broken
select loop somewhere, perhaps looping over the select() system call
on a file descriptor that's in non-blocking mode (this is a common
error.  Correct sample code and a discussion of the potential
problems is in at least one of the W. Richard Stevens books; actually
I think it's in both _Advanced Programming in the Unix Environment_
and in _Unix Network Programming).

> I then decided to profile the kernel and see what the kernel is doing while
> these applications are running. As soon as I switch the profile on in the
> kernel, the applications come to a dead halt... cpu is 100% idle... as

Yep.  I think your application is spinning calling some system call
(probably select() or read() on a nonblocking descriptor that has no
data available, or both) when you intended for it to be sleeping waiting
for data to show up.  With kernel profiling turned on, spinning over
a system call has a distasterous impact on system performance, rather
than just an annoying impact as it does with kernel profiling off.

Thor