Laine Stump writes:
> Has anybody done any work profiling the kernel? Are tools available to
> do this?

Aaron Brown writes:
> 
> I've done quite a bit, but not to address the particular problem you're
> asking about.
> 
> To profile the kernel:
>
> [perfect instructions]
>

Thanks for such a complete reply! I rebuilt my kernel (using 1.2.1
sources) and did two separate profiling sessions. In both cases the
profiling was turned on during a 20 minute build of a whole raft of c
source residing on a local disk. In the first, I had all the Ix86_CPU
options turned on, in the second only I586_CPU was turned on (didn't
seem to make any difference).

The output of gprof can be found at
http://shad.morningstar.com/gmon.profile* (both gzipped and plain text).
If I understand the output correctly, 29% of the sys time during this
build was spent in a call cycle that starts out with vm_fault().
Does vm_fault occur any time new memory is allocated or something? This
machine has 128MB of RAM, and at least 50MB of it was free during this
entire run (and it was freshly booted, so there wasn't anything left in
swap that had to be dug out).

The next most time consuming cycle was from syscall, which is to be
expected (26%).

The top function (after idle()) was bzero() at 10%. I looked at that,
though, and it's as efficient as it could get without going to the
FreeBSD optimizations which claim occasional 2x performance increase by
using FPU instructions instead of stosl. This gets me wondering if maybe
there aren't some unnecessary bcopy()'s in critical places (you know,
"I'll initialize this to be safe" when it doesn't have any real
effect). However, there are hundreds of bcopy() calls, and it's not
clear to me if gmon/gprof has a way of telling you who is the most
common caller of a particular function.