> Has anybody done any work profiling the kernel? Are tools available to
> do this?

I've done quite a bit, but not to address the particular problem you're
asking about.

To profile the kernel:

1) Build a profiling kernel:

	cd /sys/arch/<whatever>/conf
	config -p MYCONFIG
	cd ../compile/MYCONFIG.PROF
	make depend
	make
	<install profiling kernel and reboot>

2) To start profiling, as root run:

	kgmon -r -h		# this is probably unnecessary
	kgmon -b

3) Now run your application or whatever.

4) When you want to stop the profile, run (as root)

	kgmon -h
	kgmon -p

This produces a gmon.out file. Now you can use gprof to analyze the profile:

5) Run gprof:

	gprof /netbsd gmon.out > profile

6) Now go look at the "profile" file. "man gprof" for details on the format;
basically, you get a hierarchical call graph as well as a flat profile. To 
start out, the flat profile is the easiest to understand; search for "flat"
in the profile output.

Something similar should work under FreeBSD if they supply kernel profiling
tools; I know it works under BSDI.

--Aaron

> The reason I ask is that, in recent pissing contests here, we've seen
> FreeBSD to spend about 3/5 as much time in the kernel as NetBSD for a 15
> minute build of 1500 C files (something like 1:30 minutes vs
> 2:30). Total build times were 14:34 and 16:16 (user time was 12:10 and
> 12:22, the rest of the difference was in waiting for disk, I guess. I
> believe the machines are similarly configured (P6-200 w/256k cache, SCSI
> disk w/2940UW controller, 128MB RAM) and the cache was large enough on
> the NetBSD system that it kept disk accesses down below 100k/sec during
> most of the cc's (ie, all the .h's were coming from cache).
> 
> Anywho, I'd really like to see the NetBSD build time match the
> FreeBSD. If there is a "cookbook" for collecting kernel profile
> statistics, I'd be willing to follow it, collect data during several of
> these builds, and provide the results to people competent enough to do
> some hand-optimization in the kernel.
> 
> (By the way, all these timings were with NetBSD 1.2 release and FreeBSD
> 2.1.something (whatever was released as of December). Have there been
> any speed improvements in the kernel since 1.2?
>