slow pipes chewing up system time (and real time)?

[Lloyd Parkes <lloyd%must-have-coffee.gen.nz@localhost>]

Hi,
I have finally got around to modifying the Mercurial CVS conversion tool so 
that it handles the NetBSD CVS repositories and I would like to run it on 
NetBSD to provide daily incremental repository updates, but I can't because the 
NetBSD performance is so bad.

I did all my development work on OS X because that's the biggest box I own and 
converting the src repository requires about 13GB of RAM. When I moved to 
running the tool on NetBSD, I started with the othersrc repository because that 
makes testing somewhat more manageable. My problem is that the conversion is 
five times slower on NetBSD than it is on OS X in real time even though CPU 
usage appears to be comparable. 

My Mac is 3.4GHz Intel Core i7 with 4 cores and one extra virtual hyper thread 
core for each real core. The OS X 10.8.2 is run from an external thunderbolt 
attached RAID 0 disk pair and the data I'm working with is on the internal WD 
Caviar Black formatted with a case sensitive journaled HFS+ file system. The 
cvs server uses a RAM disk for it's temporary storage.

My NetBSD box is a 3.3GHz AMD Phenom II  with 6 cores. NetBSD 6.0 is run from 
an AHCI attached SSD (an old SSD) and the data I'm working with is variously on 
tmpfs or the SSD. The SSD file systems are formatted with FFSv2 without 
logging, with noatime, and for the designated destination file system, with 
async. The use of tmpfs or async doesn't make an obvious change to performance.

"time -l" on the Mac and NetBSD give me similar user CPU and system CPU numbers 
for the conversion of othersrc (half a minute and five seconds respectively), 
but real time on the Mac is 68s and on NetBSD it is 359s. Even though the 
reported system time is only five seconds, when I run top on NetBSD, one of the 
CPUs just sits there at 100% system.

I don't think the horrible wall clock time is caused by hardware constraints 
(disk performance etc.) because I have tested this on tmpfs with no change and 
after running ktrace I'm a bit suspicious of the pipe performance. I can't see 
a way to find out how long each system call is taking, so I don't know for sure 
though. The application talks to the cvs server with a stop and wait 
implementation of the cvs protocol and the workload is a lot of small requests. 
I see support in NetBSD for pipes handling lots of large requests and I'm 
wondering is anyone knows of any problems with NetBSD pipes handling lots of 
small requests.

Cheers,
Lloyd