FYAmusement, I've been testing a NetBSD, FreeBSD 5.3, debian/redhat combo
with the same hardware with some people here, and while we can reliably
get 400-600mbit/sec on back-to-back gig-e copper from netperf, when we do
scp <bigfile> even to /dev/null, we're getting much much lower total
throughput, around 100mbit max.

Its reasonably new SATA disks on RAID, we suspect that its a combination
of application overhead, disk I/O, PCI, phase-of-the-moon.

Not much difference Linux/BSD on this stuff in the network thus far,
arguably marginally better TCP from BSD.  ReiserFS is overwhelmingly faster than ext3, and pretty much the same as UFS2 (1.5Tb RAID5 backed disks) but without the inode overhead, so you get about 50Gb more usable diskspace which was nice.

-George