null mounts vs NFS (and RAIDframe) performance?

["John D. Baker" <jdbaker%consolidated.net@localhost>]

I have a nice fast machine with a decent number of CPUs and lots of RAM
(and power consumption/heat dissipation to match).  It currently runs
netbsd-9/amd64 from local disk.

When I want to run -current on it I netboot it and its root file system is
on NFS.

Back in 7.99.* and early 8.99.* days, I arranged to null-mount "/usr"
from a directory on one of the local hardware RAID volumes on the machine
to speed up operations when loading compilers, etc.

I made the mountpoint for the RAID volume part of
"critcal_filesystems_local" and then defined null mounts for "/usr",
"/var/db/pkg" and "/var/db/pkg.refcount" that lived in a subdirectory
on the RAID volume.  At the time it was very fast, almost as fast as
the locally-installed OS (7.x_STABLE at the time and later 8.0_STABLE).
It also helped that build results went to another local RAID volume and
${WRKOBJDIR} was on tmpfs.

Lately, I fired the machine up again and was rather surprised and
disappointed that my null-mounted "/usr" arrangement now dragged all
network performance to a crawl, but especially pummelled my file server
using a RAIDframe RAID where diskless machines filesystems lived.

It was especially bad when all CPUs were busy with the C++ compiler
(I run MAKE_JOBS=$(($(sysctl -n hw.ncpu) + 1)).)

Meanwhile, another machine with all normal filesystems on NFS and only
swap and scratch space on local disk (and ${WRKOBJDIR} in tmpfs) runs
very well and doesn't stress my network or the file server.

A comment in another thread regarding nullfs behavior prompted me
to finally write about my observations.

-- 
|/"\ John D. Baker, KN5UKS               NetBSD     Darwin/MacOS X
|\ / jdbaker[snail]consolidated[flyspeck]net  OpenBSD            FreeBSD
| X  No HTML/proprietary data in email.   BSD just sits there and works!
|/ \ GPGkeyID:  D703 4A7E 479F 63F8 D3F4  BD99 9572 8F23 E4AD 1645