libc.so's vmobjlock / v_interlock

[Andrew Doran <ad%netbsd.org@localhost>]

Hi,

Here's a dtrace flamegraph for a kernel build on my system, while running a
kernel from the ad-namecache branch.

	http://www.netbsd.org/~ad/2020/jan19.svg

The biggest single remaining concurency problem is the mutex on libc.so's
uvm_object / vnode.  It pops up in a number of places, but most clearly seen
above the "uvm_fault_internal" box.

I think making the vmobjlock / v_interlock a rwlock could help with making
inroads on this problem, because in the libc.so case it's not modifying too
much under cover of the lock (mostly pmap state).

That's not a small undertaking, but I reckon it would take about a day of
donkey work to change every mutex_enter(..) to a rw_enter(.., RW_WRITER) and
make a start at choosing rw_write_held() or rw_lock_held() for the
assertions, followed by a bit of benchmarking to check for a regression and
a jumbo commit.  From there on things could be changed incrementally.

Thoughts?

Andrew