below and see them through. I'm happy to test patches, although I wont
necessarily have access to the same hardware used for current tests.
Hardware specs:
Intel(R) Xeon(R) Gold 5115 CPU @ 2.40GHz
2 sockets * 10 cores * 2 hardware threads
32GB of ram
I assigned all 40 threads to the vm + gave it 16GB of ram.
The host is otherwise idle.
I installed the 7.1 release, downloaded recent git snapshot and built the
trunk kernel while using config stolen from the release (had to edit out
something about 3g modems to make it compile). I presume this is enough
to not have debug of any sort enabled.
The filesystem is just ufs mounted with noatime.
Attempts to use virtio for storage resulted in extremely abysmall
performance which I did not investigate. Using SATA gave read errors
and the vm failed to boot multiuser. I settled for IDE which works
reasonbly fine, but inherently makes the test worse.
All tests were performed with the trunk kernel booted.
Here is a bunch of "./build.sh -j 40 kernel=MYCONF > /dev/null" on stock
kernel:
618.65s user 1097.80s system 2502% cpu 1:08.60 total
628.73s user 1128.71s system 2540% cpu 1:09.18 total
629.05s user 1082.58s system 2517% cpu 1:07.99 total
641.11s user 1081.05s system 2545% cpu 1:07.65 total
641.18s user 1079.89s system 2522% cpu 1:08.24 total
And on kernel with total hacks:
594.08s user 693.11s system 2459% cpu 52.331 total
594.81s user 711.90s system 2498% cpu 52.292 total
600.34s user 676.39s system 2486% cpu 51.336 total
597.33s user 725.78s system 2536% cpu 52.157 total
597.13s user 708.79s system 2510% cpu 52.011 total
i.e. it's still pretty bad, with system time being above user. However,
real time dropped from ~68 to ~52 and %sys from ~1100 to ~700.
Hacks can be seen here (wear gloves and something to protect eyes):
https://people.freebsd.org/~mjg/netbsd/hacks.diff1) #define UBC_NWINS 1024
The parameter was set in 2001 and is used on amd64 to this very day.
lockstat says:
51.63 585505 321201.06 ffffe4011d8304c0 <all>
40.39 291550 251302.17 ffffe4011d8304c0 ubc_alloc+69
9.13 255967 56776.26 ffffe4011d8304c0 ubc_release+a5
1.72 35632 10680.06 ffffe4011d8304c0 uvm_fault_internal+532
[snip]
The contention is on the global ubc vmobj lock just prior to hash lookup.
I recompiled the kernel with randomly slapped value of 65536 and the the
problem cleared itself with ubc_alloc going way down.
I made no attempts to check what value makes sense or how to autoscale it.
This change alone accounts for most of the speed up by giving:
586.87s user 919.99s system 2612% cpu 57.676 total
2. uvm_pageidlezero
Idle zeroing these days definitely makes no sense on amd64. Any amount of
pages possibly prepared is quickly shredded and vast majority of all
allocations end up zeroing in place. With rep stosb this is even less of
a problem.
Here it turned out to be harmful by inducing avoidable cacheline traffic.
Look at nm kernel | sort -nk 1:
----------------
ffffffff810b8fc0 B uvm_swap_data_lock
ffffffff810b8fc8 B uvm_kentry_lock
ffffffff810b8fd0 B uvm_fpageqlock
ffffffff810b8fd8 B uvm_pageqlock
ffffffff810b8fe0 B uvm_kernel_object
----------------
All these locks false-share a cacheline. In particular fpagqlock is
obstructing uvm_pageqlock.
Attempt to run zeroing performs mutex_tryenter. It uncoditionally does
lock cmpxchg which dirties the cacheline, thus even if zeroing would
end up not being performed the damage was already done. Chances are
succesfull zeroing is also a problem, but that I did not investigate.
Doing #if 0'ing the uvm_pageidlezero call in the idle func shaved about 2
seconds real time:
589.02s user 792.62s system 2541% cpu 54.365 total
This should definitely be disabled for amd64 altogether and probably
removed in general.
3. false sharing
Followed the issue noted earlier I __cacheline_aligned aforementioned
locks. But also moved atomically updated counters out of uvmexp.
uvmexp is full of counters updated with mere increments possibly by
multiple threads, thus the issue of this obj was not resolved.
Nonetheless, said annotations applied combined with the rest give the
improvement mentioned earlier.
======================================
Here is a flamegraph from a fully patched kernel:
https://people.freebsd.org/~mjg/netbsd/build-kernel-j40.svgAnd here are top mutex spinners:
59.42 1560022 184255.00 ffffe40138351180 <all>
57.52 1538978 178356.84 ffffe40138351180 uvm_fault_internal+7e0
1.23 8884 3819.43 ffffe40138351180 uvm_unmap_remove+101
0.67 12159 2078.61 ffffe40138351180 cache_lookup+97
(see
https://people.freebsd.org/~mjg/netbsd/build-kernel-j40-lockstat.txt )
Note that netbsd`0xffffffff802249ba is x86_pause. Since the function
does not push frame pointer it is shown next to the actual caller, as
opposed to above it. Sometimes called functions get misplaced anyway,
I don't know why.
1. exclusive vnode locking (genfs_lock)
It is used even for path lookup which as can be seen leads to avoidable
contention. From what I'm told the primary reason is ufs constructing
some state just in case it has to create an inode at the end of the lookup.
However, since most lookups are not intended to create anything, this
behavior can be made conditional. I don't know the details, but ufs on
FreeBSD most certainly uses shared locking for common case lookups.
2. uvm_fault_internal
It's shown as the main waiter for a vm obj lock. The flamegraph hints
the real problem is with uvm_pageqlock & friends taken elsewhere, while
most page fault handlers serialize on the vm obj lock, while the holder
waits for uvm_pageqlock.
3. pmap
It seems most issues stem from slow pmap handling. Chances are there are
perfectly avoidable shootdowns and in fact cases where there is no need
to alter KVA in the first place.
4. vm locks in general
Most likely there are trivial cases where operations can be batched,
especially so on process exit where there are multiple pages to operate
on.
======================================
I would like to add a remark about locking primitives.
Today the rage is with MCS locks, which are fine but not trivial to
integrate with sleepable locks like your mutexes. Even so, the current
implementation is significantly slower than it has to be.
First, the lock word is read twice on entry to mutex_vector_enter - once
to determine the lock type and then to read the owner.
Spinning mutexes should probably be handled by a different routine.
lock cmpxchg already returns the found value (the owner). It can be
passed by the assembly routine to the slow path. This allows to make
an initial pass at backoff without accessing the lock in the meantime.
In face of contention the cacheline could have changed ownership by the
time you get to the read, thus using the value we already saw avoids
spurious bus transactions. Given low initial spin loop it should not have
negative effects.
backoff parameters were hardcoded last decade and are really off even
when looking today's modest servers. For kicks I changed the max spin
count to 1024 and in a trivial microbenchmark of doing dup2 + close
in 40 threads I got almost double the throughput.
Interestingly this change caused a regression for kernel build.
I did not investigate, I suspect the cause was that the vm obj lock
holder was now less aggressive on trying to grab the lock and that
caused problmes for everyone else waiting on the vm obj lock.
The spin loop itself is weird in the sense that instead of just having
the pause instruction embedded it calls a function. This is probably
unnecessarily less power/other thread friendly than it needs to be.
