smr(9) and pool_cache_set_smr(9)

[Kevin Bowling <kevin.bowling%kev009.com@localhost>]

Hi,

I am working on making netinet MPSAFE and am using FreeBSD's locking
model as a guide because there is a lot of nuance and it has seen
decades of wear.  The stacks have not diverged fundamentally, and the
approach has proven sound so far where I can apply a similar locking
template to this stack.

In doing so, it is desirable to port Jeff Roberson's smr(9) which is
used in FreeBSD [1].  This is a concurrency primitive that generally
compliments pserialize(9) with different tradeoffs and each are
appropriate for different areas in the kernel.  The biggest difference
is on the write side, where a shared sequence number (default) or
ticks (SMR_LAZY) are used to clear lifecycle hazards that will be
familiar to pserialize(9) users and are comparatively cheap against an
IPI.

On top of this, and inspired by Jeff's UMA integration, I have
implemented a new pool_cache(9) type which uses SMR to manage FIFO
batches of objects with an efficient deferred free based on the safety
guarantees of the algorithm.  This solves a different problem than
PR_PSERIALIZE and is again complementary.

Both of these APIs are immediately useful in the netinet work.
Particularly in_pcbs where we need rapid allocation for new
connections and don't want to hang on to garbage to handle connection
churn or attacks.  There are other candidates that could potentially
use it in various subsystems like file descriptors, namecache, npf
connection tracking etc.

My implementation and usage have minor novelty, I have solved some
weak ordering concerns Jeff called out in the original annoucnement
[1] that matter here since NetBSD runs on a lot more architectures.  I
use SMR_LAZY in both the netinet code and the pool_cache which trades
a small amount of free latency in exchange for reads that match TSO
architectures.

I will post the netinet changes when I am finished to show concrete
usage, but smr and the pool type is self-contained and available for
feedback now:  https://people.freebsd.org/~kbowling/netbsd/smr.patch

[1] https://lists.freebsd.org/pipermail/freebsd-arch/2020-January/019866.html

Regards,
Kevin Bowling