Subject: Re: SMP re-entrancy in kernel drivers/"bottom half?"
To: Jonathan Stone <email@example.com>
From: Jason Thorpe <firstname.lastname@example.org>
Date: 12/17/2003 18:50:27
Content-Type: text/plain; charset=US-ASCII; format=flowed
On Dec 17, 2003, at 2:58 PM, Jonathan Stone wrote:
> Elementary: we have to maintain the invariant ``at most one CPU at or
> above any given [hardware] prioritly level' or we lose the
> synchronization semantics of SPLs (higher SPls than the hypothetical
> SMP-safe interrupt-routine driver entrypoints).
I don't think that's the way we want to move the kernel, in general.
There's also the question of what "above" is. Technically, splnet is
not "above" splbio, but it is allowed to be, by convention, in order to
allow network devices to have better interrupt latency than disk
Think of this this way -- splbio and splnet lock two different sets of
data structures. They are orthogonal, and there is no defined "locking
order" for moving between them.
We currently have a small set of interrupt-frobbing-simplelocks in the
kernel that are implemented in an ad hoc way:
s = splfoo();
/* manipulate a data structure that foo_slock protects */
This is all usually hidden inside of macros.
The logic goes this way:
1. The data structure is actually protected by foo_slock. It is not
actually protected by splfoo().
2. Because foo_slock protects the data structure, that prevents other
CPUs from getting at the data structure while we have it.
3. Because foo_slock can be acquired in interrupt context, we must
prevent *our* CPU from running that interrupt code path while we
acquire/hold the lock, otherwise deadlock could result. Therefore,
we go do splfoo() before we acquire the lock, and drop ipl after
we release it.
The branch I was working on basically implemented a couple of things,
in the Solaris style:
1. Adaptive and interrupt-safe mutexes.
An adaptive mutex is really more like lockmgr(LK_EXCLUSIVE) because
it is a sleeping mutex (the "adaptive" part is to spin if the thread
that holds the mutex is running on another CPU, in the hope that it
will be finished with it soon, thus saving at least 2 context
An interrupt-safe mutex is one that does precisely the same dance
that my ad hoc example above does. These types of mutexes would
always spin, in our kernel.
Both of these mutexes are contained in a single kmutex_t and
2. Reader-writer locks. Like lockmgr(LK_SHARED) and
combined, they are lighter-weight than the lockmgr equivalents.
3. Turnstiles. The context switching primitive used to support the
implementation of mutexes and rwlocks.
-- Jason R. Thorpe <email@example.com>
content-type: application/pgp-signature; x-mac-type=70674453;
content-description: This is a digitally signed message part
content-disposition: inline; filename=PGP.sig
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.3 (Darwin)
-----END PGP SIGNATURE-----