Re: Interrupt handlers and mutex

[David Holland <dholland-tech%netbsd.org@localhost>]

On Thu, Dec 31, 2009 at 01:52:48PM -0600, Frank Zerangue wrote:
 > Help request -- Mutex(9) indicates that mutex replaces the spl(9) system.

Here are some general (non-NetBSD-specific) answers based on
underlying principles that will hopefully explain the situation better.

 > (1) When writing an interrupt handler, should the handler acquire a
 > spin mutex before modifying some IO that may be accessed also by a
 > LWP?

Yes. In general, in a multiprocessor kernel, the interrupt handler
will not necessarily run on the same CPU that has been posting
requests to the device... and in general more than one CPU may be
doing that. It is prohibitively expensive (as well as generally
undesirable) to disable interrupts for all CPUs at once. Therefore,
disabling interrupts only affects the current CPU, so in order to keep
everything from being tied in knots, you need one or more locks. And
because you can't sleep in an interrupt handler, these must be
spinlocks.

Spinlocks that are used from interrupt handlers must themselves also
disable interrupts. Otherwise, if a thread holding the spinlock is
interrupted by an interrupt handler that tries to acquire the same
spinlock, you get a deadlock.

For this reason, in essentially all multiprocessor systems, the
spinlocks that you use for mutual exclusion in interrupt code disable
and re-enable interrupts for you. In NetBSD each mutex can have an
interrupt level associated with it; if that interrupt level is not
IPL_NONE, acquiring the mutex raises the current interrupt level and
releasing the mutex lowers the current interrupt level.

And in turn, in such systems one generally never sees or uses the
splfoo() functions except in code that hasn't yet been
multiprocessorized.

Currently in NetBSD the interrupt level is only lowered to zero and
only when all spinlocks have been released, instead of any time the
necessary interrupt level drops. This is to avoid complexity when
spinlocks are not released in order, and it mostly makes no practical
difference. (It isn't a good idea to embed e.g. a small block using an
IPL_HIGH mutex inside a large block using a lower-interrupt-level
mutex. But then again, it also isn't a good idea to have a large block
disabling interrupts or using a spinlock anyway.)

NetBSD also has "soft interrupts" that have more process context than
ordinary interrupt handlers; instead of borrowing the context of
whatever's running when the interrupt arrives, they run on dedicated
kernel threads; this means they can sleep to acquire mutexes. AIUI,
the intended design is that most hard interrupts will do as little
work as possible and trigger a soft interrupt to do the rest; this
reduces the number of mutexes used from real interrupt handlers and
reduces the overall amount of spinning. I'm not entirely up on the
exact details at the moment and hopefully someone else will clarify if
there are questions.

 > (2) What happens when the interrupt handler cannot acquire the
 > mutex? Will the LWP that holds it ever be able to run again?

Define "cannot acquire". If the mutex is held by some thread on
another processor, the interrupt handler will spin until it's
released. If it's never released, the interrupt handler will spin
forever, but that's not supposed to happen.

If the mutex is held by some thread on the same processor, then you're
deadlocked. This is why you have to disable interrupts before
attempting to get a spinlock that's used from an interrupt handler.
(Note: the exact details depend somewhat on the implementation. But
you don't want to be writing code that pushes the boundaries.)

 > (3) Will a LWP that holds a spin mutex be pre-empted by the scheduler?

In general, that depends on the interrupt level associated with the
spin mutex. 

-- 
David A. Holland
dholland%netbsd.org@localhost