Re: Interrupt handlers and mutex

[Frank Zerangue <frank.zerangue%gmail.com@localhost>]

Thanks you David -- excellent explanation.

Frank

On Dec 31, 2009, at 5:51 PM, David Holland wrote:

> On Thu, Dec 31, 2009 at 01:52:48PM -0600, Frank Zerangue wrote:
>> Help request -- Mutex(9) indicates that mutex replaces the spl(9) system.
> 
> Here are some general (non-NetBSD-specific) answers based on
> underlying principles that will hopefully explain the situation better.
> 
>> (1) When writing an interrupt handler, should the handler acquire a
>> spin mutex before modifying some IO that may be accessed also by a
>> LWP?
> 
> Yes. In general, in a multiprocessor kernel, the interrupt handler
> will not necessarily run on the same CPU that has been posting
> requests to the device... and in general more than one CPU may be
> doing that. It is prohibitively expensive (as well as generally
> undesirable) to disable interrupts for all CPUs at once. Therefore,
> disabling interrupts only affects the current CPU, so in order to keep
> everything from being tied in knots, you need one or more locks. And
> because you can't sleep in an interrupt handler, these must be
> spinlocks.
> 
> Spinlocks that are used from interrupt handlers must themselves also
> disable interrupts. Otherwise, if a thread holding the spinlock is
> interrupted by an interrupt handler that tries to acquire the same
> spinlock, you get a deadlock.
> 
> For this reason, in essentially all multiprocessor systems, the
> spinlocks that you use for mutual exclusion in interrupt code disable
> and re-enable interrupts for you. In NetBSD each mutex can have an
> interrupt level associated with it; if that interrupt level is not
> IPL_NONE, acquiring the mutex raises the current interrupt level and
> releasing the mutex lowers the current interrupt level.
> 
> And in turn, in such systems one generally never sees or uses the
> splfoo() functions except in code that hasn't yet been
> multiprocessorized.
> 
> Currently in NetBSD the interrupt level is only lowered to zero and
> only when all spinlocks have been released, instead of any time the
> necessary interrupt level drops. This is to avoid complexity when
> spinlocks are not released in order, and it mostly makes no practical
> difference. (It isn't a good idea to embed e.g. a small block using an
> IPL_HIGH mutex inside a large block using a lower-interrupt-level
> mutex. But then again, it also isn't a good idea to have a large block
> disabling interrupts or using a spinlock anyway.)
> 
> NetBSD also has "soft interrupts" that have more process context than
> ordinary interrupt handlers; instead of borrowing the context of
> whatever's running when the interrupt arrives, they run on dedicated
> kernel threads; this means they can sleep to acquire mutexes. AIUI,
> the intended design is that most hard interrupts will do as little
> work as possible and trigger a soft interrupt to do the rest; this
> reduces the number of mutexes used from real interrupt handlers and
> reduces the overall amount of spinning. I'm not entirely up on the
> exact details at the moment and hopefully someone else will clarify if
> there are questions.
> 
>> (2) What happens when the interrupt handler cannot acquire the
>> mutex? Will the LWP that holds it ever be able to run again?
> 
> Define "cannot acquire". If the mutex is held by some thread on
> another processor, the interrupt handler will spin until it's
> released. If it's never released, the interrupt handler will spin
> forever, but that's not supposed to happen.
> 
> If the mutex is held by some thread on the same processor, then you're
> deadlocked. This is why you have to disable interrupts before
> attempting to get a spinlock that's used from an interrupt handler.
> (Note: the exact details depend somewhat on the implementation. But
> you don't want to be writing code that pushes the boundaries.)
> 
>> (3) Will a LWP that holds a spin mutex be pre-empted by the scheduler?
> 
> In general, that depends on the interrupt level associated with the
> spin mutex. 
> 
> -- 
> David A. Holland
> dholland%netbsd.org@localhost