Subject: Re: Interrupt, interrupt threads, continuations, and kernel lwps
To: Jason Thorpe <thorpej@shagadelic.org>
From: Andrew Doran <ad@netbsd.org>
List: tech-kern
Date: 02/21/2007 22:51:03
Hi Jason,
On Wed, Feb 21, 2007 at 10:01:44AM -0800, Jason Thorpe wrote:
> The way I see it, interrupt threads are an attempt at solving the
> synchronization problem from the wrong direction. Instead, I think
> the right approach is to fully split every driver into a "top
> half" (runs with thread context, possibly as a high-priority kernel
> thread) and "bottom half" (runs in interrupt context).
>
> Furthermore, I believe that the bottom half should manipulate state
> that is local only to the instance of the driver associated with the
> device that is interrupting (and have that state be spin-mutex
> protected).
>
> This approach would have a few advantages:
>
> 1- Simplicity. Interrupt dispatch would be largely as it is today.
Interrupt dispatch will be simpler yes. Device drivers would become more
complicated, and they are more numerous.
> 2- Speed. Because the low-level interrupt dispatch code could be
> simple and avoid magic, it could be very fast, which would help
> devices that are particularly sensitive to interrupt latency.
If you have a device that's particularly sensitive to interrupt latency,
then the two level model is a good one, I agree completely. There is
nothing to prevent that model from being used where it's a good fit.
For the general case, I don't agree though. On recent processors with long
instruction pipelines, serializing control operations are really expensive.
x86 chips from 5 years ago are faster in this regard than the current Intel
offerings - in real time, not clock cycles. I'm keen to avoid that kind of
"funneling" of work unless it's really neccessary - meaning, it's existence
actually has a justifiable benefit, So I have been trying to eliminate these
kinds of operations where they are unnecessary, e.g: during lock release,
during splx(), during syscalls, in the locking scheme devised for the
scheduler and so on.
What I want to do will on x86 add 29 arithmetic instructions to the
interrupt path (as of now). That's means in the common, non blocking case,
it's ~free. I'd be pretty bummed if we undo some of that and the reasoning
for doing so is to make the system a better fit for ~20 year old systems.
> 3- Portability. Because the low-level interrupt dispatch would work
> basically like it does today, we know it will work on all of our
> extant platforms.
Yup, modifing the interrupt handling is tricky..
> I think there are some particularly nasty "gotchas" with
> interrupt-dispatch-as-continuation-or-thread that can be hard to fix on
> platforms like VAX (especially) or even SPARC and m68k.
Vax and m68k we can deal with. The interrupt LWP scheme probably doesn't
suit them particularly well, but it is a good match for current offerings.
> 4- Consistency. We already have this sort of model "sort of" today;
> consider serial drivers that have hard-interrupt handlers that run at
> extremely high priority to read data into a local ring buffer and then
> schedule a soft interrupt to do the TTY processing. What I'm looking
> for here is to formalize this for EVERY kind of device. This will
> make it easier to write drivers for NetBSD. We could even provide
> some API to help people write drivers that conform to the model.
Mmm.. No sale on the consistency ticket, sorry. :-)
Cheers,
Andrew