Subject: Re: Interrupt, interrupt threads, continuations, and kernel lwps
To: None <>
From: Bill Studenmund <>
List: tech-kern
Date: 02/22/2007 20:27:56
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable

On Thu, Feb 22, 2007 at 04:49:40PM -0800, wrote:
> In message <>, Bill Studenmund writes:
> [...]
> >I think the problem is you've assumed an implementation, and specifically
> >you've assumed one other than what Andy was suggesting.
> I came to that conclusion after re-reading Andrew's reply.  i think
> the problem is terminology.  I'm not used to seeing taking an
> interrupt called a "context switch" instead of, well, taking an
> interrupt or a trap.

Well, it is. :-) It is one form of context switch, and all of the=20
context-switchy things it does are a main part of its expense.

> >My understanding is that Andy has figured out a way to have, at least on
> >x86, the interrupt handler borrow the context of the interrupted thread.
> >So the interrupt context switch is also the context switch to the thread.
> >
> >That's why he said it was the same as what we do now.
> That's an unfortunate choice of terminology. it'S inviting confusion
> to say that a new approach does just what we did before, when it
> doesn't, and the details matter.


For the common case, we will have almost the same behavior as now, except=
that we can take mutexes in an interrupt handler.

As I understand it, the difficulty comes in when the mutex is held by a=20
thread that is not running. In that case, the interrupt handler blocks. In=
that case, the interrupt handler has to be blocked, and the interrupt has=
to be disabled/ignored until serviced. For systems with a PIC, we disable=
the interrupt and cope.

For systems without a PIC, like many m68k systems (and VAX as I understand=
matt), we would need to disable said interrupt in the CPU (set the chip's=
interrupt level) until the service routine completes.

As noted, this will hopefully be a rare occurrence, where a thread that=20
holds an interrupt mutex is no longer on a processor. We can structure our=
code so that this is very unlikely, if not impossible.

Note that in the case of the mutex being locked but the thread being on=20
another processor, we will just spin-wait. That's a feature of the locking=
we copied from Solaris. I expect that this _will_ happen, but we then get=
the exact same behaviors as if we had spinlocks.

> >Let's try it and see! Rather than discuss hypotheticals, let's get #s.
> >Either the numbers will show little change, which will make ALL of us
> >happy, or the numbers will show a marked decrease, which will displease
> >ALL of us. :-)
> Numbers would be interesting.  But what kind of numbers?  Single-CPU?
> Multi-CPU?  Or CPUs with (gack) virtually-indexed caches and TLBs
> without address-space-IDs?  I don't see how Andrew's scheme would
> really work there, but maybe I'm missing something.

Single ande multi-CPU to be sure. But as for the other stuff, it's mostly=
what we have now. So I don't see how we will have radially-different=20

We should test stuff over time to make sure we don't do something stupid,=
to be sure!

> The other issue I see is that Andrew is assuming that blocking (in his
> sense, meaning doing the full work of the deferred context switch) is
> a rare case.  I don't buy that, not for networking.  Look at the
> resources put into making FreeBSD a fine-grained kernel, and notice
> that FreeBSD-6 still has one big lock around the network stack and
> socket-layer code.

I don't think that will mater for interupt handlers. Yes, I think there's=
a lot of work to do for the networking stack. But I think it'll be=20
different work.

As above, trying to get the lock while a thread running on another CPU has=
it turns into a spin wait. That's not the slow case. So as long as we=20
don't hold a mutex we take in interrupt context while we do something else=
that can block, we will NOT trigger the slow path in an interrupt handler.=
We don't sleep while holding SPL now (or we aren't supposed to), so a very=
direct 1:1 translation should be fine.

We can run into issues, of course, and we need to look for them. Like a
case where the process that we interrupted holds a mutex that the thread
that holds the interrupt mutex is waiting for. That's a deadlock if we
don't detect it & trigger slow-path. It can also be argued that that's a
bad design. :-)

While I think we need to think about some architectures more, I believe=20
this is a good direction to go in.

Also, I remember you wanting to work on MP-ing the network stack. newlock2=
+ interrupt handling will be a definite springboard for that!

Take care,


Content-Type: application/pgp-signature
Content-Disposition: inline

Version: GnuPG v1.4.3 (NetBSD)