Subject: Re: Interrupt, interrupt threads, continuations, and kernel lwps
To: None <firstname.lastname@example.org>
From: Bill Studenmund <email@example.com>
Date: 02/22/2007 20:27:56
Content-Type: text/plain; charset=us-ascii
On Thu, Feb 22, 2007 at 04:49:40PM -0800, firstname.lastname@example.org wrote:
> In message <20070222174642.GB24922@netbsd.org>, Bill Studenmund writes:
> >I think the problem is you've assumed an implementation, and specifically
> >you've assumed one other than what Andy was suggesting.
> I came to that conclusion after re-reading Andrew's reply. i think
> the problem is terminology. I'm not used to seeing taking an
> interrupt called a "context switch" instead of, well, taking an
> interrupt or a trap.
Well, it is. :-) It is one form of context switch, and all of the=20
context-switchy things it does are a main part of its expense.
> >My understanding is that Andy has figured out a way to have, at least on
> >x86, the interrupt handler borrow the context of the interrupted thread.
> >So the interrupt context switch is also the context switch to the thread.
> >That's why he said it was the same as what we do now.
> That's an unfortunate choice of terminology. it'S inviting confusion
> to say that a new approach does just what we did before, when it
> doesn't, and the details matter.
For the common case, we will have almost the same behavior as now, except=
that we can take mutexes in an interrupt handler.
As I understand it, the difficulty comes in when the mutex is held by a=20
thread that is not running. In that case, the interrupt handler blocks. In=
that case, the interrupt handler has to be blocked, and the interrupt has=
to be disabled/ignored until serviced. For systems with a PIC, we disable=
the interrupt and cope.
For systems without a PIC, like many m68k systems (and VAX as I understand=
matt), we would need to disable said interrupt in the CPU (set the chip's=
interrupt level) until the service routine completes.
As noted, this will hopefully be a rare occurrence, where a thread that=20
holds an interrupt mutex is no longer on a processor. We can structure our=
code so that this is very unlikely, if not impossible.
Note that in the case of the mutex being locked but the thread being on=20
another processor, we will just spin-wait. That's a feature of the locking=
we copied from Solaris. I expect that this _will_ happen, but we then get=
the exact same behaviors as if we had spinlocks.
> >Let's try it and see! Rather than discuss hypotheticals, let's get #s.
> >Either the numbers will show little change, which will make ALL of us
> >happy, or the numbers will show a marked decrease, which will displease
> >ALL of us. :-)
> Numbers would be interesting. But what kind of numbers? Single-CPU?
> Multi-CPU? Or CPUs with (gack) virtually-indexed caches and TLBs
> without address-space-IDs? I don't see how Andrew's scheme would
> really work there, but maybe I'm missing something.
Single ande multi-CPU to be sure. But as for the other stuff, it's mostly=
what we have now. So I don't see how we will have radially-different=20
We should test stuff over time to make sure we don't do something stupid,=
to be sure!
> The other issue I see is that Andrew is assuming that blocking (in his
> sense, meaning doing the full work of the deferred context switch) is
> a rare case. I don't buy that, not for networking. Look at the
> resources put into making FreeBSD a fine-grained kernel, and notice
> that FreeBSD-6 still has one big lock around the network stack and
> socket-layer code.
I don't think that will mater for interupt handlers. Yes, I think there's=
a lot of work to do for the networking stack. But I think it'll be=20
As above, trying to get the lock while a thread running on another CPU has=
it turns into a spin wait. That's not the slow case. So as long as we=20
don't hold a mutex we take in interrupt context while we do something else=
that can block, we will NOT trigger the slow path in an interrupt handler.=
We don't sleep while holding SPL now (or we aren't supposed to), so a very=
direct 1:1 translation should be fine.
We can run into issues, of course, and we need to look for them. Like a
case where the process that we interrupted holds a mutex that the thread
that holds the interrupt mutex is waiting for. That's a deadlock if we
don't detect it & trigger slow-path. It can also be argued that that's a
bad design. :-)
While I think we need to think about some architectures more, I believe=20
this is a good direction to go in.
Also, I remember you wanting to work on MP-ing the network stack. newlock2=
+ interrupt handling will be a definite springboard for that!
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.3 (NetBSD)
-----END PGP SIGNATURE-----