Subject: Re: Interrupt, interrupt threads, continuations, and kernel lwps
To: None <firstname.lastname@example.org>
From: Bill Studenmund <email@example.com>
Date: 02/21/2007 21:48:25
Content-Type: text/plain; charset=us-ascii
On Wed, Feb 21, 2007 at 05:57:41PM -0800, firstname.lastname@example.org wrote:
> In message <20070221234224.GF26468@netbsd.org>,
> Bill Studenmund writes:
> >On Wed, Feb 21, 2007 at 10:51:03PM +0000, Andrew Doran wrote:
> >> What I want to do will on x86 add 29 arithmetic instructions to the
> >> interrupt path (as of now). That's means in the common, non blocking c=
> >> it's ~free. I'd be pretty bummed if we undo some of that and the reaso=
> >> for doing so is to make the system a better fit for ~20 year old syste=
> Andrew, just what will trigger the blocking (expensive) case?
> And how expensive is it?
I'm not Andy, but as I understand it (and I mention this so I'll be=20
corrected if I'm wrong), the blocking case would be where another CPU=20
holds the mutex you need to acess the hardware. i.e. CPU 3 is configuring=
a piece of hardware then CPU 2 receives an interrupt from that device.
> Can I offer the following example? Suppose for the sake of discussion,
> that I'm running IPsec traffic with FAST_IPSEC, and handling a modest
> 50,000 packets per second. Suppose further that there's also always
> userland processes ready to run.
> My first guess (perhaps overly naive) on how thread-per-interrupt
> would work is something like this:
> (...new IPsec'd packet comes in, asserts NIC interrupt)
> 1. switch from user to NIC interrupt thread
> (... NIC calls ether_input which demuxes packet, enqueues on
> protocol input routine, requests softint processing, blocks)
> 2. switch from NIC hardware thread to softint thread
> (... OCF submits job, blocks ...)
> 3. switch from softint to user=09
> (... crypto hardware finishes, requests interrupt...)
> 4. switch from user to crypto-interrupt thread
> (... crypto driver calls OCF which wakes up softint processing...)
> 5. switch to softint thread, process cleartext packet
> (... done with local kernel packet processing, softint thread ...)
> 6. switch back to user.=20
> [[Let's ignore, for now, whether there'd be a single softint thread,
> or a separate thread for sofnet, or multiple softnet threads.]]
> Is the above roughly right? If it is, then (to steal a comment I made
> to Sam Leffler early in FAST_IPSEC development): 300,000 context
> switches/sec are _not_ your friend. And even if the NIC->softint and
> crypto->softint are a same-VM-context switch (and thus cheap, or at
> least optimizable as kthread-to-kthread), the cost of rescheduling is
> high relative to the work done per packet. And remember, the cost is
> not just instructions, but the additional cache-footprint (and thus
> evictions) of all these switches. Ditto for TLB state; doubly so for
> arches with software-filled TLBs.
> So I assume I'm missing something; can you clue me in?
I don't think that's a function of interrupts-as-threads. I think our=20
current code will do the same thing now, as will anything where one packet=
=3D=3D one interrupt.
I think this is a reason for interrupt mitigation in NICs and, if=20
possible, in crypto cards. And/or for doing something completely different=
for handling packet reception, as Jason hinted at.
> >Ok, that sounds good. How do we make this work on SPARC, m68k, and VAX?
> >Right now, interrupts-as-threads sounds like a total loss for them. If we
> >can turn it into something that's "about as good" then I think you will
> >just get a green light. :-)
> Bill, you mean "about as good as NetBSD is now", right?=20
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.3 (NetBSD)
-----END PGP SIGNATURE-----