Subject: Re: Interrupt, interrupt threads, continuations, and kernel lwps
To: None <firstname.lastname@example.org>
From: Jason Thorpe <email@example.com>
Date: 02/21/2007 21:24:16
On Feb 21, 2007, at 5:57 PM, firstname.lastname@example.org wrote:
> (...new IPsec'd packet comes in, asserts NIC interrupt)
> 1. switch from user to NIC interrupt thread
> (... NIC calls ether_input which demuxes packet, enqueues on
> protocol input routine, requests softint processing, blocks)
I think eventually we won't have this push-up-the-stack's-throat model
anymore. A better solution is for the NIC interrupt to simply
schedule a puller-thread to grab the packet and process it to
completion without having to reschedule anything.
> 2. switch from NIC hardware thread to softint thread
> (... OCF submits job, blocks ...)
> 3. switch from softint to user
> (... crypto hardware finishes, requests interrupt...)
> 4. switch from user to crypto-interrupt thread
> (... crypto driver calls OCF which wakes up softint processing...)
> 5. switch to softint thread, process cleartext packet
> (... done with local kernel packet processing, softint thread ...)
> 6. switch back to user.
> [[Let's ignore, for now, whether there'd be a single softint thread,
> or a separate thread for sofnet, or multiple softnet threads.]]
> Is the above roughly right? If it is, then (to steal a comment I made
> to Sam Leffler early in FAST_IPSEC development): 300,000 context
> switches/sec are _not_ your friend. And even if the NIC->softint and
> crypto->softint are a same-VM-context switch (and thus cheap, or at
> least optimizable as kthread-to-kthread), the cost of rescheduling is
> high relative to the work done per packet. And remember, the cost is
> not just instructions, but the additional cache-footprint (and thus
> evictions) of all these switches. Ditto for TLB state; doubly so for
> arches with software-filled TLBs.
> So I assume I'm missing something; can you clue me in?
>> Ok, that sounds good. How do we make this work on SPARC, m68k, and
>> Right now, interrupts-as-threads sounds like a total loss for them.
>> If we
>> can turn it into something that's "about as good" then I think you
>> just get a green light. :-)
> Bill, you mean "about as good as NetBSD is now", right?