I've been reading this topic and have kept silent while I mull it over.
I've also been thinking about TCP Offload Engine's (TOE's) recently.

In the past, I've proposed that we change our network drivers to do the
packet de-queuing at soft IPL, do away with netisr routines, and call the
network protocols directly to deal with incoming packets.  I still think
that's the right thing to do.  The advantage of doing processing this way
is that if we get packet drops, they happen at the hardware FIFO instead
of the ipintrq.  This means the processor doesn't waste valuable cycles
dealing with packets it would have to toss.

However, it seems to me that the run-to-completion method we are using
in the current TCP/IP stack needs to be changed.  With the advent of
better TOEs, I think we need to adapt our TCP/IP stack to efficiently
take advantage of the TOE's.  This means that we allow the network card
to segment and transmit data and reassemble input.  It also means sharing
state with the TOE.

The primary purpose of TOE is to allow a CPU to deal with high-traffic
flows in larger chunks than could be done on a per-packet basis.  This
results in less overhead per packet and thus more CPU cycles for other
things.

Once we have a framework for dealing with TOE's, I think we should
make use the new TOE-aware stack.  If we treat each CPU fetching packets
off an interface as a TOE, we can let it do the minimum re-assembly and
processing.  This will keeps processing time down and the caches hot.

Once the receiving interface is idle (or sufficient data is there to be
made available to the user), the needed parts of the TCP protocol engine
can be run to deal the data.

This could allow the per-packet stuff to happen on CPU while the protocol
processing happens on another.


-- 
Matt Thomas                     email: matt@3am-software.com
3am Software Foundry              www: http://3am-software.com/bio/matt/
Cupertino, CA              disclaimer: I avow all knowledge of this message.