Subject: Re: multi-core router
To: Bill Stouder-Studenmund <wrstuden@netbsd.org>
From: Darren Reed <darrenr@netbsd.org>
List: tech-net
Date: 11/04/2007 09:57:26
Bill Stouder-Studenmund wrote:
> ...
> So that leads me to two questions for these lists.
>
> 1) How would you design a router to make use of multiple cores. We've 
> talked about different multi-core setups for running the TCP stack on one 
> core and IP on another and so on to sustain 10 GigE TCP connections. But 
> this case is different in that all we're doing is NAT and forwarding.
>   

Depends on how good the NICs are and how you can interact with them.

On a good 10GE NIC, you can do some decent hardware classification into
multiple receive rings...providing the O/S supports that.  Solaris is 
going that
way, so is Microsoft Windows (there was some press release earlier in the
year about them rewriting/redesigning their networking stack to take
advantage of more capable NICs.)

And what is "multiple cores"? 2 or 4 or more?
And is it 1 thread per core or more?
(Think niagara which has 8 cores and 4 threads per core.)

On a 4 core router, why not dedicate a core per descriptor ring used, so for
a 2 port box, you've got 4 active areas (rx port A, tx port A, rx port B, tx
port B.)

But all of this is wasted if you're using pf - this will make your NAT 
single threaded.

So in order to get good multi-core performance, you need to use ipfilter 
or ipfw
(both of these use locking that's more intelligent than giant-lock style.)

Darren