Subject: Re: multiple rx/tx rings and interrupt delivery on newer nics
To: Darren Reed <darrenr@fastmail.net>
From: Cherry G. Mathew <cherry.g.mathew@gmail.com>
List: tech-net
Date: 06/15/2007 04:14:39
On 6/14/07, Darren Reed <darrenr@fastmail.net> wrote:
> Thor Lancelot Simon wrote:
> > On Wed, Jun 13, 2007 at 01:12:23AM -0700, Darren Reed wrote:
> >
> >> Has anyone started to think about how NetBSD can take advantage
> >> of NICs that have a larger number of rx/tx descriptor rings,
> >> and/or MSI interrupts?
> >>
> >
> > Think, yes -- do, no.
> >
> >
> >> For example, would this get tied in with ALTQ or something else?
> >>
> >
> > Ooh, I hope not.  ALTQ is slow.
> >
> >
> >> Should I be able to dedicate a rx/tx ring pair to http traffic
> >> and another to ssh, etc?
> >>
> >
> > I've been thinking about this -- I know Solaris can do this now -- and
> > I have to say I'm somewhat skeptical.  Does this really save much overhead?
> >
>
> It's not just about speeding up delivery to an application, but being
> able to deliver a certain QOS too because you don't have to deal
> with each queue equally.
>
> For local delivery, if you have specific rings being served by specific
> cores/cpus then it stands to reason you should see some locality of
> execution style benefits.
>
> > I might rather have the packets all land in one ring, tagged for what
> > rule they've matched in classification.  Now, on the other hand, what would
> > be of great performance benefit for _my_ application, at least, on a
> > multiprocessor, would be the ability to use multiple rings according to
> > a slightly different set of classification parameters, e.g. destination
> > IP address.
>
> I believe the standard matching for packets is a 5-tuple:
> source address, destination address, protocol, source port, destination port
>
> > That way you can avoid stomping the cache of CPU A when
> > packets come in that will only ever be touched by the network stack on
> > CPU B.
> >
>
> Right, see above.
>
> >> ...but to do any of that will require some sort of framework.
> >>
> >
> > Yes.  Do you know how what Sun recently did has been working out?
> >
>
> Yes, I do...the people working on the project are my co-workers, so
> this is about as involved in any NetBSD stuff as I can realisticly be.
> The project to do this is known as "crossbow" and is hosted on
> the opensolaris.org website at:
> http://www.opensolaris.org/os/project/crossbow/
>
> Darren
>
>

http://www.xensource.com/files/xensummit_4/xen-ny-summit-smartnic_Pratt.pdf


-- 
~Cherry