--pgp-sign-Multipart_Fri_Jun_17_02:01:38_2005-1
Content-Type: text/plain; charset=US-ASCII

>>>>> "js" == Jonathan Stone <jonathan@dsg.stanford.edu> writes:

    js> suggest that any attempt at policy-based routing be either
    js> sufficiently fast to handle hundreds of thousands of packets
    js> per second with minimal overhaed; or be configurable into a
    js> mode where performance is acceptable; or be roff by default.

I did some secret testing with a proprietary packet generator
(Imperfect Networks ``ThreatEX''), and in my Alpha PWS433, I found the
machine froze up (became completely unresponsive at the console, and
lost RTC time) at:

    interface      pps
    tlp            6500
    bge            70000 - 90000

For the tlp, the serial console filled with ``receive overrun''---no
rate limiting of the error messages, apparently.  For bge, I'd tuned
the hw.bge.rx_lvl to 5, the highest value it would accept.  For the
tlp, the thing collapsed at about the same spot whether it was
forwarding the traffic or blocking it.  For bge, I had only one card,
so I had to block the traffic---otherwise it'd be forwarded out a tlp,
and that brought the pps down to 6500 again.

I tried the test with pf both on and off, and it didn't make any
difference.  What I didn't do is keep track of exactly how many PF
rules were getting evaluated on each packet.  sorry---I only had a
small amount of patience and short time with the packet generator.

so, at least for me, preliminary looks like interrupt handling is
probably costly enough that I can do basically whatever I need to with
the firewall without really caring, but I could be wrong.  Anyway
there's an order of magnitude between these two cards.  It's the
difference between forwarding/blocking a 3Mbit/s SYN flood (useless)
and a 30Mbit/s SYN flood (has some practical value).

I think that enshrining existing code ought to be done _after_ taking
a more holistic look at how many pps a NetBSD router can
forward/block, and finishing the job completely enough that sysadmins
know what cards to buy and what secret variables to set to get good
performance.  What is ``fastforwarding''?  We have it for IPv4 only,
right?  Is it on by default?  Does anything (firewalls?  ALTQ?
hardware checksums?)  implicitly turn off ``fastforwarding''?  Is
there an interrupt mitigation knob for the wm(4) driver as well?
Which is better, wm or bge?  Do any other cards mitigate interrupts?
Are there problems with the _way_ i'm using the cards, ex. if they are
quad port cards and share interrupts, or if they are on a slot that's
behind a PCI-PCI bridge, does that hurt the pps capacity?  Is there
such thing as transmit-side interrupt mitigation, and if so will
ALTQ's token buffer mess it up?  Is there a way to make my box drop
packets rather than stupidly freeze the control plane when pps gets
too high, so I can go to the console and alter firewall rules or bring
down interfaces instead of just waiting for an attack to end, or
manually unplugging a cable to get back control?  Under what
circumstances will adding more CPUs help (on FreeBSD I'm told the
basic rule is ``one CPU per card.''  multiple CPUs won't help the
capacity of a single card.  Is NetBSD this good, or multiple CPUs
don't help at all?).  I think in the spirit of NetBSD's claiming we
optimize things that matter rather than boasting about excessively
clever hand-tuned algorithms in spots that aren't a bottleneck, this
interrupt/softint/ipl thing needs to get better understood before
applying excessive pressure to people working on the routing table,
because it looks pretty clear that interrupts-per-second is the
limiting value right now, not size/scalability of the routing table.

--pgp-sign-Multipart_Fri_Jun_17_02:01:38_2005-1
Content-Type: application/pgp-signature
Content-Transfer-Encoding: 7bit

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.6 (NetBSD)

iQCVAwUAQrJnT4nCBbTaW/4dAQICxgP/eyVKk1GHDsaBoBbi1QlzbBmvU+eUYniz
DcMNpm85EvKNr/VHlUh1X+IHzsiETIkhlqZz9Moyk9eKQlbmmdtMc6Y9COkAKhku
JgnKS0ttCiULY6Iu4RKp4C1epT8T9zcQkwVQw3EhEUBsOIvD8IchegmQfuFhfCbJ
z4KKPra10cg=
=79L2
-----END PGP SIGNATURE-----

--pgp-sign-Multipart_Fri_Jun_17_02:01:38_2005-1--