Re: Surprising observations on the NetBSD TCP stack

[Greg Troxel <gdt%lexort.com@localhost>]

Frank Kardel <kardel%netbsd.org@localhost> writes:

> Surprising observations on the NetBSD TCP stack

Sounds like mlelstv@ has found some things to fix, and pointed you at
receive buffers.

> I just got my Fiber 1GBit/sec internet connection (don't ask).

It's funny that you say don't ask.  It's hard to guess if you mean
"don't ask why it took me so long" or "don't ask why I got 1 Gb/s
instead of 400 Mb/s", or something else.  With my fiber provider, 300
Mb/s is less money than 1 Gb/s so I went with the lower speed.  I know
people where there's just a price for 1 Gb/s and that's all there is too
it.

A long way of saying "getting 1 Gb/s fiber would be unremarkable without
your parenthetical remark"!

> The windows test shows that the Fiber connection lives up to its
> performance claims.
> The NetBSD performance is not competative at all - barely able to keep
> up with ADSL modems, though
> the send rate is ok.

A big issue in speed testing (besides buffer tuning, but that's about
performance of a single TCP connection more than testing) is whether
there is one TCP connection or multiple.  I use iperf3 and it can run
multiple connections in parallel.  Some browser speed tests use multiple
connections, e.g. speedtest.net.

> When running openSuse Linux in qemu on the NetBSD host we reach 5
> times higher download rates.

That's a huge clue, because the NetBSD driver is still sending the
actual packets on the interface.  That basically says that the TCP
implementation OR the buffer sizing is the issue.

> Any ideas what to look for?

Two things to look at:

  netstat -s, before and after your test, and diff -u, so that you see
  the counters changing that you expect, and the counters that changed
  that you didn't expect.

  pkgsrc/graphics/xplot (and xplot-devel) come with a script to process
  tcpdump output to a plot showing the data and ack lines.  The
  distributed script probably isn't right for current tcpdump which
  doesn't have a stable otuput format.  But this is by a wide margin the
  best way to understand TCP behavior.  You can see missed data packets,
  dupacks, sack.  It's clear if the window is too small.  Feel free to
  ask for offlist help using this.

Long ago, I was looking into some TCP issues, and didn't quite figure it
out, but it <very very fuzzy>seemed there were edge cases in ack
processing that weren't quite right</>.