Re: Surprising observations on the NetBSD TCP stack

[Frank Kardel <kardel%netbsd.org@localhost>]

Hi Greg!

On 2025-10-09 00:34, Greg Troxel wrote:
> Frank Kardel <kardel%netbsd.org@localhost> writes:
>
> > Surprising observations on the NetBSD TCP stack
> Sounds like mlelstv@ has found some things to fix, and pointed you at
> receive buffers.

receive buffers improve a bit.

I played with the buffersizes. It is still asymmetric between gateway 
and notebook.

notebook -> gateway

[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  1.09 GBytes   935 Mbits/sec 0 sender
[  5]   0.00-10.00  sec  1.09 GBytes   934 Mbits/sec                  
receiver

gateway -> notebook (iperf3 -R)

[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.01  sec  1.02 GBytes   871 Mbits/sec 0 sender
[  5]   0.00-10.01  sec  1.01 GBytes   871 Mbits/sec                  
receiver

> > I just got my Fiber 1GBit/sec internet connection (don't ask).
> It's funny that you say don't ask.  It's hard to guess if you mean
> "don't ask why it took me so long" or "don't ask why I got 1 Gb/s
> instead of 400 Mb/s", or something else.  With my fiber provider, 300
> Mb/s is less money than 1 Gb/s so I went with the lower speed.  I know
> people where there's just a price for 1 Gb/s and that's all there is too
> it.
>
> A long way of saying "getting 1 Gb/s fiber would be unremarkable without
> your parenthetical remark"!

Well Fiber is a long story. At my location near Frankfurt am 
Main/Germany I have a signed contract since 2021.

That provider had some serious rollout hickups. The offering was good 
though 1Gb/s symmetric for around 100€.

At my current local near Kiel Germany signing to activation took 2.5 
months and 1Gb/s down 250 Mb/up fixed IP is around 96€.

Higher upload rate cost more an brings some more static IPs.

A college in Bad Vilbel waits for Fiber now for a year and has at least 
the hole in the wall now - waiting again.


> > The windows test shows that the Fiber connection lives up to its
> > performance claims.
> > The NetBSD performance is not competative at all - barely able to keep
> > up with ADSL modems, though
> > the send rate is ok.
> A big issue in speed testing (besides buffer tuning, but that's about
> performance of a single TCP connection more than testing) is whether
> there is one TCP connection or multiple.  I use iperf3 and it can run
> multiple connections in parallel.  Some browser speed tests use multiple
> connections, e.g. speedtest.net.
Yes.
> > When running openSuse Linux in qemu on the NetBSD host we reach 5
> > times higher download rates.
> That's a huge clue, because the NetBSD driver is still sending the
> actual packets on the interface.  That basically says that the TCP
> implementation OR the buffer sizing is the issue.

The current state is

It is still asymetric between gateway and notebook.

notebook -> gateway

[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  1.09 GBytes   935 Mbits/sec 0 sender
[  5]   0.00-10.00  sec  1.09 GBytes   934 Mbits/sec                  
receiver

gateway -> notebook (iperf3 -R)

[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.01  sec  1.02 GBytes   871 Mbits/sec 0 sender
[  5]   0.00-10.01  sec  1.01 GBytes   871 Mbits/sec                  
receiver

> > Any ideas what to look for?
> Two things to look at:
>
>    netstat -s, before and after your test, and diff -u, so that you see
>    the counters changing that you expect, and the counters that changed
>    that you didn't expect.
>
>    pkgsrc/graphics/xplot (and xplot-devel) come with a script to process
>    tcpdump output to a plot showing the data and ack lines.  The
>    distributed script probably isn't right for current tcpdump which
>    doesn't have a stable otuput format.  But this is by a wide margin the
>    best way to understand TCP behavior.  You can see missed data packets,
>    dupacks, sack.  It's clear if the window is too small.  Feel free to
>    ask for offlist help using this.
Yes, I thought about xplot. I see what I can find. I may not be able to 
do tests for a while though.
> Long ago, I was looking into some TCP issues, and didn't quite figure it
> out, but it <very very fuzzy>seemed there were edge cases in ack
> processing that weren't quite right</>.

I think the SACK handling is also broken. There may be a bug from me 
some years ago.

Frank