Re: Surprising observations on the NetBSD TCP stack

[Frank Kardel <kardel%kardel.name@localhost>]

Thanks for the feedback see below.

On 2025-10-08 19:45, Michael van Elst wrote:
> kardel%netbsd.org@localhost (Frank Kardel) writes:
>
> > I just got my Fiber 1GBit/sec internet connection (don't ask).
> > When testing the throughput with breitbandmessung.de I got a meager
> > 88Mbit/sec download rate.
> There are lots of things that do have an effect.
>
> Download speeds are mostly affected by local buffering.
> Upload speeds are mostly affected by congestion control.
I played with the buffersizes. It is still asymetric between gateway and 
notebook.

notebook -> gateway

[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  1.09 GBytes   935 Mbits/sec 0             sender
[  5]   0.00-10.00  sec  1.09 GBytes   934 Mbits/sec                  
receiver

gateway -> notebook (iperf3 -R)

[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.01  sec  1.02 GBytes   871 Mbits/sec 0             sender
[  5]   0.00-10.01  sec  1.01 GBytes   871 Mbits/sec                  
receiver

> > Testing local connectivity on the gigbit interfaces found:
> > Â Â Â Â Â Â Â  gateway->notebook xfer rate: 61201.87 KB/sec !!! should be more
> > like 113000 KB/sec
> > Â Â Â Â Â Â Â  notebook->gateway xfer rate: 113958.41 KB/sec
> > This begs the question what is limiting the transfer rate asymmetrically?
> Usually different network interfaces on either side, bad or nonexistent
> interrupt mitigation, bad use of hardware queues, etc..

Maybe there are issues in the igc driver. sending from there does not 
look completely perfect.

> This is two NetBSD machines (Xeon+wm vs Ryzen+wm) on the local network.
>
> iperf
> [  7]   0.00-10.00  sec  1.06 GBytes   914 Mbits/sec    0             sender
> [  7]   0.00-10.04  sec  1.06 GBytes   909 Mbits/sec                  receiver
>
> iperf -R
> [  7]   0.00-10.02  sec  1.07 GBytes   920 Mbits/sec    0             sender
> [  7]   0.00-10.00  sec  1.07 GBytes   919 Mbits/sec                  receiver
>
>
> > The NetBSD performance is not competative at all - barely able to keep
> > up with ADSL modems, though
> > the send rate is ok.
> I don't have a fibre connection, so testing on the internet gets
> difficult.
>
> Instead I put an RPI running Linux as a "WAN simulator" between
> two NetBSD hosts to analyze the issues. I now have a couple of
> changes in the TCP stack and workaround in the wm driver to
> get almost 1Gbit/s over a simulated WAN with 200ms TTL.
>
> This also helped a bit with internet upload speed, but at
> 40Mbit/s (VDSL) the effect is small.
Good to hear.
> > net.inet.tcp.recvbuf_max = 262144
> > net.inet.tcp.sendbuf_max = 262144
> You may need larger buffers (and larger increments).
I now have

net.inet.tcp.keepinit = 150
net.inet.tcp.recvbuf_auto = 1
net.inet.tcp.recvbuf_inc = 131072
net.inet.tcp.recvbuf_max = 1048576
net.inet.tcp.sendbuf_auto = 1
net.inet.tcp.sendbuf_inc = 131072
net.inet.tcp.sendbuf_max = 1048576