Surprising observations on the NetBSD TCP stack

[Frank Kardel <kardel%netbsd.org@localhost>]

Surprising observations on the NetBSD TCP stack

I just got my Fiber 1GBit/sec internet connection (don't ask).

When testing the throughput with breitbandmessung.de I got a meager
88MSurprising observations on the NetBSD TCP stack

I just got my Fiber 1GBit/sec internet connection (don't ask).

When testing the throughput with breitbandmessung.de I got a meager
88Mbit/sec download rate.
Testing local connectivity on the gigbit interfaces found:
        gateway->notebook xfer rate: 61201.87 KB/sec !!! should be more 
like 113000 KB/sec
        notebook->gateway xfer rate: 113958.41 KB/sec
This begs the question what is limiting the transfer rate asymmetrically?

Remote tests with a Browsertest on firefox gave:
win-notebook (W10) -> breitbandmessung.de (Browsertest Firefox)
12ms    925Mbit/sec download    245 Mbit/sec upload

notebook (NetBSD 10.1_STABLE) -> breitbandmessung.de (Browsertest Firefox)
12ms    88Mbit/sec download     248 Mbit/sec upload

openSuse Linux Leap 15.5 via Qemu/nvmm on notebook -> 
breitbandmessung.de (Browsertest Firefox)
12ms    530Mbit/sec download    247 Mbit/sec upload

The windows test shows that the Fiber connection lives up to its 
performance claims.
The NetBSD performance is not competative at all - barely able to keep 
up with ADSL modems, though
the send rate is ok.

When running openSuse Linux in qemu on the NetBSD host we reach 5 times 
higher download rates.
Running Windows 10 on the notebook hardware reach the same high rates as 
on the win-notebook.

That leaves the question why the receive rate is so low on the NetBSD 10 
notebook. It seems
to be a software issue.

Can this be reproduced?

Is that already fixed in -current?

Any ideas what to look for?

Frank

==========DETAILS===========

Network topology
notebook-------------switch--------FritzBox 7590AX =========> Fiber 
1Gbit down, 250Mbit up
10.200.100.152/25 1G   | |          10.200.100.129/25 1G
                       | |
gateway----------------+ |
10.200.100.148/25 1G     |
                         |
win-notebook W10  1G-----+

TTCP tests

gateway
ttcp-t: buflen=8192, nbuf=16384, align=16384/0, port=5001  tcp -> 
10.200.100.152
ttcp-t: socket
ttcp-t: connect
ttcp-t: 134217728 bytes in 2.14 real seconds = 61241.73 KB/sec +++
ttcp-t: 16384 I/O calls, msec/call = 0.13, calls/sec = 7655.22
ttcp-t: 0.0user 0.2sys 0:02real 11% 0i+0d 722maxrss 0+2pf 9796+1csw
notebook
ttcp-r: buflen=8192, nbuf=16384, align=16384/0, port=5001 tcp
ttcp-r: socket
ttcp-r: accept from 10.200.100.148
ttcp-r: 134217728 bytes in 2.14 real seconds = 61201.87 KB/sec +++
ttcp-r: 19163 I/O calls, msec/call = 0.11, calls/sec = 8947.84
ttcp-r: 0.0user 0.0sys 0:02real 1% 0i+0d 750maxrss 0+2pf 5266+0csw

gateway->notebook xfer rate: 61201.87 KB/sec

notebook
ttcp-t: buflen=8192, nbuf=16384, align=16384/0, port=5001  tcp -> 
10.200.100.148
ttcp-t: socket
ttcp-t: connect
ttcp-t: 134217728 bytes in 1.15 real seconds = 114100.87 KB/sec +++
ttcp-t: 16384 I/O calls, msec/call = 0.07, calls/sec = 14262.61
ttcp-t: 0.0user 0.1sys 0:01real 12% 0i+0d 718maxrss 0+2pf 3102+4csw
gateway
ttcp-r: buflen=8192, nbuf=16384, align=16384/0, port=5001  tcp
ttcp-r: socket
ttcp-r: accept from 10.200.100.152
ttcp-r: 134217728 bytes in 1.15 real seconds = 113958.41 KB/sec +++
ttcp-r: 18393 I/O calls, msec/call = 0.06, calls/sec = 15991.49
ttcp-r: 0.0user 0.0sys 0:01real 5% 0i+0d 760maxrss 0+2pf 9199+0csw

notebook->gateway xfer rate: 113958.41 KB/sec

Breitbandmessung.de tests

win-notebook (W10) -> breitbandmessung.de (Browsertest Firefox)
12ms    925Mbit/sec download    245 Mbit/sec upload

notebook (NetBSD 10.1_STABLE) -> breitbandmessung.de (Browsertest Firefox)
12ms    88Mbit/sec download     248 Mbit/sec upload

openSuse Linux Leap 15.5 via Qemu/nvmm on notebook -> 
breitbandmessung.de (Browsertest Firefox)
12ms    530Mbit/sec download    247 Mbit/sec upload

Technical parameters

notebook
        Lenovo ThinkPad T15p Gen 1
        no packet filters
        no recvq drop
        machdep.cpu.frequency.current = 2600
        net.inet.tcp.recvbuf_max = 262144
        net.inet.tcp.sendbuf_max = 262144
        wm0 at pci0 dev 31 function 6, 64-bit DMA: I219 V (11) Ethernet 
Connection (rev. 0x00)
        wm0: interrupting at msi5 vec 0
        wm0: PCI-Express bus
        wm0: 4096 words FLASH, version 0.4.4
        wm0: RX packet buffer size: 26KB
        wm0: Ethernet address 6c:24:08:74:59:54
        wm0: 0x6a4080<FLASH,PCIE,ASF_FIRM,AMT,WOL,EEE>

win-notebook
        Lenovo E15 Gen 3
        Windows 10 Pro

gateway
        machdep.dmi.processor-version = Intel(R) Celeron(R) J6413 @ 1.80GHz
        machdep.dmi.processor-frequency = 1782 MHz
        igc1 at pci2 dev 0 function 0: Intel(R) Ethernet Controller 
I226-V (rev. 0x04)
        igc1: txrx interrupting at msix3 vec 0
        igc1: txrx interrupting at msix3 vec 1
        igc1: txrx interrupting at msix3 vec 2
        igc1: txrx interrupting at msix3 vec 3
        igc1: link interrupting at msix3 vec 4
        igc1: PHY: OUI 0x984fee model 0x0005, rev. 8, PHY FW version 0x888d
        igc1: NVM image version 2.17, EtrackID 80000308
        igc1: Ethernet address e4:3a:6e:61:c7:44