wm + vlan tso4 problem with 10_BETA

[Thor Lancelot Simon <tls%panix.com@localhost>]

Since upgrading my home gateway from an older version of NetBSD 9 to
NetBSD 10.0_BETA, I've been having some throughput issues - forwarding
performance seems to have dropped by between 30% and 50%, in a way
that is highly sensitive to client TCP stack behavior.  More perhaps on
that later.

However this has resulted in my testing every link in my internal network,
to establish a baseline.  In the course of this I discovered that if TSO4
is enabled on a vlan interface which is on wm, transmitting bulk data over
TCP from the NetBSD gateway to a local client (on the same vlan) very
quickly stalls.

Has anyone else seen anything like this recently?

This configuration will stall almost immediately when sending to a local
peer with ttcp -s -t -b $((192 * 1024)) -l 65536:

hotpoint# ifconfig vlan2
vlan2: flags=0x8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
	capabilities=0x7ff80<TSO4,IP4CSUM_Rx,IP4CSUM_Tx,TCP4CSUM_Rx>
	capabilities=0x7ff80<TCP4CSUM_Tx,UDP4CSUM_Rx,UDP4CSUM_Tx,TCP6CSUM_Rx>
	capabilities=0x7ff80<TCP6CSUM_Tx,UDP6CSUM_Rx,UDP6CSUM_Tx,TSO6>
	enabled=0xf80<TSO4,IP4CSUM_Rx,IP4CSUM_Tx,TCP4CSUM_Rx,TCP4CSUM_Tx>
	vlan: 2 parent: wm1
	address: 00:e0:67:10:75:91
	status: active
	inet6 fe80::2e0:67ff:fe10:7591%vlan2/64 flags 0 scopeid 0x9
	inet6 2001:470:89fa:1::1/64 flags 0
	inet 192.168.100.1/24 broadcast 192.168.100.255 flags 0

But "ifconfig vlan2 -tso4" will make the problem go away.

Packets seem to be stuck somewhere in the offload path, because if I
interrupt the sender while things are "stuck", the receiver never gets FIN,
until I "ifconfig vlan2 -tso4" on the sender at which point the receiver
immediately gets a FIN.

Here's some dmesg output:

NetBSD 10.0_BETA (GENERIC) #0: Mon Oct  2 17:39:38 UTC 2023
        mkrepro%mkrepro.NetBSD.org@localhost:/usr/src/sys/arch/amd64/compile/GENERIC
total memory = 3987 MB
avail memory = 3830 MB
timecounter: Timecounters tick every 10.000 msec
Kernelized RAIDframe activated
i915drmkms* disabled
timecounter: Timecounter "i8254" frequency 1193182 Hz quality 100
mainbus0 (root)
[...]
wm1 at pci2 dev 0 function 0, 64-bit DMA: Intel i82583V (rev. 0x00)
wm1: interrupting at msi3 vec 0
wm1: PCI-Express bus
wm1: 64 words (8 address bits) SPI EEPROM, version 1.10.0
wm1: ASPM L0s and L1 are disabled to workaround the errata.
wm1: Ethernet address 00:e0:67:10:75:91
wm1: 0x2a4040<SPI,PCIE,ASF_FIRM,AMT,WOL>
makphy1 at wm1 phy 1: Marvell 88E1149 Gigabit PHY, rev. 1
makphy1: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT,
1000baseT-FDX, auto