Re: wm + vlan tso4 problem with 10_BETA

[Masanobu SAITOH <msaitoh%execsw.org@localhost>]

Hi.

On 2023/11/01 10:56, Masanobu SAITOH wrote:
> Hi, Thor.
> 
> I'm very sorry that I've not noticed this mail until now.
> 
> On 2023/10/15 3:09, Thor Lancelot Simon wrote:
>> Since upgrading my home gateway from an older version of NetBSD 9 to
>> NetBSD 10.0_BETA, I've been having some throughput issues - forwarding
>> performance seems to have dropped by between 30% and 50%, in a way
>> that is highly sensitive to client TCP stack behavior.  More perhaps on
>> that later.
>>
>> However this has resulted in my testing every link in my internal network,
>> to establish a baseline.  In the course of this I discovered that if TSO4
>> is enabled on a vlan interface which is on wm, transmitting bulk data over
>> TCP from the NetBSD gateway to a local client (on the same vlan) very
>> quickly stalls.
>>
>> Has anyone else seen anything like this recently?
> 
> No. I've never used TSO[46] on vlan interface...
> 
>> This configuration will stall almost immediately when sending to a local
>> peer with ttcp -s -t -b $((192 * 1024)) -l 65536:
>>
>> hotpoint# ifconfig vlan2
>> vlan2: flags=0x8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
>> 	capabilities=0x7ff80<TSO4,IP4CSUM_Rx,IP4CSUM_Tx,TCP4CSUM_Rx>
>> 	capabilities=0x7ff80<TCP4CSUM_Tx,UDP4CSUM_Rx,UDP4CSUM_Tx,TCP6CSUM_Rx>
>> 	capabilities=0x7ff80<TCP6CSUM_Tx,UDP6CSUM_Rx,UDP6CSUM_Tx,TSO6>
>> 	enabled=0xf80<TSO4,IP4CSUM_Rx,IP4CSUM_Tx,TCP4CSUM_Rx,TCP4CSUM_Tx>
>> 	vlan: 2 parent: wm1
>> 	address: 00:e0:67:10:75:91
>> 	status: active
>> 	inet6 fe80::2e0:67ff:fe10:7591%vlan2/64 flags 0 scopeid 0x9
>> 	inet6 2001:470:89fa:1::1/64 flags 0
>> 	inet 192.168.100.1/24 broadcast 192.168.100.255 flags 0
>>
>> But "ifconfig vlan2 -tso4" will make the problem go away.
>>
>> Packets seem to be stuck somewhere in the offload path, because if I
>> interrupt the sender while things are "stuck", the receiver never gets FIN,
>> until I "ifconfig vlan2 -tso4" on the sender at which point the receiver
>> immediately gets a FIN.
> 
> One of possibility is that the offset of the IP header is miscalculated...

It works for me...

zen2# ifconfig wm0
wm0: flags=0x8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
        capabilities=0x7ff80<TSO4,IP4CSUM_Rx,IP4CSUM_Tx,TCP4CSUM_Rx>
        capabilities=0x7ff80<TCP4CSUM_Tx,UDP4CSUM_Rx,UDP4CSUM_Tx,TCP6CSUM_Rx>
        capabilities=0x7ff80<TCP6CSUM_Tx,UDP6CSUM_Rx,UDP6CSUM_Tx,TSO6>
        enabled=0x7ff80<TSO4,IP4CSUM_Rx,IP4CSUM_Tx,TCP4CSUM_Rx>
        enabled=0x7ff80<TCP4CSUM_Tx,UDP4CSUM_Rx,UDP4CSUM_Tx,TCP6CSUM_Rx>
        enabled=0x7ff80<TCP6CSUM_Tx,UDP6CSUM_Rx,UDP6CSUM_Tx,TSO6>
        ec_capabilities=0x17<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,EEE>
        ec_enabled=0x3<VLAN_MTU,VLAN_HWTAGGING>
        address: 9c:5c:8e:bc:22:44
        media: Ethernet autoselect (1000baseT full-duplex,flowcontrol,rxpause,txpause)
        status: active
        inet6 fe80::9e5c:8eff:febc:2244%wm0/64 flags 0 scopeid 0x3
zen2# ifconfig vlan1
vlan1: flags=0x8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
        capabilities=0x7ff80<TSO4,IP4CSUM_Rx,IP4CSUM_Tx,TCP4CSUM_Rx>
        capabilities=0x7ff80<TCP4CSUM_Tx,UDP4CSUM_Rx,UDP4CSUM_Tx,TCP6CSUM_Rx>
        capabilities=0x7ff80<TCP6CSUM_Tx,UDP6CSUM_Rx,UDP6CSUM_Tx,TSO6>
        enabled=0x7ff80<TSO4,IP4CSUM_Rx,IP4CSUM_Tx,TCP4CSUM_Rx>
        enabled=0x7ff80<TCP4CSUM_Tx,UDP4CSUM_Rx,UDP4CSUM_Tx,TCP6CSUM_Rx>
        enabled=0x7ff80<TCP6CSUM_Tx,UDP6CSUM_Rx,UDP6CSUM_Tx,TSO6>
        vlan: 1 parent: wm0
        address: 9c:5c:8e:bc:22:44
        status: active
        inet6 fe80::9e5c:8eff:febc:2244%vlan1/64 flags 0 scopeid 0x6
        inet 10.0.0.1/24 broadcast 10.0.0.255 flags 0

IPv4, IPv6, iperf3 -w 8m, ttcp -b 196608 -l 65536, both direction works on
my -current machine. Not tested on netbsd-10 yet though...


> 
>> Here's some dmesg output:
>>
>> NetBSD 10.0_BETA (GENERIC) #0: Mon Oct  2 17:39:38 UTC 2023
>>         mkrepro%mkrepro.NetBSD.org@localhost:/usr/src/sys/arch/amd64/compile/GENERIC
>> total memory = 3987 MB
>> avail memory = 3830 MB
>> timecounter: Timecounters tick every 10.000 msec
>> Kernelized RAIDframe activated
>> i915drmkms* disabled
>> timecounter: Timecounter "i8254" frequency 1193182 Hz quality 100
>> mainbus0 (root)
>> [...]
>> wm1 at pci2 dev 0 function 0, 64-bit DMA: Intel i82583V (rev. 0x00)
>> wm1: interrupting at msi3 vec 0
>> wm1: PCI-Express bus
>> wm1: 64 words (8 address bits) SPI EEPROM, version 1.10.0
>> wm1: ASPM L0s and L1 are disabled to workaround the errata.
>> wm1: Ethernet address 00:e0:67:10:75:91
>> wm1: 0x2a4040<SPI,PCIE,ASF_FIRM,AMT,WOL>
>> makphy1 at wm1 phy 1: Marvell 88E1149 Gigabit PHY, rev. 1
>> makphy1: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT,
>> 1000baseT-FDX, auto
>>
> 

-- 
-----------------------------------------------
                SAITOH Masanobu (msaitoh%execsw.org@localhost
                                 msaitoh%netbsd.org@localhost)