Re: wm workqueue and agr device

[Masanobu SAITOH <msaitoh%execsw.org@localhost>]

Hi.

On 2022/06/02 19:08, BERTRAND Joël wrote:
> 	Hello,
> 
> 	In a recent mail (if_wm.c workqueue), I have seen that it is possible
> to activate workqueue on all ethernet devices contained in my server:
> 
> [     1.002997] wm0 at pci2 dev 0 function 0: I350 Gigabit Network
> Connection (rev. 0x01)
> [     1.002997] wm0: for TX and RX interrupting at msix0 vec 0 affinity to 0
> [     1.002997] wm0: for TX and RX interrupting at msix0 vec 1 affinity to 1
> [     1.002997] wm0: for TX and RX interrupting at msix0 vec 2 affinity to 2
> [     1.002997] wm0: for TX and RX interrupting at msix0 vec 3 affinity to 3
> [     1.002997] wm0: for TX and RX interrupting at msix0 vec 4 affinity to 4
> [     1.002997] wm0: for TX and RX interrupting at msix0 vec 5 affinity to 5
> [     1.002997] wm0: for TX and RX interrupting at msix0 vec 6 affinity to 6
> [     1.002997] wm0: for TX and RX interrupting at msix0 vec 7 affinity to 7
> [     1.002997] wm0: for LINK interrupting at msix0 vec 8
> [     1.002997] wm0: PCI-Express bus
> [     1.002997] wm0: 8192 words (16 address bits) SPI EEPROM, version
> 1.63, Image Unique ID 00000ffd
> [     1.002997] wm0: Ethernet address b4:96:91:92:77:6e
> [     1.002997] wm0: Copper
> [     1.002997] wm0:
> 0x40674040<SPI,PCIE,NEWQUEUE,ASF_FIRM,ARC_SUBSYS,WOL,EEE,CRC_STRIP>
> [     1.002997] ihphy0 at wm0 phy 1: I350 10/100/1000 media interface,
> rev. 1
> [     1.002997] wm1 at pci2 dev 0 function 1: I350 Gigabit Network
> Connection (rev. 0x01)
> [     1.002997] wm1: for TX and RX interrupting at msix1 vec 0 affinity to 0
> [     1.002997] wm1: for TX and RX interrupting at msix1 vec 1 affinity to 1
> [     1.002997] wm1: for TX and RX interrupting at msix1 vec 2 affinity to 2
> [     1.002997] wm1: for TX and RX interrupting at msix1 vec 3 affinity to 3
> [     1.002997] wm1: for TX and RX interrupting at msix1 vec 4 affinity to 4
> [     1.002997] wm1: for TX and RX interrupting at msix1 vec 5 affinity to 5
> [     1.002997] wm1: for TX and RX interrupting at msix1 vec 6 affinity to 6
> [     1.002997] wm1: for TX and RX interrupting at msix1 vec 7 affinity to 7
> [     1.002997] wm1: for LINK interrupting at msix1 vec 8
> [     1.002997] wm1: PCI-Express bus
> [     1.002997] wm1: 8192 words (16 address bits) SPI EEPROM, version
> 1.63, Image Unique ID 00000ffd
> [     1.002997] wm1: Ethernet address b4:96:91:92:77:6f
> [     1.002997] wm1: Copper
> [     1.002997] wm1:
> 0x40474040<SPI,PCIE,NEWQUEUE,ASF_FIRM,ARC_SUBSYS,EEE,CRC_STRIP>
> [     1.002997] ihphy1 at wm1 phy 1: I350 10/100/1000 media interface,
> rev. 1
> [     1.002997] wm2 at pci0 dev 25 function 0: I218 V Ethernet
> Connection (rev. 0x00)
> [     1.002997] wm2: interrupting at msi4 vec 0
> [     1.002997] wm2: PCI-Express bus
> [     1.002997] wm2: 2048 words FLASH, version 0.1.4
> [     1.002997] wm2: Ethernet address 08:62:66:47:63:99
> [     1.002997] wm2: 0x6a4080<FLASH,PCIE,ASF_FIRM,AMT,WOL,EEE>
> [     1.002997] ihphy2 at wm2 phy 2: i217 10/100/1000 media interface,
> rev. 5
> [     1.002997] wm3 at pci6 dev 0 function 0: Intel i82574L (rev. 0x00)
> [     1.002997] wm3: for TX and RX interrupting at msix6 vec 0 affinity to 1
> [     1.002997] wm3: for TX and RX interrupting at msix6 vec 1 affinity to 2
> [     1.002997] wm3: for LINK interrupting at msix6 vec 2
> [     1.002997] wm3: PCI-Express bus
> [     1.002997] wm3: 2048 words FLASH, version 1.8.0, Image Unique ID
> 0000ffff
> [     1.002997] wm3: ASPM L0s and L1 are disabled to workaround the errata.
> [     1.002997] wm3: Ethernet address 68:05:ca:02:b2:59
> [     1.002997] wm3: 0x224080<FLASH,PCIE,ASF_FIRM,WOL>
> [     1.002997] makphy0 at wm3 phy 1: Marvell 88E1149 Gigabit PHY, rev. 1
> [     1.002997] wm4 at pci7 dev 0 function 0: Intel i82574L (rev. 0x00)
> [     1.002997] wm4: for TX and RX interrupting at msix7 vec 0 affinity to 1
> [     1.002997] wm4: for TX and RX interrupting at msix7 vec 1 affinity to 2
> [     1.002997] wm4: for LINK interrupting at msix7 vec 2
> [     1.002997] wm4: PCI-Express bus
> [     1.002997] wm4: 2048 words FLASH, version 1.8.0, Image Unique ID
> 0000ffff
> [     1.002997] wm4: ASPM L0s and L1 are disabled to workaround the errata.
> [     1.002997] wm4: Ethernet address 68:05:ca:44:cf:30
> [     1.002997] wm4: 0x224080<FLASH,PCIE,ASF_FIRM,WOL>
> [     1.002997] makphy1 at wm4 phy 1: Marvell 88E1149 Gigabit PHY, rev. 1
> 
> wm0 and wm1 are bridged (bridge0) and connected to two NAS.
> wm2 is connected to WAN
> wm3 and wm4 are aggregated (agr0) and connected to LAN.
> 
> 	I have tried to set in /etc/sysctl.conf:
> 
> hw.wm0.txrx_workqueue=1
> hw.wm1.txrx_workqueue=1
> hw.wm2.txrx_workqueue=1
> hw.wm3.txrx_workqueue=1
> hw.wm4.txrx_workqueue=1
> 
> and system quickly hangs (no panic, no internal debugger, no serial
> console available, only a hard lock). Before lock, agr0 was very very slow.

agr(4) uses softint. Scheduling priority of softint is higher
than workqueue. It might cause the problem.

> 	If I set:
> 
> hw.wm0.txrx_workqueue=1
> hw.wm1.txrx_workqueue=1
> hw.wm2.txrx_workqueue=1
> hw.wm3.txrx_workqueue=0
> hw.wm4.txrx_workqueue=0
> 
> system seems to run as expected. Thus I suppose that workqueue and agr
> don't run together.
> 
> 	My kernel is:
> NetBSD legendre.systella.fr 9.2_STABLE NetBSD 9.2_STABLE (CUSTOM) #11:
> Thu Mar 31 11:43:25 CEST 2022
> root%legendre.systella.fr@localhost:/usr/src/netbsd-9/obj/sys/arch/amd64/compile/CUSTOM
> amd64
> 
> CUSTOM as I have built ATQ support. I use NetBSD 9 (not 9.2) as rolling
> release.
> 
> 	Is it possible to use workqueue with agr ?

Perhaps it isn't possible. lagg(4) can use with workqueue'd Ethernet driver,
so -current or NetBSD 10 can be use with it.

 Regards.

> 	Best regards,
> 
> 	JB

-- 
-----------------------------------------------
                SAITOH Masanobu (msaitoh%execsw.org@localhost
                                 msaitoh%netbsd.org@localhost)