kern/58124: lagg(4) fails to work with lacp protocol on mcx(4)

[vezhlys%gmail.com@localhost]

>Number:         58124
>Category:       kern
>Synopsis:       lagg(4) fails to work with lacp protocol on mcx(4)
>Confidential:   no
>Severity:       serious
>Priority:       medium
>Responsible:    kern-bug-people
>State:          open
>Class:          sw-bug
>Submitter-Id:   net
>Arrival-Date:   Sun Apr 07 18:05:00 +0000 2024
>Originator:     Andrius V
>Release:        NetBSD 10
>Organization:
>Environment:
>Description:
Hi,

I am trying to configure lagg(4) interface using lacp (802.3ad) protocol between
my NAS server (amd64 machine) and Mikrotik CRS309-1G-8S+ router (RouterOS).
My network adapter is QNAP QXG-10G2SF-CX4 (Mellanox ConnectX-4 Lx mcx(4) SFP+ DAC).
Unfortunately I can't manage to make it work and communicate with my switch or 
receive IP address from my router's DHCP server.

Switch should be configured correctly, I used it for Linux setup for years, also
deprecated agr(4) interface "sometimes" works with certain manual actions applied.
agr(4) is unstable though and looses connection soon or after reboot requiring
manual interaction again, thus also unusable in general. It has "LACP Partner System ID"
in the interface, but it stays empty for NetBSD (should be mac address of nas server).

I am using dhcpcd DHCP client.

Alternatively I tried loadbalance protocol, it works without issues in comparison.

Besides these I have 2x igc(4) interfaces, but I am not using them for link aggregation.

ifconfig.lagg0
create
laggproto lacp laggport mcx0 laggport mcx1
up

/etc/rc.conf
...
net_interfaces="igc1 lagg0"
dhcpcd=YES
dhcpcd_flags="-qLM lagg0 igc1"
...

ifconfig lagg0
lagg0: flags=0x8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
	capabilities=0x3ff00<IP4CSUM_Rx,IP4CSUM_Tx,TCP4CSUM_Rx,TCP4CSUM_Tx>
	capabilities=0x3ff00<UDP4CSUM_Rx,UDP4CSUM_Tx,TCP6CSUM_Rx,TCP6CSUM_Tx>
	capabilities=0x3ff00<UDP6CSUM_Rx,UDP6CSUM_Tx>
	enabled=0x3ff00<IP4CSUM_Rx,IP4CSUM_Tx,TCP4CSUM_Rx,TCP4CSUM_Tx>
	enabled=0x3ff00<UDP4CSUM_Rx,UDP4CSUM_Tx,TCP6CSUM_Rx,TCP6CSUM_Tx>
	enabled=0x3ff00<UDP6CSUM_Rx,UDP6CSUM_Tx>
	ec_capabilities=0x7<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU>
	ec_enabled=0x2<VLAN_HWTAGGING>
	laggproto lacp
	laggport:
		mcx0 pri=32768 flags=0
		mcx1 pri=32768 flags=0
	address: xx:xx:xx:xx:xx:xx
	status: no carrier

mcx0: flags=0x8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
	capabilities=0x3ff00<IP4CSUM_Rx,IP4CSUM_Tx,TCP4CSUM_Rx,TCP4CSUM_Tx>
	capabilities=0x3ff00<UDP4CSUM_Rx,UDP4CSUM_Tx,TCP6CSUM_Rx,TCP6CSUM_Tx>
	capabilities=0x3ff00<UDP6CSUM_Rx,UDP6CSUM_Tx>
	enabled=0x3ff00<IP4CSUM_Rx,IP4CSUM_Tx,TCP4CSUM_Rx,TCP4CSUM_Tx>
	enabled=0x3ff00<UDP4CSUM_Rx,UDP4CSUM_Tx,TCP6CSUM_Rx,TCP6CSUM_Tx>
	enabled=0x3ff00<UDP6CSUM_Rx,UDP6CSUM_Tx>
	ec_capabilities=0x7<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU>
	ec_enabled=0x2<VLAN_HWTAGGING>
	address: xx:xx:xx:xx:xx:xx
	media: Ethernet autoselect (10GBASE-CR1)
	status: active

mcx1: flags=0x8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
	capabilities=0x3ff00<IP4CSUM_Rx,IP4CSUM_Tx,TCP4CSUM_Rx,TCP4CSUM_Tx>
	capabilities=0x3ff00<UDP4CSUM_Rx,UDP4CSUM_Tx,TCP6CSUM_Rx,TCP6CSUM_Tx>
	capabilities=0x3ff00<UDP6CSUM_Rx,UDP6CSUM_Tx>
	enabled=0x3ff00<IP4CSUM_Rx,IP4CSUM_Tx,TCP4CSUM_Rx,TCP4CSUM_Tx>
	enabled=0x3ff00<UDP4CSUM_Rx,UDP4CSUM_Tx,TCP6CSUM_Rx,TCP6CSUM_Tx>
	enabled=0x3ff00<UDP6CSUM_Rx,UDP6CSUM_Tx>
	ec_capabilities=0x7<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU>
	ec_enabled=0x2<VLAN_HWTAGGING>
	address: xx:xx:xx:xx:xx:xx
	media: Ethernet autoselect (10GBASE-CR1)
	status: active
	link xx:xx:xx:xx:xx:xx

dmesg for mcx:
mcx0 at pci9 dev 0 function 0: Mellanox Technologies ConnectX-4 Lx (rev. 0x00)
mcx0: FW 14.22.1002
mcx0: Ethernet address xx:xx:xx:xx:xx:xx
mcx1 at pci9 dev 0 function 1: Mellanox Technologies ConnectX-4 Lx (rev. 0x00)
mcx1: FW 14.22.1002
mcx1: Ethernet address xx:xx:xx:xx:xx:xx

>How-To-Repeat:
ifconfig lagg0 create
ifconfig lagg0 laggproto lacp laggport mcx0 laggport mcx1
ifconfig lagg0 up

Start dhcpcd service or run dhcpcd lagg0 and observe that no IP is assigned, "status: no carrier" is displayed.
>Fix: