kern/55974: Broadcom NetXtreme II BCM5708 stopped working

[hannken%eis.cs.tu-bs.de@localhost]

>Number:         55974
>Category:       kern
>Synopsis:       Broadcom NetXtreme II BCM5708 stopped working
>Confidential:   no
>Severity:       critical
>Priority:       high
>Responsible:    kern-bug-people
>State:          open
>Class:          sw-bug
>Submitter-Id:   net
>Arrival-Date:   Thu Feb 04 10:15:00 +0000 2021
>Originator:     Juergen Hannken-Illjes
>Release:        NetBSD 9.99.77
>Organization:

>Environment:
System: NetBSD poweredge.isf.cs.tu-bs.de 9.99.77 NetBSD 9.99.77 (generic-dlog.amd64) #5: Wed Feb  3 16:21:58 MET 2021  hannken%builder.isf.cs.tu-bs.de@localhost:/work/build/obj/obj.amd64/sys/arch/amd64/compile/generic-dlog.amd64 amd64
Architecture: x86_64
Machine: amd64
>Description:
Since MSI/MSI-X got enabled for if_bnx with this commit:

Module Name:	src
Committed By:	jdolecek
Date:		Sun Jul 12 19:05:32 UTC 2020

Modified Files:
	src/sys/dev/pci: if_bnx.c if_bnxvar.h

Log Message:
	enable MSI/MSI-X if supported by adapter

	tested MSI-X with Broadcom NetXtreme II BCM5709 1000Base-T

the Dell PowerEdge 2950 with two Broadcom NetXtreme II BCM5708 1000Base-T
seems to loose tx interrupts and its watchdog fires.

[  68.1828359] bnx0: Watchdog timeout -- resetting!
[  88.6042909] bnx0: Watchdog timeout -- resetting!
[ 119.0265230] bnx0: Watchdog timeout -- resetting!
[ 145.4484562] bnx0: Watchdog timeout -- resetting!

Dmesg before is:

[     1.017306] pci4 at ppb3 bus 7
[     1.017306] pci4: i/o space, memory space enabled, rd/line, wr/inv ok
[     1.017306] bnx0 at pci4 dev 0 function 0: Broadcom NetXtreme II BCM5708 1000Base-T
[     1.017306] bnx0: Ethernet address 00:24:e8:67:4b:db
[     1.017306] bnx0: ASIC BCM5708 B2 (0x57081020)
[     1.017306] bnx0: PCI-X 64bit 133MHz
[     1.017306] bnx0: B/C (4.6.0); Bufs (RX:2;TX:2); Flags (MFW); MFW (ipms 1.6.0)
[     1.017306] bnx0: Coal (RX:6,6,18,18; TX:20,20,80,80)
[     1.017306] brgphy0 at bnx0 phy 1: BCM5708C 1000BASE-T media interface, rev. 6
[     1.017306] brgphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-FDX, auto
[     1.017306] bnx0: interrupting at ioapic0 pin 16

while dmesg after is:

[     1.017262] pci4 at ppb3 bus 7
[     1.017262] pci4: i/o space, memory space enabled, rd/line, wr/inv ok
[     1.017262] bnx0 at pci4 dev 0 function 0: Broadcom NetXtreme II BCM5708 1000Base-T
[     1.017262] bnx0: Ethernet address 00:24:e8:67:4b:db
[     1.017262] bnx0: ASIC BCM5708 B2 (0x57081020)
[     1.017262] bnx0: PCI-X 64bit 133MHz
[     1.017262] bnx0: B/C (4.6.0); Bufs (RX:2;TX:2); Flags (MFW); MFW (ipms 1.6.0)
[     1.017262] bnx0: Coal (RX:6,6,18,18; TX:20,20,80,80)
[     1.017262] brgphy0 at bnx0 phy 1: BCM5708C 1000BASE-T media interface, rev. 6
[     1.017262] brgphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-FDX, auto
[     1.017262] bnx0: interrupting at msi0 vec 0

Pcictl dump gives (in the MSI-X case):

PCI Message Signaled Interrupt
Message Control register: 0x0081
MSI Enabled: on
Multiple Message Capable: no (1 vector)
Multiple Message Enabled: off (1 vector)
64 Bit Address Capable: on
Per-Vector Masking Capable: off
Extended Message Data Capable: off
Extended Message Data Enable: off
Message Address (lower) register: 0xfee00000
Message Address (upper) register: 0x00000000
Message Data register: 0x0064
>How-To-Repeat:
Install a -current kernel and try to use the network.
>Fix:
unknown