kern/56857: ixg(4) doesn't work in legacy interrupt mode

[bouyer%antioche.eu.org@localhost]

>Number:         56857
>Category:       kern
>Synopsis:       ixg(4) doesn't work in legacy interrupt mode
>Confidential:   no
>Severity:       serious
>Priority:       high
>Responsible:    kern-bug-people
>State:          open
>Class:          sw-bug
>Submitter-Id:   net
>Arrival-Date:   Thu May 26 11:35:01 +0000 2022
>Originator:     Manuel Bouyer
>Release:        NetBSD 9.99.97
>Organization:
>Environment:
System: NetBSD bolero.soc.lip6.fr 9.99.97 NetBSD 9.99.97 (GENERIC) #4: Thu May 26 13:16:41 CEST 2022  bouyer@bip:/dsk/l1/misc/bouyer/tmp/amd64/obj/dsk/l1/misc/bouyer/HEAD/4commit/src/sys/arch/amd64/compile/GENERIC amd64
Architecture: x86_64
Machine: amd64
>Description:
	Forcing a system to use legacy interrupts with this patch:
Index: pci_machdep.c
===================================================================
RCS file: /cvsroot/src/sys/arch/x86/pci/pci_machdep.c,v
retrieving revision 1.91
diff -u -p -u -r1.91 pci_machdep.c
--- pci_machdep.c	24 May 2022 14:00:23 -0000	1.91
+++ pci_machdep.c	26 May 2022 11:18:59 -0000
@@ -565,6 +565,9 @@ pci_attach_hook(device_t parent, device_
 		}
 	}
 
+	pba->pba_flags &= ~PCI_FLAGS_MSIX_OKAY;
+	pba->pba_flags &= ~PCI_FLAGS_MSI_OKAY;
+
 #endif /* __HAVE_PCI_MSI_MSIX */
 }

	cause igx(4) device to fail to receive packet (or to receive them
	with very high latency). Other devices, including nvme(4) and wm(4)
	works fine in legacy mode. This is a show stopper for Xen (see PR
	kern/55667).
	The device works fine with MSI-X. I coulnd't test with MSI as
	disabling only MSI-X cause the kernel to panic (probably unrelated to
	igx(4)), but experiments with Xen suggest that it fails too.
	Here is how the device shows up (boot -vx):
[   1.1197710] ppb3 at pci4 dev 2 function 0: Intel product 2032 (rev. 0x07)
[   1.1197710] ppb3: PCI Express capability version 2 <Root Port of PCI-E Root Complex> x4 @ 8.0GT/s                                                          
[   1.1197710] pci5 at ppb3 bus 24
[   1.1197710] acpi0: MCFG: 024:00:0: Ok (cfg[0x100]=0x14020001 extconf=Y)
[   1.1197710] acpi0: MCFG: 024:00:1: Ok (cfg[0x100]=0x14020001 extconf=Y)
[   1.1197710] acpi0: MCFG: bus 24: valid devices
[   1.1197710] acpi0: MCFG: 024:00:0
[   1.1197710] acpi0: MCFG: 024:00:1
[   1.1197710] acpi0: acpimcfg_map_bus done
[   1.1197710] pci5: i/o space, memory space enabled
[   1.1197710] ixg0 at pci5 dev 0 function 0: Intel(R) PRO/10GbE PCI-Express Network Driver, Version - 4.0.1-k                                                
[   1.1197710] ixg0: clearing prefetchable bit
[   1.1197710] ixg0: device X550
[   1.1197710] ixg0: NVM Image Version 2.00, PHY FW Revision 2.0b ID 0x9, NVM Map version 2.52, OEM NVM Image version 0.06, ETrackID 80000d62                 
[   1.1197710] ixg0: PBA number K15087-004
[   1.1197710] ixg0: failed to allocate MSI-X interrupt
[   1.1197710] allocated pic ioapic2 type level pin 2 level 6 to cpu0 slot 4 idt entry 98                                                                    
[   1.1197710] ixg0: interrupting at ioapic2 pin 2
[   1.1197710] ixg0: Ethernet address 78:ac:44:86:b0:6c
[   1.1197710] ixg0: PHY: OUI 0x00aa00 model 0x0022, rev. 0
[   1.1197710] ixg0: PCI Express Bus: Speed 8.0GT/s Width x4
[   1.1197710] ixg0: feature cap 0x97a0<TEMP_SENSOR,LEGACY_TX,FDIR,MSI,MSIX,LEGACY_IRQ,RECOVERY_MODE>                                                         
[   1.1197710] ixg0: feature ena 0x9020<TEMP_SENSOR,LEGACY_IRQ,RECOVERY_MODE>

After a multiuser boot (including dhcp address request):
bolero# vmstat -i
interrupt              total rate
TLB shootdown            554    3
cpu0 timer             16580   97
ioapic0 pin 3            308    1
ioapic0 pin 16           314    1
ioapic2 pin 2            190    1
ixg0 Link event            2    0
ixg0 q0 IRQs on queue      1    0
ioapic4 pin 2           3854   22
Total                  21803  128

tcpdump shows that packets are sent.
 
>How-To-Repeat:
	disable MSI and MSI-X with the above patch, try to use the ixg device.
>Fix: