usb xhci interrupt failure on TrueNAS Mini X+

["Darrin B. Jewell" <dbj%netbsd.org@localhost>]

I recently obtained a TrueNAS Mini X+ system from
  https://www.truenas.com/truenas-mini/
which comes installed with a FreeBSD based NAS configuration.
This includes a supermicro A2SDi-H-TF motherboard.

I attempted to boot NetBSD on it and discovered that
usb devices do not enumerate, giving this error for
each device:
   uhub1: device problem, disabling port 1
   uhub1: device problem, disabling port 3

Attaching release 9.1 INSTALL boot log below, but this
also occurs with -current recently compiled.

After investigation, I discovered that interrupts
were not getting delivered to the xhci driver.
I then reimplemented XHCI_DISABLE_MSI in sys/dev/pci/xhci_pci.c
and was able to get usb devices to work.

Anyone have ideas on why the default configuration fails?
I'm able to provide experimentation or additional debug logs. 

Darrin

[   1.0000000] Copyright (c) 1996, 1997, 1998, 1999, 2000, 2001, 2002, 2003, 2004, 2005,
[   1.0000000]     2006, 2007, 2008, 2009, 2010, 2011, 2012, 2013, 2014, 2015, 2016, 2017,
[   1.0000000]     2018, 2019, 2020 The NetBSD Foundation, Inc.  All rights reserved.
[   1.0000000] Copyright (c) 1982, 1986, 1989, 1991, 1993
[   1.0000000]     The Regents of the University of California.  All rights reserved.

[   1.0000000] NetBSD 9.1 (INSTALL) #0: Sun Oct 18 19:24:30 UTC 2020
[   1.0000000]        mkrepro%mkrepro.NetBSD.org@localhost:/usr/src/sys/arch/amd64/compile/INSTALL
[   1.0000000] total memory = 65497 MB
[   1.0000000] avail memory = 63571 MB
[   1.0000000] cpu_rng: RDSEED
[   1.0000000] running cgd selftest aes-xts-256 aes-xts-512 done
[   1.0000030] mainbus0 (root)
[   1.0000030] ACPI: RSDP 0x00000000000F05B0 000024 (v02 SUPERM)
[   1.0000030] ACPI: XSDT 0x000000007E1FD098 0000AC (v01 SUPERM SMCI--MB 01072009 AMI  00010013)
[   1.0000030] ACPI: FACP 0x000000007E201A20 000114 (v06 SUPERM SMCI--MB 01072009 AMI  00010013)
[   1.0000030] ACPI: DSDT 0x000000007E1FD1D8 004848 (v02 SUPERM SMCI--MB 01072009 INTL 20061109)
[   1.0000030] ACPI: FACS 0x000000007E268080 000040
[   1.0000030] ACPI: FPDT 0x000000007E201B38 000044 (v01 SUPERM SMCI--MB 01072009 AMI  00010013)
[   1.0000030] ACPI: FIDT 0x000000007E201B80 00009C (v01 SUPERM SMCI--MB 01072009 AMI  00010013)
[   1.0000030] ACPI: SPMI 0x000000007E201C20 000041 (v05 SUPERM SMCI--MB 00000000 AMI. 00000000)
[   1.0000030] ACPI: MCFG 0x000000007E201C68 00003C (v01 SUPERM SMCI--MB 01072009 MSFT 00000097)
[   1.0000030] ACPI: WDAT 0x000000007E201CA8 0001AC (v01 SUPERM SMCI--MB 01072009 MSFT 00010013)
[   1.0000030] ACPI: APIC 0x000000007E201E58 000098 (v04 INTEL  TIANO    00000001 MSFT 00000000)
[   1.0000030] ACPI: BDAT 0x000000007E201EF0 000030 (v01                 00000000      00000000)
[   1.0000030] ACPI: HPET 0x000000007E201F20 000038 (v01 SUPERM SMCI--MB 00000001 MSFT 01000013)
[   1.0000030] ACPI: UEFI 0x000000007E201F58 000042 (v01 SUPERM SMCI--MB 00000002      01000013)
[   1.0000030] ACPI: SSDT 0x000000007E201FA0 001901 (v02 PmRef  CpuPm    00003000 INTL 20061109)
[   1.0000030] ACPI: DMAR 0x000000007E2038A8 000070 (v01 INTEL  BDW      00000001 INTL 00000001)
[   1.0000030] ACPI: HEST 0x000000007E203918 0000A8 (v01 INTEL  VND      00000001 INTL 00000001)
[   1.0000030] ACPI: BERT 0x000000007E2039C0 000030 (v01 INTEL  VND      00000001 INTL 00000001)
[   1.0000030] ACPI: ERST 0x000000007E2039F0 000230 (v01 INTEL  VND      00000001 INTL 00000001)
[   1.0000030] ACPI: EINJ 0x000000007E203C20 000150 (v01 INTEL  VND      00000001 INTL 00000001)
[   1.0000030] ACPI: WSMT 0x000000007E203D70 000028 (v01 u?              01072009 AMI  00010013)
[   1.0000030] ACPI: 2 ACPI AML tables successfully acquired and loaded
[   1.0000030] ioapic0 at mainbus0 apid 2
[   1.0000030] cpu0 at mainbus0 apid 0
[   1.0000030] cpu0: Intel(R) Atom(TM) CPU C3758 @ 2.20GHz, id 0x506f1
[   1.0000030] cpu0: package 0, core 0, smt 0
[   1.0000030] cpu1 at mainbus0 apid 4
[   1.0000030] cpu1: Intel(R) Atom(TM) CPU C3758 @ 2.20GHz, id 0x506f1
[   1.0000030] cpu1: package 0, core 2, smt 0
[   1.0000030] cpu2 at mainbus0 apid 8
[   1.0000030] cpu2: Intel(R) Atom(TM) CPU C3758 @ 2.20GHz, id 0x506f1
[   1.0000030] cpu2: package 0, core 4, smt 0
[   1.0000030] cpu3 at mainbus0 apid 12
[   1.0000030] cpu3: Intel(R) Atom(TM) CPU C3758 @ 2.20GHz, id 0x506f1
[   1.0000030] cpu3: package 0, core 6, smt 0
[   1.0000030] cpu4 at mainbus0 apid 16
[   1.0000030] cpu4: Intel(R) Atom(TM) CPU C3758 @ 2.20GHz, id 0x506f1
[   1.0000030] cpu4: package 0, core 8, smt 0
[   1.0000030] cpu5 at mainbus0 apid 20
[   1.0000030] cpu5: Intel(R) Atom(TM) CPU C3758 @ 2.20GHz, id 0x506f1
[   1.0000030] cpu5: package 0, core 10, smt 0
[   1.0000030] cpu6 at mainbus0 apid 24
[   1.0000030] cpu6: Intel(R) Atom(TM) CPU C3758 @ 2.20GHz, id 0x506f1
[   1.0000030] cpu6: package 0, core 12, smt 0
[   1.0000030] cpu7 at mainbus0 apid 28
[   1.0000030] cpu7: Intel(R) Atom(TM) CPU C3758 @ 2.20GHz, id 0x506f1
[   1.0000030] cpu7: package 0, core 14, smt 0
[   1.0000030] acpi0 at mainbus0: Intel ACPICA 20190405
[   1.0000030] ACPI: Dynamic OEM Table Load:
[   1.0000030] ACPI: SSDT 0xFFFFF99418C704B0 00007A (v02 PmRef  Cpu0Cst  00003001 INTL 20061109)
[   1.0000030] ACPI: Dynamic OEM Table Load:
[   1.0000030] ACPI: SSDT 0xFFFFF99418D30008 0004B3 (v02 PmRef  Cpu0Ist  00003000 INTL 20061109)
[   1.0000030] ACPI: Dynamic OEM Table Load:
[   1.0000030] ACPI: SSDT 0xFFFFF99418D30808 00063B (v02 PmRef  ApIst    00003000 INTL 20061109)
[   1.0000030] ACPI: Dynamic OEM Table Load:
[   1.0000030] ACPI: SSDT 0xFFFFF9A313C5C388 000231 (v02 PmRef  ApCst    00003000 INTL 20061109)
[   1.0000030] acpi0: fixed power button present
[   1.0535204] hpet0 at acpi0: high precision event timer (mem 0xfed00000-0xfed00400)
[   1.0608532] APIC (PNP0003) at acpi0 not configured
[   1.0608532] PDRC (PNP0C02) at acpi0 not configured
[   1.0608532] MCH (PNP0C01) at acpi0 not configured
[   1.0608532] LDRC (PNP0C02) at acpi0 not configured
[   1.0608532] attimer1 at acpi0 (TIMR, PNP0100): io 0x40-0x43,0x50-0x53 irq 0
[   1.0608532] SIO1 (PNP0C02) at acpi0 not configured
[   1.0608532] UAR1 (PNP0501) at acpi0 not configured
[   1.0608532] UAR2 (PNP0501) at acpi0 not configured
[   1.0608532] ipmi_acpi0 at acpi0 (SPMI, IPI0001-0): io 0xca2-0xca3
[   1.0608532] ipmi0 at ipmi_acpi0
[   1.0608532] WERR (PNP0C33) at acpi0 not configured
[   1.0608532] ACPI: Enabled 1 GPEs in block 00 to 7F
[   1.0608532] pci0 at mainbus0 bus 0: configuration mode 1
[   1.0608532] pchb0 at pci0 dev 0 function 0: vendor 8086 product 1980 (rev. 0x11)
[   1.0608532] pchb1 at pci0 dev 4 function 0: vendor 8086 product 19a1 (rev. 0x11)
[   1.0608532] vendor 8086 product 19a2 (Root Complex Event Collector system, revision 0x11) at pci0 dev 5 function 0 not configured
[   1.0608532] ppb0 at pci0 dev 6 function 0: vendor 8086 product 19a3 (rev. 0x11)
[   1.0608532] ppb0: PCI Express capability version 2 <Root Port of PCI-E Root Complex> x1 @ 2.5GT/s
[   1.0608532] pci1 at ppb0 bus 1
[   1.0608532] vendor 8086 product 19e2 (Co-processor processor, revision 0x11) at pci1 dev 0 function 0 not configured
[   1.0608532] ppb1 at pci0 dev 9 function 0: vendor 8086 product 19a4 (rev. 0x11)
[   1.0608532] ppb1: PCI Express capability version 2 <Root Port of PCI-E Root Complex> x4 @ 8.0GT/s
[   1.0608532] pci2 at ppb1 bus 2
[   1.0608532] ppb2 at pci0 dev 16 function 0: vendor 8086 product 19aa (rev. 0x11)
[   1.0608532] ppb2: PCI Express capability version 2 <Root Port of PCI-E Root Complex> x2 @ 8.0GT/s
[   1.0608532] pci3 at ppb2 bus 3
[   1.0608532] ppb3 at pci0 dev 17 function 0: vendor 8086 product 19ab (rev. 0x11)
[   1.0608532] ppb3: PCI Express capability version 2 <Root Port of PCI-E Root Complex> x1 @ 8.0GT/s
[   1.0608532] ppb3: link is x1 @ 2.5GT/s
[   1.0608532] pci4 at ppb3 bus 4
[   1.0608532] ppb4 at pci4 dev 0 function 0: vendor 1a03 product 1150 (rev. 0x03)
[   1.0608532] ppb4: PCI Express capability version 1 <PCI-E to PCI/PCI-X Bridge>
[   1.0608532] pci5 at ppb4 bus 5
[   1.0608532] vga0 at pci5 dev 0 function 0: vendor 1a03 product 2000 (rev. 0x30)
[   1.0608532] wsdisplay0 at vga0 kbdmux 1
[   1.0608532] drm at vga0 not configured
[   1.0608532] ismt0 at pci0 dev 18 function 0: vendor 8086 product 19ac (rev. 0x11)
[   1.0608532] ismt0: interrupting at msi0 vec 0
[   1.0608532] ahcisata0 at pci0 dev 19 function 0: vendor 8086 product 19b2 (rev. 0x11)
[   1.0608532] ahcisata0: AHCI revision 1.31, 8 ports, 32 slots, CAP 0xc3349f47<EMS,PMD,SAM,ISS=0x3=Gen3,SCLO,SAL,SNCQ,S64A>
[   1.0608532] ahcisata0: interrupting at msix1 vec 0
[   1.0608532] atabus0 at ahcisata0 channel 0
[   1.0608532] ahcisata0: interrupting at msix1 vec 1
[   1.0608532] atabus1 at ahcisata0 channel 1
[   1.0608532] ahcisata0: interrupting at msix1 vec 2
[   1.0608532] atabus2 at ahcisata0 channel 2
[   1.0608532] ahcisata0: interrupting at msix1 vec 3
[   1.0608532] atabus3 at ahcisata0 channel 3
[   1.0608532] ahcisata0: interrupting at msix1 vec 4
[   1.0608532] atabus4 at ahcisata0 channel 4
[   1.0608532] ahcisata0: interrupting at msix1 vec 5
[   1.0608532] atabus5 at ahcisata0 channel 5
[   1.0608532] ahcisata0: interrupting at msix1 vec 6
[   1.0608532] atabus6 at ahcisata0 channel 6
[   1.0608532] ahcisata0: interrupting at msix1 vec 7
[   1.0608532] atabus7 at ahcisata0 channel 7
[   1.0608532] ahcisata1 at pci0 dev 20 function 0: vendor 8086 product 19c2 (rev. 0x11)
[   1.0608532] ahcisata1: AHCI revision 1.31, 4 ports, 32 slots, CAP 0xc3349f43<EMS,PMD,SAM,ISS=0x3=Gen3,SCLO,SAL,SNCQ,S64A>
[   1.0608532] ahcisata1: interrupting at msix2 vec 3
[   1.0608532] atabus8 at ahcisata1 channel 3
[   1.0608532] ahcisata1: interrupting at msix2 vec 4
[   1.0608532] atabus9 at ahcisata1 channel 4
[   1.0608532] ahcisata1: interrupting at msix2 vec 5
[   1.0608532] atabus10 at ahcisata1 channel 5
[   1.0608532] ahcisata1: interrupting at msix2 vec 6
[   1.0608532] atabus11 at ahcisata1 channel 6
[   1.0608532] xhci0 at pci0 dev 21 function 0: vendor 8086 product 19d0 (rev. 0x11)
[   1.0608532] xhci0: interrupting at msi3 vec 0
[   1.0608532] usb0 at xhci0: USB revision 3.0
[   1.0608532] usb1 at xhci0: USB revision 2.0
[   1.0608532] ppb5 at pci0 dev 22 function 0: vendor 8086 product 19d1 (rev. 0x11)
[   1.0608532] ppb5: PCI Express capability version 2 <Root Port of PCI-E Root Complex> x1 @ 2.5GT/s
[   1.0608532] pci6 at ppb5 bus 6
[   1.0608532] ixg0 at pci6 dev 0 function 0: Intel(R) PRO/10GbE PCI-Express Network Driver, Version - 4.0.1-k
[   1.0608532] ixg0: clearing prefetchable bit
[   1.0608532] ixg0: device X550EM A
[   1.0608532] ixg0: NVM Image Version 0.05 ID 0x8, NVM Map version 1.16, ETrackID 8000083e
[   1.0608532] ixg0: for TX/RX, interrupting at msix4 vec 0, bound queue 0 to cpu 0
[   1.0608532] ixg0: for TX/RX, interrupting at msix4 vec 1, bound queue 1 to cpu 1
[   1.0608532] ixg0: for TX/RX, interrupting at msix4 vec 2, bound queue 2 to cpu 2
[   1.0608532] ixg0: for TX/RX, interrupting at msix4 vec 3, bound queue 3 to cpu 3
[   1.0608532] ixg0: for TX/RX, interrupting at msix4 vec 4, bound queue 4 to cpu 4
[   1.0608532] ixg0: for TX/RX, interrupting at msix4 vec 5, bound queue 5 to cpu 5
[   1.0608532] ixg0: for TX/RX, interrupting at msix4 vec 6, bound queue 6 to cpu 6
[   1.0608532] ixg0: for TX/RX, interrupting at msix4 vec 7, bound queue 7 to cpu 7
[   1.0608532] ixg0: for link, interrupting at msix4 vec 8, affinity to cpu 0
[   1.0608532] ixg0: Using MSI-X interrupts with 9 vectors
[   1.0608532] ixg0: Ethernet address 3c:ec:ef:0c:fc:70
[   1.0608532] ixg0: PHY OUI 0x00aa00, model 0x0024, rev. 0
[   1.0608532] ixg1 at pci6 dev 0 function 1: Intel(R) PRO/10GbE PCI-Express Network Driver, Version - 4.0.1-k
[   1.0608532] ixg1: clearing prefetchable bit
[   1.0608532] ixg1: device X550EM A
[   1.0608532] ixg1: NVM Image Version 0.05 ID 0x8, NVM Map version 1.16, ETrackID 8000083e
[   1.0608532] ixg1: for TX/RX, interrupting at msix5 vec 0, bound queue 0 to cpu 0
[   1.0608532] ixg1: for TX/RX, interrupting at msix5 vec 1, bound queue 1 to cpu 1
[   1.0608532] ixg1: for TX/RX, interrupting at msix5 vec 2, bound queue 2 to cpu 2
[   1.0608532] ixg1: for TX/RX, interrupting at msix5 vec 3, bound queue 3 to cpu 3
[   1.0608532] ixg1: for TX/RX, interrupting at msix5 vec 4, bound queue 4 to cpu 4
[   1.0608532] ixg1: for TX/RX, interrupting at msix5 vec 5, bound queue 5 to cpu 5
[   1.0608532] ixg1: for TX/RX, interrupting at msix5 vec 6, bound queue 6 to cpu 6
[   1.0608532] ixg1: for TX/RX, interrupting at msix5 vec 7, bound queue 7 to cpu 7
[   1.0608532] ixg1: for link, interrupting at msix5 vec 8, affinity to cpu 0
[   1.0608532] ixg1: Using MSI-X interrupts with 9 vectors
[   1.0608532] ixg1: Ethernet address 3c:ec:ef:0c:fc:71
[   1.0608532] ixg1: PHY OUI 0x00aa00, model 0x0024, rev. 0
[   1.0608532] vendor 8086 product 19d3 (miscellaneous communications, revision 0x11) at pci0 dev 24 function 0 not configured
[   1.0608532] pcib0 at pci0 dev 31 function 0: vendor 8086 product 19dc (rev. 0x11)
[   1.0608532] vendor 8086 product 19de (miscellaneous memory, revision 0x11) at pci0 dev 31 function 2 not configured
[   1.0608532] ichsmb0 at pci0 dev 31 function 4: vendor 8086 product 19df (rev. 0x11)
[   1.0608532] ichsmb0: interrupting at ioapic0 pin 23
[   1.0608532] iic0 at ichsmb0: I2C bus
[   1.0608532] vendor 8086 product 19e0 (miscellaneous serial bus, revision 0x11) at pci0 dev 31 function 5 not configured
[   1.0608532] isa0 at pcib0
[   1.0608532] com0 at isa0 port 0x3f8-0x3ff irq 4: ns16550a, working fifo
[   1.0608532] com1 at isa0 port 0x2f8-0x2ff irq 3: ns16550a, working fifo
[   1.0608532] com1: console
[   1.0608532] pcppi0 at isa0 port 0x61
[   1.0608532] spkr0 at pcppi0: PC Speaker
[   1.0608532] wsbell at spkr0 not configured
[   1.0608532] midi0 at pcppi0: PC speaker
[   1.0608532] sysbeep0 at pcppi0
[   1.0608532] attimer1: attached to pcppi0
[   1.0608532] acpicpu0 at cpu0: ACPI CPU
[   1.0608532] coretemp0 at cpu0: thermal sensor, 1 C resolution
[   1.0608532] acpicpu1 at cpu1: ACPI CPU
[   1.0608532] coretemp1 at cpu1: thermal sensor, 1 C resolution
[   1.0608532] acpicpu2 at cpu2: ACPI CPU
[   1.0608532] coretemp2 at cpu2: thermal sensor, 1 C resolution
[   1.0608532] acpicpu3 at cpu3: ACPI CPU
[   1.0608532] coretemp3 at cpu3: thermal sensor, 1 C resolution
[   1.0608532] acpicpu4 at cpu4: ACPI CPU
[   1.0608532] coretemp4 at cpu4: thermal sensor, 1 C resolution
[   1.0608532] acpicpu5 at cpu5: ACPI CPU
[   1.0608532] coretemp5 at cpu5: thermal sensor, 1 C resolution
[   1.0608532] acpicpu6 at cpu6: ACPI CPU
[   1.0608532] coretemp6 at cpu6: thermal sensor, 1 C resolution
[   1.0608532] acpicpu7 at cpu7: ACPI CPU
[   1.0608532] coretemp7 at cpu7: thermal sensor, 1 C resolution
[   1.9108219] iic1 at ismt0: I2C bus
[   1.9108219] uhub0 at usb0: NetBSD (0000) xHCI root hub (0000), class 9/0, rev 3.00/1.00, addr 0
[   1.9267420] uhub1 at usb1: NetBSD (0000) xHCI root hub (0000), class 9/0, rev 2.00/1.00, addr 0
[   2.0108207] ahcisata0 port 0: device present, speed: 6.0Gb/s
[   2.0108207] ahcisata0 port 5: device present, speed: 6.0Gb/s
[   2.0220285] ahcisata1 port 6: device present, speed: 6.0Gb/s
[   3.6108205] wd0 at atabus0 drive 0
[   3.6208207] wd0: <ST8000NM0055-1RM112>
[   3.6208207] wd0: 7452 GB, 15504021 cyl, 16 head, 63 sec, 512 bytes/sect x 15628053168 sectors (0 bytes/physsect; first aligned sector: 8)
[   3.6608199] wd1 at atabus5 drive 0
[   3.6608199] wd1: <WDC WDS200T2B0A-00SM50>
[   3.6708202] wd1: 1863 GB, 3876021 cyl, 16 head, 63 sec, 512 bytes/sect x 3907029168 sectors
[   3.6808209] wd2 at atabus11 drive 0
[   3.6808209] wd2: <16GB SATA Flash Drive>
[   3.6908219] wd2: 15272 MB, 31029 cyl, 16 head, 63 sec, 512 bytes/sect x 31277232 sectors
[   3.7008214] dk0 at wd2: "3b189fba-5f04-11eb-bf74-3cecef0cfc70", 532480 blocks at 40, type: msdos
[   3.7108208] dk1 at wd2: "3b1edda2-5f04-11eb-bf74-3cecef0cfc70", 30736384 blocks at 532520, type: <unknown>
[   8.5308202] uhub1: device problem, disabling port 1
[  10.1708209] ipmi0: ID 32.1 IPMI 2.0 Available
[  14.4208209] uhub1: device problem, disabling port 3
[  14.4308214] WARNING: 2 errors while detecting hardware; check system log.
[  14.4408219] boot device: ixg0
[  14.4508212] root on md0a dumps on md0b
[  14.4508212] root file system type: ffs
[  14.4508212] kern.module.path=/stand/amd64/9.1/modules
Created tmpfs /dev (1753088 byte, 3392 inodes)
erase ^?, werase ^W, kill ^U, intr ^C