Re: port-i386/53894: "ehci_sync_hc: timed out" with ACPI/SMP enabled on eBox 3352DX3-AP

[Andrius V <vezhlys%gmail.com@localhost>]

The following reply was made to PR port-i386/53894; it has been noted by GNATS.

From: Andrius V <vezhlys%gmail.com@localhost>
To: gnats-bugs%netbsd.org@localhost
Cc: 
Subject: Re: port-i386/53894: "ehci_sync_hc: timed out" with ACPI/SMP enabled
 on eBox 3352DX3-AP
Date: Tue, 5 Mar 2024 09:57:31 +0200

 On Wed, Mar 16, 2022 at 10:25=E2=80=AFAM Andrius V <vezhlys%gmail.com@localhost> wrot=
 e:
 >
 > Hi,
 >
 > I did some investigation on this issue in recent days and here are my fin=
 dings:
 > * I can confirm, that it is interrupts issue, more specifically level
 > type PCI interrupts are not working (thus, USB, network, SCI, probably
 > audio too (untested)) interrupts doesn't work (checked by intrctl and
 > vmstat).
 > * The interrupt establishing flow seems OK though, did some debugging,
 > which shows that the flow doesn't seem to be different from similar
 > system which has no issues and visually everything is "established".
 > * By advice I checked MADT overrides: system reports two overrides,
 > and indeed, it seems that IRQ 0 NetBSD do not override. Unfortunately,
 > I have no idea, where to do that in NetBSD code (FreeBSD seems to go
 > through all MADT table and do overrides, in NetBSD case only SCI
 > interrupt individually comes to override code). Would that fix the
 > issue, honestly, I am not certain, I did try silly hacks, like not
 > masking interrupts by default, which added "i82489_ipi_init: ESR
 > 00000040" to dmesg, but none helped to make PCI interrupts work.
 > acpi0: [MADT] INTR OVERRIDE  : BUS 0, IRQ 0, GSI 2, FLAGS 0x00
 > acpi0: [MADT] INTR OVERRIDE  : BUS 0, IRQ 9, GSI 9, FLAGS 0x0D
 > * SoC does not support MSI.
 > * current kernel doesn't report "ehci_sync_hc: timed out" anymore
 > compared to my original dmesg report, just silently boots without any
 > uhidev devices attached, likely related to recent USB changes.
 > * FreeBSD 12.0 (the last release which boots out of the box on the
 > system), and recent Linux kernels has no problems with these
 > interrupts. OpenBSD have likely the same issue as NetBSD though, since
 > USB and network also doesn't work.
 > * Booting without ACPI/SMP enabled, makes interrupts work (it doesn't
 > go through apic in this case of course).
 >
 > Unfortunately, I am out of ideas for now on next steps, but will keep
 > investigating once I have some, any help would be welcome. Thank you.
 >
 
 Hi,
 
 I had few additional attempts to investigate the issue, unfortunately
 without any results which would lead to understanding the issue. I
 would be willing to give up my eBox 3360 with RS-485 serial ports if
 anybody would be interested to work on this.
 
 From the latest attempts I was still focussing on interrupt
 initialization, which increasingly feels like a wrong place to look
 at. Likely level type interrupts themselves should be investigated,
 but I don't know how to do that effectively.
 
 I did debug the flow from lapic.c, ioapic.c, pci_machdep.c, intrc.c
 and recently mpacpi.c, compared to my VIA system which is somewhat
 similar. Compared to FreeBSD code, which has differences, but I can't
 pinpoint anything of importance, extINT being the most visible
 difference, but it doesn't seem to affect level type interrupts in any
 way (14 now can boot as well). In general, from side it looks like all
 those attempts show "flawless" initialization, without errors or going
 to erroneous path. System boots without issues as well, however the
 end result is that USB, Ethernet doesn't work since no level type
 interrupts are acknowledged from them (or at least it looks like so).
 ISA interrupts though work, thus serial can be used and system itself
 seems to be stable otherwise.
 
 System BIOS also has "RDC engineer mode" by clicking F4 while in BIOS.
 It provides few additional options like disabling APIC bus, which
 allows to boot system with ACPI enabled and interrupts working,
 however they are wired through PIC IRQs and second core needs to be
 disabled to boot.
 
 In recent  attempts I enabled MPDEBUG which gives more log messages
 about configuring the interrupts, also shows that mpacpi takes care of
 both MATD overrides, all logs are expected and matches working
 systems:
 ....
 mpacpi: found root PCI bus 0
 mpacpi: found subordinate bus 1
 mpacpi: found subordinate bus 2
 mpacpi: 3 PCI busses
 mpacpi: ISA interrupt override 0 -> 2 (0/0)
 mpacpi: ISA interrupt override 9 -> 9 (1/3)
 mpacpi: configuring PCI bus 0 int routing
 pciroute: dev 13 INTA on globint 22
 pciroute: done adding entry
 pciroute: dev 1 INTA on globint 16
 pciroute: done adding entry
 pciroute: dev 1 INTB on globint 17
 pciroute: done adding entry
 pciroute: dev 1 INTC on globint 18
 pciroute: done adding entry
 pciroute: dev 1 INTD on globint 19
 pciroute: done adding entry
 pciroute: dev 2 INTA on globint 17
 pciroute: done adding entry
 pciroute: dev 2 INTB on globint 18
 pciroute: done adding entry
 pciroute: dev 2 INTC on globint 19
 pciroute: done adding entry
 pciroute: dev 2 INTD on globint 16
 pciroute: done adding entry
 pciroute: dev 3 INTA on globint 18
 pciroute: done adding entry
 pciroute: dev 3 INTB on globint 19
 pciroute: done adding entry
 pciroute: dev 3 INTC on globint 16
 pciroute: done adding entry
 pciroute: dev 3 INTD on globint 17
 pciroute: done adding entry
 pciroute: dev 8 INTA on globint 19
 pciroute: done adding entry
 pciroute: dev 10 INTA on globint 18
 pciroute: done adding entry
 pciroute: dev 10 INTB on globint 21
 pciroute: done adding entry
 pciroute: dev 12 INTA on globint 23
 pciroute: done adding entry
 pciroute: dev 14 INTA on globint 18
 pciroute: done adding entry
 pciroute: dev 15 INTA on globint 21
 pciroute: done adding entry
 pciroute: done
 mpacpi: configuring PCI bus 1 int routing
 pciroute: dev 0 INTA on globint 16
 pciroute: done adding entry
 pciroute: dev 0 INTB on globint 17
 pciroute: done adding entry
 pciroute: dev 0 INTC on globint 18
 pciroute: done adding entry
 pciroute: dev 0 INTD on globint 19
 pciroute: done adding entry
 pciroute: done
 mpacpi: configuring PCI bus 2 int routing
 pciroute: dev 0 INTA on globint 17
 pciroute: done adding entry
 pciroute: dev 0 INTB on globint 18
 pciroute: done adding entry
 pciroute: dev 0 INTC on globint 19
 pciroute: done adding entry
 pciroute: dev 0 INTD on globint 16
 pciroute: done adding entry
 pciroute: done
 ioapic0: pin 2 attached to isa0 irq 0 (type 0<type=3D0> flags 0<pol=3D0,tri=
 g=3D0>)
 ioapic0: pin 1 attached to isa0 irq 1 (type 0<type=3D0> flags 0<pol=3D0,tri=
 g=3D0>)
 ioapic0: pin 3 attached to isa0 irq 3 (type 0<type=3D0> flags 0<pol=3D0,tri=
 g=3D0>)
 ioapic0: pin 4 attached to isa0 irq 4 (type 0<type=3D0> flags 0<pol=3D0,tri=
 g=3D0>)
 ioapic0: pin 5 attached to isa0 irq 5 (type 0<type=3D0> flags 0<pol=3D0,tri=
 g=3D0>)
 ioapic0: pin 6 attached to isa0 irq 6 (type 0<type=3D0> flags 0<pol=3D0,tri=
 g=3D0>)
 ioapic0: pin 7 attached to isa0 irq 7 (type 0<type=3D0> flags 0<pol=3D0,tri=
 g=3D0>)
 ioapic0: pin 8 attached to isa0 irq 8 (type 0<type=3D0> flags 0<pol=3D0,tri=
 g=3D0>)
 ioapic0: pin 9 attached to isa0 irq 9 (type 0<type=3D0> flags
 0xd<pol=3D0x1=3DAct Hi,trig=3D0x3=3DLevel>)
 ioapic0: pin 10 attached to isa0 irq 10 (type 0<type=3D0> flags 0<pol=3D0,t=
 rig=3D0>)
 ioapic0: pin 11 attached to isa0 irq 11 (type 0<type=3D0> flags 0<pol=3D0,t=
 rig=3D0>)
 ioapic0: pin 12 attached to isa0 irq 12 (type 0<type=3D0> flags 0<pol=3D0,t=
 rig=3D0>)
 ioapic0: pin 13 attached to isa0 irq 13 (type 0<type=3D0> flags 0<pol=3D0,t=
 rig=3D0>)
 ioapic0: pin 14 attached to isa0 irq 14 (type 0<type=3D0> flags 0<pol=3D0,t=
 rig=3D0>)
 ioapic0: pin 15 attached to isa0 irq 15 (type 0<type=3D0> flags 0<pol=3D0,t=
 rig=3D0>)
 ioapic0: pin 22 attached to pci0 device 13 INT_A (type 0<type=3D0> flags
 0xf<pol=3D0x3=3DAct Lo,trig=3D0x3=3DLevel>)
 ioapic0: pin 16 attached to pci0 device 1 INT_A (type 0<type=3D0> flags
 0xf<pol=3D0x3=3DAct Lo,trig=3D0x3=3DLevel>)
 ioapic0: pin 17 attached to pci0 device 1 INT_B (type 0<type=3D0> flags
 0xf<pol=3D0x3=3DAct Lo,trig=3D0x3=3DLevel>)
 ioapic0: pin 18 attached to pci0 device 1 INT_C (type 0<type=3D0> flags
 0xf<pol=3D0x3=3DAct Lo,trig=3D0x3=3DLevel>)
 ioapic0: pin 19 attached to pci0 device 1 INT_D (type 0<type=3D0> flags
 0xf<pol=3D0x3=3DAct Lo,trig=3D0x3=3DLevel>)
 ioapic0: pin 17 attached to pci0 device 2 INT_A (type 0<type=3D0> flags
 0xf<pol=3D0x3=3DAct Lo,trig=3D0x3=3DLevel>)
 ioapic0: pin 18 attached to pci0 device 2 INT_B (type 0<type=3D0> flags
 0xf<pol=3D0x3=3DAct Lo,trig=3D0x3=3DLevel>)
 ioapic0: pin 19 attached to pci0 device 2 INT_C (type 0<type=3D0> flags
 0xf<pol=3D0x3=3DAct Lo,trig=3D0x3=3DLevel>)
 ioapic0: pin 16 attached to pci0 device 2 INT_D (type 0<type=3D0> flags
 0xf<pol=3D0x3=3DAct Lo,trig=3D0x3=3DLevel>)
 ioapic0: pin 18 attached to pci0 device 3 INT_A (type 0<type=3D0> flags
 0xf<pol=3D0x3=3DAct Lo,trig=3D0x3=3DLevel>)
 ioapic0: pin 19 attached to pci0 device 3 INT_B (type 0<type=3D0> flags
 0xf<pol=3D0x3=3DAct Lo,trig=3D0x3=3DLevel>)
 ioapic0: pin 16 attached to pci0 device 3 INT_C (type 0<type=3D0> flags
 0xf<pol=3D0x3=3DAct Lo,trig=3D0x3=3DLevel>)
 ioapic0: pin 17 attached to pci0 device 3 INT_D (type 0<type=3D0> flags
 0xf<pol=3D0x3=3DAct Lo,trig=3D0x3=3DLevel>)
 ioapic0: pin 19 attached to pci0 device 8 INT_A (type 0<type=3D0> flags
 0xf<pol=3D0x3=3DAct Lo,trig=3D0x3=3DLevel>)
 ioapic0: pin 18 attached to pci0 device 10 INT_A (type 0<type=3D0> flags
 0xf<pol=3D0x3=3DAct Lo,trig=3D0x3=3DLevel>)
 ioapic0: pin 21 attached to pci0 device 10 INT_B (type 0<type=3D0> flags
 0xf<pol=3D0x3=3DAct Lo,trig=3D0x3=3DLevel>)
 ioapic0: pin 23 attached to pci0 device 12 INT_A (type 0<type=3D0> flags
 0xf<pol=3D0x3=3DAct Lo,trig=3D0x3=3DLevel>)
 ioapic0: pin 18 attached to pci0 device 14 INT_A (type 0<type=3D0> flags
 0xf<pol=3D0x3=3DAct Lo,trig=3D0x3=3DLevel>)
 ioapic0: pin 21 attached to pci0 device 15 INT_A (type 0<type=3D0> flags
 0xf<pol=3D0x3=3DAct Lo,trig=3D0x3=3DLevel>)
 ioapic0: pin 16 attached to pci1 device 0 INT_A (type 0<type=3D0> flags
 0xf<pol=3D0x3=3DAct Lo,trig=3D0x3=3DLevel>)
 ioapic0: pin 17 attached to pci1 device 0 INT_B (type 0<type=3D0> flags
 0xf<pol=3D0x3=3DAct Lo,trig=3D0x3=3DLevel>)
 ioapic0: pin 18 attached to pci1 device 0 INT_C (type 0<type=3D0> flags
 0xf<pol=3D0x3=3DAct Lo,trig=3D0x3=3DLevel>)
 ioapic0: pin 19 attached to pci1 device 0 INT_D (type 0<type=3D0> flags
 0xf<pol=3D0x3=3DAct Lo,trig=3D0x3=3DLevel>)
 ioapic0: pin 17 attached to pci2 device 0 INT_A (type 0<type=3D0> flags
 0xf<pol=3D0x3=3DAct Lo,trig=3D0x3=3DLevel>)
 ioapic0: pin 18 attached to pci2 device 0 INT_B (type 0<type=3D0> flags
 0xf<pol=3D0x3=3DAct Lo,trig=3D0x3=3DLevel>)
 ioapic0: pin 19 attached to pci2 device 0 INT_C (type 0<type=3D0> flags
 0xf<pol=3D0x3=3DAct Lo,trig=3D0x3=3DLevel>)
 ioapic0: pin 16 attached to pci2 device 0 INT_D (type 0<type=3D0> flags
 0xf<pol=3D0x3=3DAct Lo,trig=3D0x3=3DLevel>)
 ...
 
 Regards,
 Andrius V