Re: port-i386/53894: "ehci_sync_hc: timed out" with ACPI/SMP enabled on eBox 3352DX3-AP

["Andrius V via gnats" <gnats-admin%NetBSD.org@localhost>]

The following reply was made to PR port-i386/53894; it has been noted by GNATS.

From: Andrius V <vezhlys%gmail.com@localhost>
To: gnats-bugs%netbsd.org@localhost
Cc: 
Subject: Re: port-i386/53894: "ehci_sync_hc: timed out" with ACPI/SMP enabled
 on eBox 3352DX3-AP
Date: Fri, 27 Mar 2026 11:02:26 +0200

 On Wed, Mar 18, 2026 at 7:35=E2=80=AFPM Andrius V via gnats
 <gnats-admin%netbsd.org@localhost> wrote:
 >
 > The following reply was made to PR port-i386/53894; it has been noted by =
 GNATS.
 >
 > From: Andrius V <vezhlys%gmail.com@localhost>
 > To: gnats-bugs%netbsd.org@localhost
 > Cc:
 > Subject: Re: port-i386/53894: "ehci_sync_hc: timed out" with ACPI/SMP ena=
 bled
 >  on eBox 3352DX3-AP
 > Date: Wed, 18 Mar 2026 19:34:43 +0200
 >
 >  On Tue, Mar 5, 2024 at 9:57=3DE2=3D80=3DAFAM Andrius V <vezhlys%gmail.co@localhost=
 m> wrote:
 >  >
 >  > On Wed, Mar 16, 2022 at 10:25=3DE2=3D80=3DAFAM Andrius V <vezhlys@gmai=
 l.com> wr=3D
 >  ote:
 >  > >
 >  > > Hi,
 >  > >
 >  > > I did some investigation on this issue in recent days and here are m=
 y f=3D
 >  indings:
 >  > > * I can confirm, that it is interrupts issue, more specifically leve=
 l
 >  > > type PCI interrupts are not working (thus, USB, network, SCI, probab=
 ly
 >  > > audio too (untested)) interrupts doesn't work (checked by intrctl an=
 d
 >  > > vmstat).
 >  > > * The interrupt establishing flow seems OK though, did some debuggin=
 g,
 >  > > which shows that the flow doesn't seem to be different from similar
 >  > > system which has no issues and visually everything is "established".
 >  > > * By advice I checked MADT overrides: system reports two overrides,
 >  > > and indeed, it seems that IRQ 0 NetBSD do not override. Unfortunatel=
 y,
 >  > > I have no idea, where to do that in NetBSD code (FreeBSD seems to go
 >  > > through all MADT table and do overrides, in NetBSD case only SCI
 >  > > interrupt individually comes to override code). Would that fix the
 >  > > issue, honestly, I am not certain, I did try silly hacks, like not
 >  > > masking interrupts by default, which added "i82489_ipi_init: ESR
 >  > > 00000040" to dmesg, but none helped to make PCI interrupts work.
 >  > > acpi0: [MADT] INTR OVERRIDE  : BUS 0, IRQ 0, GSI 2, FLAGS 0x00
 >  > > acpi0: [MADT] INTR OVERRIDE  : BUS 0, IRQ 9, GSI 9, FLAGS 0x0D
 >  > > * SoC does not support MSI.
 >  > > * current kernel doesn't report "ehci_sync_hc: timed out" anymore
 >  > > compared to my original dmesg report, just silently boots without an=
 y
 >  > > uhidev devices attached, likely related to recent USB changes.
 >  > > * FreeBSD 12.0 (the last release which boots out of the box on the
 >  > > system), and recent Linux kernels has no problems with these
 >  > > interrupts. OpenBSD have likely the same issue as NetBSD though, sin=
 ce
 >  > > USB and network also doesn't work.
 >  > > * Booting without ACPI/SMP enabled, makes interrupts work (it doesn'=
 t
 >  > > go through apic in this case of course).
 >  > >
 >  > > Unfortunately, I am out of ideas for now on next steps, but will kee=
 p
 >  > > investigating once I have some, any help would be welcome. Thank you=
 .
 >  > >
 >  >
 >  > Hi,
 >  >
 >  > I had few additional attempts to investigate the issue, unfortunately
 >  > without any results which would lead to understanding the issue. I
 >  > would be willing to give up my eBox 3360 with RS-485 serial ports if
 >  > anybody would be interested to work on this.
 >  >
 >  > From the latest attempts I was still focussing on interrupt
 >  > initialization, which increasingly feels like a wrong place to look
 >  > at. Likely level type interrupts themselves should be investigated,
 >  > but I don't know how to do that effectively.
 >  >
 >  > I did debug the flow from lapic.c, ioapic.c, pci_machdep.c, intrc.c
 >  > and recently mpacpi.c, compared to my VIA system which is somewhat
 >  > similar. Compared to FreeBSD code, which has differences, but I can't
 >  > pinpoint anything of importance, extINT being the most visible
 >  > difference, but it doesn't seem to affect level type interrupts in any
 >  > way (14 now can boot as well). In general, from side it looks like all
 >  > those attempts show "flawless" initialization, without errors or going
 >  > to erroneous path. System boots without issues as well, however the
 >  > end result is that USB, Ethernet doesn't work since no level type
 >  > interrupts are acknowledged from them (or at least it looks like so).
 >  > ISA interrupts though work, thus serial can be used and system itself
 >  > seems to be stable otherwise.
 >  >
 >  > System BIOS also has "RDC engineer mode" by clicking F4 while in BIOS.
 >  > It provides few additional options like disabling APIC bus, which
 >  > allows to boot system with ACPI enabled and interrupts working,
 >  > however they are wired through PIC IRQs and second core needs to be
 >  > disabled to boot.
 >  >
 >  > In recent  attempts I enabled MPDEBUG which gives more log messages
 >  > about configuring the interrupts, also shows that mpacpi takes care of
 >  > both MATD overrides, all logs are expected and matches working
 >  > systems:
 >  > ....
 >  > mpacpi: found root PCI bus 0
 >  > mpacpi: found subordinate bus 1
 >  > mpacpi: found subordinate bus 2
 >  > mpacpi: 3 PCI busses
 >  > mpacpi: ISA interrupt override 0 -> 2 (0/0)
 >  > mpacpi: ISA interrupt override 9 -> 9 (1/3)
 >  > mpacpi: configuring PCI bus 0 int routing
 >  > pciroute: dev 13 INTA on globint 22
 >  > pciroute: done adding entry
 >  > pciroute: dev 1 INTA on globint 16
 >  > pciroute: done adding entry
 >  > pciroute: dev 1 INTB on globint 17
 >  > pciroute: done adding entry
 >  > pciroute: dev 1 INTC on globint 18
 >  > pciroute: done adding entry
 >  > pciroute: dev 1 INTD on globint 19
 >  > pciroute: done adding entry
 >  > pciroute: dev 2 INTA on globint 17
 >  > pciroute: done adding entry
 >  > pciroute: dev 2 INTB on globint 18
 >  > pciroute: done adding entry
 >  > pciroute: dev 2 INTC on globint 19
 >  > pciroute: done adding entry
 >  > pciroute: dev 2 INTD on globint 16
 >  > pciroute: done adding entry
 >  > pciroute: dev 3 INTA on globint 18
 >  > pciroute: done adding entry
 >  > pciroute: dev 3 INTB on globint 19
 >  > pciroute: done adding entry
 >  > pciroute: dev 3 INTC on globint 16
 >  > pciroute: done adding entry
 >  > pciroute: dev 3 INTD on globint 17
 >  > pciroute: done adding entry
 >  > pciroute: dev 8 INTA on globint 19
 >  > pciroute: done adding entry
 >  > pciroute: dev 10 INTA on globint 18
 >  > pciroute: done adding entry
 >  > pciroute: dev 10 INTB on globint 21
 >  > pciroute: done adding entry
 >  > pciroute: dev 12 INTA on globint 23
 >  > pciroute: done adding entry
 >  > pciroute: dev 14 INTA on globint 18
 >  > pciroute: done adding entry
 >  > pciroute: dev 15 INTA on globint 21
 >  > pciroute: done adding entry
 >  > pciroute: done
 >  > mpacpi: configuring PCI bus 1 int routing
 >  > pciroute: dev 0 INTA on globint 16
 >  > pciroute: done adding entry
 >  > pciroute: dev 0 INTB on globint 17
 >  > pciroute: done adding entry
 >  > pciroute: dev 0 INTC on globint 18
 >  > pciroute: done adding entry
 >  > pciroute: dev 0 INTD on globint 19
 >  > pciroute: done adding entry
 >  > pciroute: done
 >  > mpacpi: configuring PCI bus 2 int routing
 >  > pciroute: dev 0 INTA on globint 17
 >  > pciroute: done adding entry
 >  > pciroute: dev 0 INTB on globint 18
 >  > pciroute: done adding entry
 >  > pciroute: dev 0 INTC on globint 19
 >  > pciroute: done adding entry
 >  > pciroute: dev 0 INTD on globint 16
 >  > pciroute: done adding entry
 >  > pciroute: done
 >  > ioapic0: pin 2 attached to isa0 irq 0 (type 0<type=3D3D0> flags 0<pol=
 =3D3D0,t=3D
 >  rig=3D3D0>)
 >  > ioapic0: pin 1 attached to isa0 irq 1 (type 0<type=3D3D0> flags 0<pol=
 =3D3D0,t=3D
 >  rig=3D3D0>)
 >  > ioapic0: pin 3 attached to isa0 irq 3 (type 0<type=3D3D0> flags 0<pol=
 =3D3D0,t=3D
 >  rig=3D3D0>)
 >  > ioapic0: pin 4 attached to isa0 irq 4 (type 0<type=3D3D0> flags 0<pol=
 =3D3D0,t=3D
 >  rig=3D3D0>)
 >  > ioapic0: pin 5 attached to isa0 irq 5 (type 0<type=3D3D0> flags 0<pol=
 =3D3D0,t=3D
 >  rig=3D3D0>)
 >  > ioapic0: pin 6 attached to isa0 irq 6 (type 0<type=3D3D0> flags 0<pol=
 =3D3D0,t=3D
 >  rig=3D3D0>)
 >  > ioapic0: pin 7 attached to isa0 irq 7 (type 0<type=3D3D0> flags 0<pol=
 =3D3D0,t=3D
 >  rig=3D3D0>)
 >  > ioapic0: pin 8 attached to isa0 irq 8 (type 0<type=3D3D0> flags 0<pol=
 =3D3D0,t=3D
 >  rig=3D3D0>)
 >  > ioapic0: pin 9 attached to isa0 irq 9 (type 0<type=3D3D0> flags
 >  > 0xd<pol=3D3D0x1=3D3DAct Hi,trig=3D3D0x3=3D3DLevel>)
 >  > ioapic0: pin 10 attached to isa0 irq 10 (type 0<type=3D3D0> flags 0<po=
 l=3D3D0=3D
 >  ,trig=3D3D0>)
 >  > ioapic0: pin 11 attached to isa0 irq 11 (type 0<type=3D3D0> flags 0<po=
 l=3D3D0=3D
 >  ,trig=3D3D0>)
 >  > ioapic0: pin 12 attached to isa0 irq 12 (type 0<type=3D3D0> flags 0<po=
 l=3D3D0=3D
 >  ,trig=3D3D0>)
 >  > ioapic0: pin 13 attached to isa0 irq 13 (type 0<type=3D3D0> flags 0<po=
 l=3D3D0=3D
 >  ,trig=3D3D0>)
 >  > ioapic0: pin 14 attached to isa0 irq 14 (type 0<type=3D3D0> flags 0<po=
 l=3D3D0=3D
 >  ,trig=3D3D0>)
 >  > ioapic0: pin 15 attached to isa0 irq 15 (type 0<type=3D3D0> flags 0<po=
 l=3D3D0=3D
 >  ,trig=3D3D0>)
 >  > ioapic0: pin 22 attached to pci0 device 13 INT_A (type 0<type=3D3D0> f=
 lags
 >  > 0xf<pol=3D3D0x3=3D3DAct Lo,trig=3D3D0x3=3D3DLevel>)
 >  > ioapic0: pin 16 attached to pci0 device 1 INT_A (type 0<type=3D3D0> fl=
 ags
 >  > 0xf<pol=3D3D0x3=3D3DAct Lo,trig=3D3D0x3=3D3DLevel>)
 >  > ioapic0: pin 17 attached to pci0 device 1 INT_B (type 0<type=3D3D0> fl=
 ags
 >  > 0xf<pol=3D3D0x3=3D3DAct Lo,trig=3D3D0x3=3D3DLevel>)
 >  > ioapic0: pin 18 attached to pci0 device 1 INT_C (type 0<type=3D3D0> fl=
 ags
 >  > 0xf<pol=3D3D0x3=3D3DAct Lo,trig=3D3D0x3=3D3DLevel>)
 >  > ioapic0: pin 19 attached to pci0 device 1 INT_D (type 0<type=3D3D0> fl=
 ags
 >  > 0xf<pol=3D3D0x3=3D3DAct Lo,trig=3D3D0x3=3D3DLevel>)
 >  > ioapic0: pin 17 attached to pci0 device 2 INT_A (type 0<type=3D3D0> fl=
 ags
 >  > 0xf<pol=3D3D0x3=3D3DAct Lo,trig=3D3D0x3=3D3DLevel>)
 >  > ioapic0: pin 18 attached to pci0 device 2 INT_B (type 0<type=3D3D0> fl=
 ags
 >  > 0xf<pol=3D3D0x3=3D3DAct Lo,trig=3D3D0x3=3D3DLevel>)
 >  > ioapic0: pin 19 attached to pci0 device 2 INT_C (type 0<type=3D3D0> fl=
 ags
 >  > 0xf<pol=3D3D0x3=3D3DAct Lo,trig=3D3D0x3=3D3DLevel>)
 >  > ioapic0: pin 16 attached to pci0 device 2 INT_D (type 0<type=3D3D0> fl=
 ags
 >  > 0xf<pol=3D3D0x3=3D3DAct Lo,trig=3D3D0x3=3D3DLevel>)
 >  > ioapic0: pin 18 attached to pci0 device 3 INT_A (type 0<type=3D3D0> fl=
 ags
 >  > 0xf<pol=3D3D0x3=3D3DAct Lo,trig=3D3D0x3=3D3DLevel>)
 >  > ioapic0: pin 19 attached to pci0 device 3 INT_B (type 0<type=3D3D0> fl=
 ags
 >  > 0xf<pol=3D3D0x3=3D3DAct Lo,trig=3D3D0x3=3D3DLevel>)
 >  > ioapic0: pin 16 attached to pci0 device 3 INT_C (type 0<type=3D3D0> fl=
 ags
 >  > 0xf<pol=3D3D0x3=3D3DAct Lo,trig=3D3D0x3=3D3DLevel>)
 >  > ioapic0: pin 17 attached to pci0 device 3 INT_D (type 0<type=3D3D0> fl=
 ags
 >  > 0xf<pol=3D3D0x3=3D3DAct Lo,trig=3D3D0x3=3D3DLevel>)
 >  > ioapic0: pin 19 attached to pci0 device 8 INT_A (type 0<type=3D3D0> fl=
 ags
 >  > 0xf<pol=3D3D0x3=3D3DAct Lo,trig=3D3D0x3=3D3DLevel>)
 >  > ioapic0: pin 18 attached to pci0 device 10 INT_A (type 0<type=3D3D0> f=
 lags
 >  > 0xf<pol=3D3D0x3=3D3DAct Lo,trig=3D3D0x3=3D3DLevel>)
 >  > ioapic0: pin 21 attached to pci0 device 10 INT_B (type 0<type=3D3D0> f=
 lags
 >  > 0xf<pol=3D3D0x3=3D3DAct Lo,trig=3D3D0x3=3D3DLevel>)
 >  > ioapic0: pin 23 attached to pci0 device 12 INT_A (type 0<type=3D3D0> f=
 lags
 >  > 0xf<pol=3D3D0x3=3D3DAct Lo,trig=3D3D0x3=3D3DLevel>)
 >  > ioapic0: pin 18 attached to pci0 device 14 INT_A (type 0<type=3D3D0> f=
 lags
 >  > 0xf<pol=3D3D0x3=3D3DAct Lo,trig=3D3D0x3=3D3DLevel>)
 >  > ioapic0: pin 21 attached to pci0 device 15 INT_A (type 0<type=3D3D0> f=
 lags
 >  > 0xf<pol=3D3D0x3=3D3DAct Lo,trig=3D3D0x3=3D3DLevel>)
 >  > ioapic0: pin 16 attached to pci1 device 0 INT_A (type 0<type=3D3D0> fl=
 ags
 >  > 0xf<pol=3D3D0x3=3D3DAct Lo,trig=3D3D0x3=3D3DLevel>)
 >  > ioapic0: pin 17 attached to pci1 device 0 INT_B (type 0<type=3D3D0> fl=
 ags
 >  > 0xf<pol=3D3D0x3=3D3DAct Lo,trig=3D3D0x3=3D3DLevel>)
 >  > ioapic0: pin 18 attached to pci1 device 0 INT_C (type 0<type=3D3D0> fl=
 ags
 >  > 0xf<pol=3D3D0x3=3D3DAct Lo,trig=3D3D0x3=3D3DLevel>)
 >  > ioapic0: pin 19 attached to pci1 device 0 INT_D (type 0<type=3D3D0> fl=
 ags
 >  > 0xf<pol=3D3D0x3=3D3DAct Lo,trig=3D3D0x3=3D3DLevel>)
 >  > ioapic0: pin 17 attached to pci2 device 0 INT_A (type 0<type=3D3D0> fl=
 ags
 >  > 0xf<pol=3D3D0x3=3D3DAct Lo,trig=3D3D0x3=3D3DLevel>)
 >  > ioapic0: pin 18 attached to pci2 device 0 INT_B (type 0<type=3D3D0> fl=
 ags
 >  > 0xf<pol=3D3D0x3=3D3DAct Lo,trig=3D3D0x3=3D3DLevel>)
 >  > ioapic0: pin 19 attached to pci2 device 0 INT_C (type 0<type=3D3D0> fl=
 ags
 >  > 0xf<pol=3D3D0x3=3D3DAct Lo,trig=3D3D0x3=3D3DLevel>)
 >  > ioapic0: pin 16 attached to pci2 device 0 INT_D (type 0<type=3D3D0> fl=
 ags
 >  > 0xf<pol=3D3D0x3=3D3DAct Lo,trig=3D3D0x3=3D3DLevel>)
 >  > ...
 >  >
 >  > Regards,
 >  > Andrius V
 >
 >
 >  I had some unexpected "breakthrough" on this bug, which probably won't
 >  bring me closer to solution, but should help at least to narrow down
 >  on the places to look at.
 >  I managed to make network interface vte(4) work by forcing high
 >  polarity on its pin 19 (instead of level low, it is level high).
 >  Network started to work "properly" and was stable (could use ssh, scp,
 >  connect to internet). The main caveat, it causes interrupt storm.
 >  intrctl list would list huge number of interrupts.
 >  So seemingly it issue only visible on level type interrupt with low
 >  polarity. For some reason they are not registered or acknowledge
 >  somewhere in the between device and ioapic.
 >
 >
 >  --- a/sys/arch/x86/x86/ioapic.c
 >  +++ b/sys/arch/x86/x86/ioapic.c
 >  @@ -447,6 +447,8 @@ apic_set_redir(struct ioapic_softc *sc, int pin,
 >  int idt_vec,
 >                          }
 >                  }
 >          }
 >  +       if (pin =3D3D=3D3D 19)
 >  +               redlo &=3D3D ~IOAPIC_REDLO_ACTLO;
 >          ioapic_write(sc, IOAPIC_REDHI(pin), redhi);
 >          ioapic_write(sc, IOAPIC_REDLO(pin), redlo);
 >          if (mp_verbose)
 >
 >
 >  I tried to change ordering in the INTRSTUB_56()
 >  (sys/arch/i386/i386/vector.S) for level interrupts in different ways,
 >  but that didn't help. So likely it is not masking/unmasking/timeouts.
 >  But it's hard to test this part, since it is an assembly code.
 >  In general, initialization still looks good, but what happens with
 >  those interrupts later I have no idea still.
 >
 >  Same trick didn't fully work for ohci/ehci, system freezes on boot. If
 >  just one of them does the trick, interrupt storm is also visible, more
 >  message on usb, but devices fail to attach.
 >  Thus, USB reacts to polarity changes, just probably other factors
 >  don't allow to work.
 >
 >  If switching polarity making it work gives you any idea on what can be
 >  checked or what can be an issue, please share with me.
 >
 >  Regards,
 >  Andrius V
 >
 
 Hi,
 
 After polarity "workaround" decided to "waste" few days on this bug
 again after long break. Unfortunately without success finding the root
 cause or proper fix.
 
 For the reminder for those not wanting to read a full thread - level
 type (PCI) interrupts are not working on Vortex86DX3 SoC when ACPI is
 enabled, leading to no USB devices detected, network and audio not
 working.
 
 If ACPI is disabled they work properly.
 
 A workaround for the vte network interface is to force high polarity
 for ioapic pin in the code as described in the previous email
 (interrupts work, but cause interrupt storm). For USB it didn't work
 properly, system freezes on boot. Didn't test audio (and it shares pin
 with ohci).
 Not sure about SCI (it is high polarity by default, no idea how to
 trigger it), thus can't tell if it is level low specific or just level
 type specific.
 
 During investigation checked and compared dumps for the lapic, ioapic,
 southbridge, northbridge, pci devices. All registers looks good and
 match freebsd and linux (these systems have working interrupts).
 Differences are mainly expected or irrelevant (different vector
 number, destination, etc). So my understanding that initialization is
 not the cause of the problem and work correctly, unless there is some
 ordering or wrong timing involved (which can in theory be, if device
 interrupts too early for some reason during ioapic init(?)).
 
 Also compared init code between netbsd and freebsd, but couldn't
 identify anything really standing out, process looks similar to my
 untrained eyes.
 
 Tried to trigger polarity (and type) from DDB on ioapic affected vte
 pin (and also separately on lapic to bypass ioapic), vte_intr reacts
 to these and prints to the log. So again, it looks like everything is
 OK with the "plumbing" itself.
 
 It does seem so though, that ioapic_level stubs (and subsequent ack,
 unmask, mask) are never triggered. Hard to verify directly due to
 assembly. But was trying to add instructions to cause crashes or
 stalls. Boot was still successful, thus my assumption was they are not
 executed.
 
 So, in general, it does seem interrupts would work properly if they
 were issued or detected. But something just prevents that. However, I
 hit the brick wall here.
 I tried reordering, early unmasking during PCI interrupt establishment
 and ioapic init in multiple ways, but with none of them changing the
 behaviour.
 Complete silence on PCI device, intrctl list show 0 interrupts.
 Unfortunately, I think I spent too much time on this again on this
 rare SoC anyway (neither I am using it for anything now), thus giving
 up again, but maybe somebody will find this info useful in the future.
 Myself I really have no more ideas on how to proceed.
 
 Potentially I will write to the SoC manufacturer if they know have any
 known errata but I don't expect much here, considering linux and
 freebsd works without issues.
 
 Regards,
 Andrius V