Re: hang when updating to 10.99.14

[Thomas Klausner <wiz%gatalith.at@localhost>]

Hi!

I just had the same panic, with a 10.99.14 GENERIC from 20250505,
during a bulk build.

Before that I was running 10.99.12 that was stable.

On Mon, Apr 21, 2025 at 03:50:08PM +0100, Patrick Welche wrote:
> On Mon, Apr 21, 2025 at 07:52:07AM +1000, matthew green wrote:
> > > (gdb) print panicstr
> > > $2 = 0xffffffff80f06020 <scratchstr> "cpu0: softints stuck for 16 seconds"
> > 
> > you'll need to find cpu0's back trace.  try feeding the core
> > into crash(8) instead of GDB, it should be easier to find.
> 
> crash> bt
> end() at 0
> kern_reboot() at kern_reboot+0x93
> vpanic() at vpanic+0x16b
> panic() at vprintf
> heartbeat() at heartbeat+0x1f2
> hardclock() at hardclock+0x9c
> Xresume_lapic_ltimer() at Xresume_lapic_ltimer+0x1e
> --- interrupt ---
> mutex_spin_exit() at mutex_spin_exit+0x5a
> callout_softclock() at callout_softclock+0xad
>  /usr/src/sys/arch/amd64/compile/QUARK/../../../../kern/kern_timeout.c:873 v1.79
> softint_dispatch() at softint_dispatch+0x8f
>  /usr/src/sys/arch/amd64/compile/QUARK/../../../../kern/kern_softint.c:605 v1.76
>  (points to an SDT_PROBE ?! so handler already called and done?)

# crash -M netbsd.10.core -N kernel.gdb
Crash version 10.99.14, image version 10.99.14.
crash: _kvm_kvatop(0)
Kernel compiled without options LOCKDEBUG.
System panicked: cpu0: softints stuck for 16 seconds
Backtrace from time of crash is available.
crash> bt
__kernel_end() at 0
kern_reboot() at kern_reboot+0x93
vpanic() at vpanic+0x17b
panic() at printf_nostamp
heartbeat() at heartbeat+0x34c
hardclock() at hardclock+0x8b
Xresume_lapic_ltimer() at Xresume_lapic_ltimer+0x1e
--- interrupt ---
Xspllower() at Xspllower+0xe
callout_softclock() at callout_softclock+0xd9
softint_dispatch() at softint_dispatch+0x112
DDB lost frame for Xsoftintr+0x4c, trying 0xffffb0a469e9b0f0
Xsoftintr() at Xsoftintr+0x4c
--- interrupt ---
c7af2521cbef2525:

crash> ps | grep '>' | grep -v idle/
5546 > 5546 7   0   8020000   ffffa6fffa942000     ghc-pkg-9.10.1
25332>25332 7   3   8020100   ffffa70449009c00                ksh
16311>16311 7   1   8020100   ffffa70547db1000                ksh
14468>14468 7   2   8020100   ffffa70bfb23e000                ksh
1560 > 1560 7   1   8020100   ffffa6ff3b4de800               cron
3827 > 3827 7   4   8020100   ffffa6ff228bec00               ntpd
0    >    5 7   0       200   ffffa70e6f261c00          softclk/0
0    >    0 7   5       240   ffffffff8188c000            swapper

crash> show proc /a ffffa6fffa942000
lwp_t ffffa6fffa942000
ghc-pkg-9.10.1: pid 5546 proc ffffa70c6c6ccac0 vmspace/map ffffa70be6e73c80 flags 4000
> lwp 5546 ffffa6fffa942000 pcb ffffb0a488044000
    stat 7 flags 8020000 cpu 0 pri 43 ref 0
crash> show proc /a ffffa70449009c00
lwp_t ffffa70449009c00
ksh: pid 25332 proc ffffa70420e5e7c0 vmspace/map ffffa6f180115580 flags 4000
> lwp 25332 ffffa70449009c00 pcb ffffb0a487120000
    stat 7 flags 8020100 cpu 3 pri 43 ref 0
crash> show proc /a ffffa70547db1000
lwp_t ffffa70547db1000
ksh: pid 16311 proc ffffa6fc1f924840 vmspace/map ffffa708220c6040 flags 4000
> lwp 16311 ffffa70547db1000 pcb ffffb0a49051c000
    stat 7 flags 8020100 cpu 1 pri 43 ref 0
crash> show proc /a ffffa70bfb23e000
lwp_t ffffa70bfb23e000
ksh: pid 14468 proc ffffa70164440b40 vmspace/map ffffa6ff26ac9900 flags 4000
> lwp 14468 ffffa70bfb23e000 pcb ffffb0a488a73000
    stat 7 flags 8020100 cpu 2 pri 43 ref 0
crash> show proc /a ffffa6ff3b4de800
lwp_t ffffa6ff3b4de800
cron: pid 1560 proc ffffa6ff313ff800 vmspace/map ffffa6ff310b2040 flags 200
> lwp 1560 ffffa6ff3b4de800 pcb ffffb0a47cfc5000
    stat 7 flags 8020100 cpu 1 pri 43 ref 0
crash> show proc /a ffffa6ff228bec00
lwp_t ffffa6ff228bec00
ntpd: pid 3827 proc ffffa6ff22b97100 vmspace/map ffffa6ff28de9040 flags 200
> lwp 3827 ffffa6ff228bec00 pcb ffffb0a47bbd0000
    stat 7 flags 8020100 cpu 4 pri 43 ref 0
crash> show proc /a ffffa70e6f261c00
lwp_t ffffa70e6f261c00
system: pid 0 proc ffffffff8188c780 vmspace/map ffffffff819769e0 flags 20002
> lwp 5 [softclk/0] ffffa70e6f261c00 pcb ffffb0a469e96000
    stat 7 flags 200 cpu 0 pri 220 ref 0
  lwp 4 [softbio/0] ffffa70e6f261800 pcb ffffb0a469e8f000
    stat 1 flags 200 cpu 0 pri 221 ref 0
  lwp 3 [softnet/0] ffffa70e6f261400 pcb ffffb0a469e88000
    stat 2 flags 200 cpu 0 pri 222 ref 0
  lwp 2 [idle/0] ffffa70e6f261000 pcb ffffb0a469a34000
    stat 1 flags 201 cpu 0 pri 0 ref 0
> lwp 0 [swapper] ffffffff8188c000 pcb ffffffff81d56000
    stat 7 flags 240 cpu 5 pri 125 ref 0
crash> show proc /a ffffffff8188c000
lwp_t ffffffff8188c000
system: pid 0 proc ffffffff8188c780 vmspace/map ffffffff819769e0 flags 20002
> lwp 0 [swapper] ffffffff8188c000 pcb ffffffff81d56000
    stat 7 flags 240 cpu 5 pri 125 ref 0


Anything else I can provide?
 Thomas