NetBSD-Bugs archive
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index][Old Index]
Re: kern/60029: panic: cpu0: softints stuck for 16 seconds
The following reply was made to PR kern/60029; it has been noted by GNATS.
From: Thomas Klausner <wiz%netbsd.org@localhost>
To: Taylor R Campbell <riastradh%netbsd.org@localhost>
Cc: NetBSD bugtracking <gnats-bugs%netbsd.org@localhost>
Subject: Re: kern/60029: panic: cpu0: softints stuck for 16 seconds
Date: Sun, 22 Feb 2026 20:48:39 +0100
On Sun, Feb 22, 2026 at 07:24:59PM +0100, Taylor R Campbell wrote:
> > Date: Sun, 22 Feb 2026 14:02:33 +0100
> > From: Thomas Klausner <wiz%netbsd.org@localhost>
> >
> > 0 > 935 7 16 40200 ffff8fb6be107400 pgdaemon
> > [...]
> > 0 > 4 7 0 200 ffff8fc60b25f800 softbio/0
>
> These two look suspicious.
How do you figure?
> Now, I think the stack traces won't be reliable because the threads in
> question are running on the CPU (so we really want `mach cpu N' and
> `bt', but PR bin/58010: crash(8) doesn't support `mach cpu N' to
> examine registers/stack of other CPUs), but just in case, here's the
> stack trace for softbio/0:
>
> > crash> bt/a ffff8fc60b25f800
> > trace: pid 0 lid 4 at 0xffffa72469e83fa0
> > uvm_aio_aiodone() at uvm_aio_aiodone+0xbe
> > dkiodone() at dkiodone+0xa8
> > lddone() at lddone+0xf
> > nvme_q_complete() at nvme_q_complete+0xf2
> > softint_dispatch() at softint_dispatch+0x112
>
> If you have netbsd.gdb, can you get output of `info line
> *(uvm_aio_aiodone+0xbe)' in gdb from it?
Sure:
(gdb) info line *(uvm_aio_aiodone+0xbe)
Line 528 of "/usr/src/sys/uvm/uvm_pager.c" starts at address 0xffffffff80dd7549 <uvm_aio_aiodone+190> and ends at 0xffffffff80dd7557 <uvm_aio_aiodone+204>.
> And can you get the stack trace for pgdaemon too?
>
> crash> bt/a ffff8fb6be107400
crash> bt/a ffff8fb6be107400
trace: pid 0 lid 935 at 0xffffa7247baabee0
pager_map_wanted_lock() at uvmpd_lock
uvm_pageout() at uvm_pageout+0x549
> My guess is the softbio thread might be waiting for this lock:
>
> 528 if (write && (bp->b_cflags & BC_AGE) != 0) {
> -> 529 mutex_enter(bp->b_objlock);
> 530 vwakeup(bp);
> 531 mutex_exit(bp->b_objlock);
> 532 }
>
> https://nxr.netbsd.org/xref/src/sys/uvm/uvm_pager.c?r=1.131#529
>
> This is probably either buffer_lock or vp->v_interlock for some vnode.
> In case it's buffer_lock, can you show this?
>
> crash> x/Lx buffer_lock
crash> x/Lx buffer_lock
buffer_lock: 0
> Since they're not _sleeping_ on a wait-channel, whoever holds the lock
> (whether it's buffer_lock or vp->v_interlock) is probably on running
> on another CPU. That narrows it down to:
>
> PID LID S CPU FLAGS STRUCT LWP * NAME WAIT
> 20857>20857 7 17 8060000 ffff8fa8dbeac000 moc
> 2482 > 2482 7 23 8060000 ffff8fb50065c800 moc
> 26437>26437 7 9 8060000 ffff8fb50065c000 moc
> 12057>12057 7 31 8020000 ffff8fbf931ab000 moc
> 18565>18565 7 21 8060000 ffff8fc129ed9c00 moc
> 12544>12544 7 22 8060000 ffff8fc129ed9000 moc
> 13508>13508 7 2 8060000 ffff8fb908ee8800 moc
> 3522 > 3522 7 20 8020000 ffff8fc1d885ac00 moc
> 23657>23657 7 3 8020000 ffff8fb90950c800 moc
> 3031 > 3031 7 28 8060000 ffff8fb10d2b3400 moc
> 13759>13759 7 5 8020000 ffff8faac5fec000 moc
> 25301>25301 7 15 8060000 ffff8fc2d6fb1400 moc
> 13934>13934 7 14 8060000 ffff8fb8f1b68800 moc
> 9619 > 9619 7 11 8060000 ffff8fb8f1b68000 moc
> 11474>11474 7 13 8020000 ffff8fb7c17d2c00 moc
> 23339>23339 7 29 8060000 ffff8fb7b9e4cc00 moc
> 5510 > 5510 7 10 8060000 ffff8fc36cf3ec00 moc
> 2567 > 2567 7 27 8020000 ffff8fc331b27c00 moc
> 29924>29924 7 4 8020000 ffff8fc561094c00 moc
> 20557>20557 7 26 8020000 ffff8fc561094800 moc
> 23674>23674 7 6 8060000 ffff8fc38a557000 moc
> 27935>27935 7 25 8060000 ffff8faaa08c1000 moc
> 19811>19811 7 7 8060000 ffff8fbdc9009800 moc
> 14383>14383 7 18 8060000 ffff8fbac25c2000 moc
> 19788>19788 7 19 8060000 ffff8fc08f854c00 moc
> 11290>11290 7 24 8060000 ffff8fb0ea2d1800 moc
> 28134>28134 7 30 8020000 ffff8fc2010df800 moc
> 27882>27882 7 8 8060000 ffff8fbd4892ac00 moc
> 21709>21709 7 12 8060000 ffff8fb7cd8e5400 moc
> 29702>29702 7 1 8020000 ffff8fb90a69dc00 moc
> 0 > 935 7 16 40200 ffff8fb6be107400 pgdaemon
> 0 > 4 7 0 200 ffff8fc60b25f800 softbio/0
>
> Could try getting the stack traces for the moc processes but they're
> probably in userland without any locks held...
Throws a lot of errors, but here is the output:
bt/a ffff8fa8dbeac000
crash: _kvm_kvatop(0)
crash: _kvm_kvatop(40000029596bc18)
crash: kvm_read(0x40000029596bc18, 8): invalid translation (invalid level 4 PDE)
Crash version 11.99.5, image version 11.99.5.
Kernel compiled without options LOCKDEBUG.
System panicked: cpu0: softints stuck for 16 seconds
Backtrace from time of crash is available.
trace: pid 20857 lid 20857 at 0x40000029596bc10
_KERNEL_OPT_NCOM_PUC() at _KERNEL_OPT_NCOM_PUC+0x5
bt/a ffff8fb50065c800
crash: _kvm_kvatop(0)
crash: _kvm_kvatop(400000200000008)
crash: kvm_read(0x400000200000008, 8): invalid translation (invalid level 4 PDE)
Crash version 11.99.5, image version 11.99.5.
Kernel compiled without options LOCKDEBUG.
System panicked: cpu0: softints stuck for 16 seconds
Backtrace from time of crash is available.
trace: pid 2482 lid 2482 at 0x400000200000000
_KERNEL_OPT_NCOM_PUC() at _KERNEL_OPT_NCOM_PUC+0x5
bt/a ffff8fb50065c000
crash: _kvm_kvatop(0)
Crash version 11.99.5, image version 11.99.5.
Kernel compiled without options LOCKDEBUG.
System panicked: cpu0: softints stuck for 16 seconds
Backtrace from time of crash is available.
trace: pid 26437 lid 26437 at 0x0
0:
bt/a ffff8fbf931ab000
crash: _kvm_kvatop(0)
crash: _kvm_kvatop(400000200000008)
crash: kvm_read(0x400000200000008, 8): invalid translation (invalid level 4 PDE)
Crash version 11.99.5, image version 11.99.5.
Kernel compiled without options LOCKDEBUG.
System panicked: cpu0: softints stuck for 16 seconds
Backtrace from time of crash is available.
trace: pid 12057 lid 12057 at 0x400000200000000
_KERNEL_OPT_NCOM_PUC() at _KERNEL_OPT_NCOM_PUC+0x5
bt/a ffff8fc129ed9c00
crash: _kvm_kvatop(0)
crash: _kvm_kvatop(95929c78)
crash: kvm_read(0x95929c78, 8): invalid translation (invalid level 4 PDE)
Crash version 11.99.5, image version 11.99.5.
Kernel compiled without options LOCKDEBUG.
System panicked: cpu0: softints stuck for 16 seconds
Backtrace from time of crash is available.
trace: pid 18565 lid 18565 at 0x95929c70
__kernel_end() at 0
bt/a ffff8fc129ed9000
crash: _kvm_kvatop(0)
crash: _kvm_kvatop(400000200000008)
crash: kvm_read(0x400000200000008, 8): invalid translation (invalid level 4 PDE)
Crash version 11.99.5, image version 11.99.5.
Kernel compiled without options LOCKDEBUG.
System panicked: cpu0: softints stuck for 16 seconds
Backtrace from time of crash is available.
trace: pid 12544 lid 12544 at 0x400000200000000
_KERNEL_OPT_NCOM_PUC() at _KERNEL_OPT_NCOM_PUC+0x5
bt/a ffff8fb908ee8800
crash: _kvm_kvatop(0)
crash: _kvm_kvatop(400000200000008)
crash: kvm_read(0x400000200000008, 8): invalid translation (invalid level 4 PDE)
Crash version 11.99.5, image version 11.99.5.
Kernel compiled without options LOCKDEBUG.
System panicked: cpu0: softints stuck for 16 seconds
Backtrace from time of crash is available.
trace: pid 13508 lid 13508 at 0x400000200000000
_KERNEL_OPT_NCOM_PUC() at _KERNEL_OPT_NCOM_PUC+0x5
bt/a ffff8fc1d885ac00
crash: _kvm_kvatop(0)
crash: _kvm_kvatop(400000200000008)
crash: kvm_read(0x400000200000008, 8): invalid translation (invalid level 4 PDE)
Crash version 11.99.5, image version 11.99.5.
Kernel compiled without options LOCKDEBUG.
System panicked: cpu0: softints stuck for 16 seconds
Backtrace from time of crash is available.
trace: pid 3522 lid 3522 at 0x400000200000000
_KERNEL_OPT_NCOM_PUC() at _KERNEL_OPT_NCOM_PUC+0x5
bt/a ffff8fb90950c800
crash: _kvm_kvatop(0)
crash: _kvm_kvatop(945e4c78)
crash: kvm_read(0x945e4c78, 8): invalid translation (invalid level 4 PDE)
Crash version 11.99.5, image version 11.99.5.
Kernel compiled without options LOCKDEBUG.
System panicked: cpu0: softints stuck for 16 seconds
Backtrace from time of crash is available.
trace: pid 23657 lid 23657 at 0x945e4c70
__kernel_end() at 0
bt/a ffff8fb10d2b3400
crash: _kvm_kvatop(0)
crash: _kvm_kvatop(400000200000008)
crash: kvm_read(0x400000200000008, 8): invalid translation (invalid level 4 PDE)
Crash version 11.99.5, image version 11.99.5.
Kernel compiled without options LOCKDEBUG.
System panicked: cpu0: softints stuck for 16 seconds
Backtrace from time of crash is available.
trace: pid 3031 lid 3031 at 0x400000200000000
_KERNEL_OPT_NCOM_PUC() at _KERNEL_OPT_NCOM_PUC+0x5
bt/a ffff8faac5fec000
crash: _kvm_kvatop(0)
Crash version 11.99.5, image version 11.99.5.
Kernel compiled without options LOCKDEBUG.
System panicked: cpu0: softints stuck for 16 seconds
Backtrace from time of crash is available.
trace: pid 13759 lid 13759 at 0xffffa724957e9cd0
uvmfault_promote() at uvmfault_promote+0x4bb
uvm_fault_internal() at uvm_fault_internal+0xd5a
trap() at trap+0x46b
--- trap (number 6) ---
6fbeee905c3f:
bt/a ffff8fc2d6fb1400
crash: _kvm_kvatop(0)
Crash version 11.99.5, image version 11.99.5.
Kernel compiled without options LOCKDEBUG.
System panicked: cpu0: softints stuck for 16 seconds
Backtrace from time of crash is available.
trace: pid 25301 lid 25301 at 0xffffa7247f844cd0
uvmfault_promote() at uvmfault_promote+0x4bb
uvm_fault_internal() at uvm_fault_internal+0xd5a
trap() at trap+0x46b
--- trap (number 6) ---
f91581:
bt/a ffff8fb8f1b68800
crash: _kvm_kvatop(0)
crash: _kvm_kvatop(400000200000008)
crash: kvm_read(0x400000200000008, 8): invalid translation (invalid level 4 PDE)
Crash version 11.99.5, image version 11.99.5.
Kernel compiled without options LOCKDEBUG.
System panicked: cpu0: softints stuck for 16 seconds
Backtrace from time of crash is available.
trace: pid 13934 lid 13934 at 0x400000200000000
_KERNEL_OPT_NCOM_PUC() at _KERNEL_OPT_NCOM_PUC+0x5
bt/a ffff8fb8f1b68000
crash: _kvm_kvatop(0)
crash: _kvm_kvatop(400000200000008)
crash: kvm_read(0x400000200000008, 8): invalid translation (invalid level 4 PDE)
Crash version 11.99.5, image version 11.99.5.
Kernel compiled without options LOCKDEBUG.
System panicked: cpu0: softints stuck for 16 seconds
Backtrace from time of crash is available.
trace: pid 9619 lid 9619 at 0x400000200000000
_KERNEL_OPT_NCOM_PUC() at _KERNEL_OPT_NCOM_PUC+0x5
bt/a ffff8fb7c17d2c00
crash: _kvm_kvatop(0)
Crash version 11.99.5, image version 11.99.5.
Kernel compiled without options LOCKDEBUG.
System panicked: cpu0: softints stuck for 16 seconds
Backtrace from time of crash is available.
trace: pid 11474 lid 11474 at 0xffffa72564f3aff0
trap() at trap+0x46b
--- trap (number 6) ---
137dee:
bt/a ffff8fb7b9e4cc00
crash: _kvm_kvatop(0)
crash: _kvm_kvatop(400000200000008)
crash: kvm_read(0x400000200000008, 8): invalid translation (invalid level 4 PDE)
Crash version 11.99.5, image version 11.99.5.
Kernel compiled without options LOCKDEBUG.
System panicked: cpu0: softints stuck for 16 seconds
Backtrace from time of crash is available.
trace: pid 23339 lid 23339 at 0x400000200000000
_KERNEL_OPT_NCOM_PUC() at _KERNEL_OPT_NCOM_PUC+0x5
bt/a ffff8fc36cf3ec00
crash: _kvm_kvatop(0)
Crash version 11.99.5, image version 11.99.5.
Kernel compiled without options LOCKDEBUG.
System panicked: cpu0: softints stuck for 16 seconds
Backtrace from time of crash is available.
trace: pid 5510 lid 5510 at 0xffff8fb848e1eb28
__kernel_end() at 0
bt/a ffff8fc331b27c00
crash: _kvm_kvatop(0)
crash: _kvm_kvatop(400000200000008)
crash: kvm_read(0x400000200000008, 8): invalid translation (invalid level 4 PDE)
Crash version 11.99.5, image version 11.99.5.
Kernel compiled without options LOCKDEBUG.
System panicked: cpu0: softints stuck for 16 seconds
Backtrace from time of crash is available.
trace: pid 2567 lid 2567 at 0x400000200000000
_KERNEL_OPT_NCOM_PUC() at _KERNEL_OPT_NCOM_PUC+0x5
bt/a ffff8fc561094c00
crash: _kvm_kvatop(0)
Crash version 11.99.5, image version 11.99.5.
Kernel compiled without options LOCKDEBUG.
System panicked: cpu0: softints stuck for 16 seconds
Backtrace from time of crash is available.
trace: pid 29924 lid 29924 at 0xffffa7249535bcd0
uvmfault_promote() at uvmfault_promote+0x4bb
uvm_fault_internal() at uvm_fault_internal+0xd5a
trap() at trap+0x46b
--- trap (number 6) ---
f2363f:
bt/a ffff8fc561094800
crash: _kvm_kvatop(0)
crash: _kvm_kvatop(7d8a1c78)
crash: kvm_read(0x7d8a1c78, 8): invalid translation (invalid level 4 PDE)
Crash version 11.99.5, image version 11.99.5.
Kernel compiled without options LOCKDEBUG.
System panicked: cpu0: softints stuck for 16 seconds
Backtrace from time of crash is available.
trace: pid 20557 lid 20557 at 0x7d8a1c70
__kernel_end() at 0
bt/a ffff8fc38a557000
crash: _kvm_kvatop(0)
crash: _kvm_kvatop(95274c78)
crash: kvm_read(0x95274c78, 8): invalid translation (invalid level 4 PDE)
Crash version 11.99.5, image version 11.99.5.
Kernel compiled without options LOCKDEBUG.
System panicked: cpu0: softints stuck for 16 seconds
Backtrace from time of crash is available.
trace: pid 23674 lid 23674 at 0x95274c70
__kernel_end() at 0
bt/a ffff8faaa08c1000
crash: _kvm_kvatop(0)
crash: _kvm_kvatop(400000200000008)
crash: kvm_read(0x400000200000008, 8): invalid translation (invalid level 4 PDE)
Crash version 11.99.5, image version 11.99.5.
Kernel compiled without options LOCKDEBUG.
System panicked: cpu0: softints stuck for 16 seconds
Backtrace from time of crash is available.
trace: pid 27935 lid 27935 at 0x400000200000000
_KERNEL_OPT_NCOM_PUC() at _KERNEL_OPT_NCOM_PUC+0x5
bt/a ffff8fbdc9009800
crash: _kvm_kvatop(0)
crash: _kvm_kvatop(400000200000008)
crash: kvm_read(0x400000200000008, 8): invalid translation (invalid level 4 PDE)
Crash version 11.99.5, image version 11.99.5.
Kernel compiled without options LOCKDEBUG.
System panicked: cpu0: softints stuck for 16 seconds
Backtrace from time of crash is available.
trace: pid 19811 lid 19811 at 0x400000200000000
_KERNEL_OPT_NCOM_PUC() at _KERNEL_OPT_NCOM_PUC+0x5
bt/a ffff8fbac25c2000
crash: _kvm_kvatop(0)
crash: _kvm_kvatop(400000200000008)
crash: kvm_read(0x400000200000008, 8): invalid translation (invalid level 4 PDE)
Crash version 11.99.5, image version 11.99.5.
Kernel compiled without options LOCKDEBUG.
System panicked: cpu0: softints stuck for 16 seconds
Backtrace from time of crash is available.
trace: pid 14383 lid 14383 at 0x400000200000000
_KERNEL_OPT_NCOM_PUC() at _KERNEL_OPT_NCOM_PUC+0x5
bt/a ffff8fc08f854c00
crash: _kvm_kvatop(0)
Crash version 11.99.5, image version 11.99.5.
Kernel compiled without options LOCKDEBUG.
System panicked: cpu0: softints stuck for 16 seconds
Backtrace from time of crash is available.
trace: pid 19788 lid 19788 at 0xffffa72480c14cd0
uvmfault_promote() at uvmfault_promote+0x4bb
uvm_fault_internal() at uvm_fault_internal+0xd5a
trap() at trap+0x46b
--- trap (number 6) ---
d0bdee:
bt/a ffff8fb0ea2d1800
crash: _kvm_kvatop(0)
crash: _kvm_kvatop(400000200000008)
crash: kvm_read(0x400000200000008, 8): invalid translation (invalid level 4 PDE)
Crash version 11.99.5, image version 11.99.5.
Kernel compiled without options LOCKDEBUG.
System panicked: cpu0: softints stuck for 16 seconds
Backtrace from time of crash is available.
trace: pid 11290 lid 11290 at 0x400000200000000
_KERNEL_OPT_NCOM_PUC() at _KERNEL_OPT_NCOM_PUC+0x5
bt/a ffff8fc2010df800
crash: _kvm_kvatop(0)
crash: _kvm_kvatop(80c6bc78)
crash: kvm_read(0x80c6bc78, 8): invalid translation (invalid level 4 PDE)
Crash version 11.99.5, image version 11.99.5.
Kernel compiled without options LOCKDEBUG.
System panicked: cpu0: softints stuck for 16 seconds
Backtrace from time of crash is available.
trace: pid 28134 lid 28134 at 0x80c6bc70
__kernel_end() at 0
bt/a ffff8fbd4892ac00
crash: _kvm_kvatop(0)
crash: _kvm_kvatop(81364c78)
crash: kvm_read(0x81364c78, 8): invalid translation (invalid level 4 PDE)
Crash version 11.99.5, image version 11.99.5.
Kernel compiled without options LOCKDEBUG.
System panicked: cpu0: softints stuck for 16 seconds
Backtrace from time of crash is available.
trace: pid 27882 lid 27882 at 0x81364c70
__kernel_end() at 0
bt/a ffff8fb7cd8e5400
crash: _kvm_kvatop(0)
crash: _kvm_kvatop(400000200000008)
crash: kvm_read(0x400000200000008, 8): invalid translation (invalid level 4 PDE)
Crash version 11.99.5, image version 11.99.5.
Kernel compiled without options LOCKDEBUG.
System panicked: cpu0: softints stuck for 16 seconds
Backtrace from time of crash is available.
trace: pid 21709 lid 21709 at 0x400000200000000
_KERNEL_OPT_NCOM_PUC() at _KERNEL_OPT_NCOM_PUC+0x5
bt/a ffff8fb90a69dc00
crash: _kvm_kvatop(0)
crash: _kvm_kvatop(7d942c78)
crash: kvm_read(0x7d942c78, 8): invalid translation (invalid level 4 PDE)
Crash version 11.99.5, image version 11.99.5.
Kernel compiled without options LOCKDEBUG.
System panicked: cpu0: softints stuck for 16 seconds
Backtrace from time of crash is available.
trace: pid 29702 lid 29702 at 0x7d942c70
__kernel_end() at 0
Cheers,
Thomas
Home |
Main Index |
Thread Index |
Old Index