NetBSD-Bugs archive
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index][Old Index]
kern/59663: ffs_snapshot_read -> uvm_fault (or pool page empty) doing dump/restore with snapshot
>Number: 59663
>Category: kern
>Synopsis: ffs_snapshot_read -> uvm_fault (or pool page empty) doing dump/restore with snapshot
>Confidential: no
>Severity: serious
>Priority: medium
>Responsible: kern-bug-people
>State: open
>Class: sw-bug
>Submitter-Id: net
>Arrival-Date: Sun Sep 21 07:30:03 +0000 2025
>Originator: Henryk Paluch
>Release: 11.0_BETA 20250916040529Z
>Organization:
N/A
>Environment:
NetBSD nbsd11-4zotac.example.com 11.0_BETA NetBSD 11.0_BETA (GENERIC) #0: Tue Sep 16 04:05:29 UTC 2025 mkrepro%mkrepro.NetBSD.org@localhost:/usr/src/sys/arch/amd64/compile/GENERIC amd64
>Description:
Having NetBSD 11.0_BETA guest (Host is Linux KVM, openSUSE LEAP 15.6).
1. NetBSD installed on Virtio-BLK disk ld0 64GB
2. target ld1 is 500GB SATA SSD attached to host via USB JMicron adapter, exposed as Virtio-BLK /dev/sdb block device from host
When I bulk copy lot of data (46GB) using dump/restore with snapshot I will after while get panic:
uvm_fault(0xffffffff81976c20, 0x0, 1) -> e
[ 883.0589643] fatal page fault in supervisor mode
[ 883.0589643] trap type 6 code 0 rip 0xffffffff80e261f3 cs 0x8 rflags 0x10202 cr2 0x9 ilevel 0 rsp 0xffff9d04c49d4c10
[ 883.0589643] curlwp 0xffff8b60ae5fe400 pid 0.1215 lowest kstack 0xffff9d04c49d02c0
kernel: page fault trap, code=0
Stopped in pid 0.1215 (system) at netbsd:pool_get+0x2b9: cmpq %r15,8(%
rax)
pool_get() at netbsd:pool_get+0x2b9
allocbuf() at netbsd:allocbuf+0x113
getblk() at netbsd:getblk+0x18c
bio_doread() at netbsd:bio_doread+0x1d
breadn() at netbsd:breadn+0x8b
ffs_snapshot_read() at netbsd:ffs_snapshot_read+0x1b2
VOP_READ() at netbsd:VOP_READ+0x42
vn_rdwr() at netbsd:vn_rdwr+0xf1
fss_bs_io() at netbsd:fss_bs_io+0x89
fss_bs_thread() at netbsd:fss_bs_thread+0x50f
ds 1
es e400
fs 4bf0
gs 5ff8
rdi ffffffff819835e0 bmempools+0x7e0
rsi 2
rbp ffff9d04c49d4c70
rbx ffffffff81983530 bmempools+0x730
rdx ffffffff
rcx 0
--db_more-- rax 1
r8 0
r9 ffff9d04cbea0030
r10 fffffffe
r11 0
r12 2
r13 ffffffff819835e0 bmempools+0x7e0
r14 ffff9d04cbea0c00
r15 ffff9d04cbea0000
rip ffffffff80e261f3 pool_get+0x2b9
cs 8
rflags 10202
rsp ffff9d04c49d4c10
ss 10
netbsd:pool_get+0x2b9: cmpq %r15,8(%rax)
db{0}> ?
Because there is ffs_snapshot_read() right before panic I suspect that it somehow exhaust all buffers.
Below is output from: vmstat -M netbsd.2.core -N netbsd.2 -m
Memory resource pool statistics
Name Size Requests Fail Releases Pgreq Pgrel Npage Hiwat Minpg Maxpg Idle
amappl 128 1171 0 130 36 1 35 36 0 inf 0
anonpl 40 76445 0 576 755 0 755 755 0 inf 0
biopl 280 54387 0 42890 1176 354 822 971 0 inf 0
buf16k 16896 183710 0 180200 50961 49790 1171 35313 0 inf 0
buf1k 1536 2 0 2 2 1 1 2 0 inf 1
buf2k 2048 372 0 344 187 172 15 173 0 inf 1
buf32k 32768 16466 0 4355 6893 837 6056 6064 0 inf 0
buf4k 4096 280578 0 173240 280579 173240 107339 226999 0 inf 1
buf512b 1024 3 0 3 1 0 1 1 0 inf 1
buf64k 65536 6 0 2 7 2 5 7 0 inf 1
buf8k 8704 1143 0 745 75 18 57 65 0 inf 0
bufpl 280 256490 0 133102 16813 5064 11749 16813 0 inf 0
dirhash 376 2262 0 411 186 0 186 186 0 inf 0
dirhashblk 2048 7684 0 1147 3269 0 3269 3269 0 inf 0
ehcixfer 376 4 0 3 1 0 1 1 0 inf 0
execargs 262144 243 0 243 8 7 1 4 0 16 1
ffsdino2 264 980776 0 155749 55861 0 55861 55861 0 inf 859
ffsino 280 980333 0 155306 59851 0 59851 59851 0 inf 920
file 128 275 0 81 8 0 8 8 0 inf 0
filedesc 832 119 0 86 28 10 18 28 0 inf 0
fstlwp 128 149 0 59 4 0 4 4 0 inf 0
in4pcbpl 256 13 0 7 1 0 1 1 0 inf 0
in6pcbpl 320 11 0 8 1 0 1 1 0 inf 0
inmltpl 56 2 0 0 1 0 1 1 0 inf 0
kcredpl 192 85 0 22 4 0 4 4 0 inf 0
kmem-00016 16 2209 0 301 8 0 8 8 0 inf 0
kmem-00032 32 3652 0 440 26 0 26 26 0 inf 0
kmem-00064 128 9172 0 6193 292 170 122 292 0 inf 0
kmem-00128 192 1907040 0 250751 80096 2 80094 80096 0 inf 1223
kmem-00192 256 362516 0 50703 20293 804 19489 20251 0 inf 0
kmem-00256 320 430 0 89 30 0 30 30 0 inf 0
kmem-00320 384 743 0 76 72 0 72 72 0 inf 0
kmem-00384 448 669 0 106 73 3 70 73 0 inf 0
kmem-00448 512 157 0 56 18 0 18 18 0 inf 0
kmem-00512 576 113 0 58 14 1 13 14 0 inf 0
kmem-00768 832 344 0 66 83 3 80 83 0 inf 0
kmem-01024 1088 723 0 47 238 0 238 238 0 inf 0
kmem-02048 2048 710 0 231 259 19 240 253 0 inf 0
kmem-04096 4096 121 0 30 116 25 91 115 0 inf 0
ksiginfo 136 359 0 260 5 1 4 4 0 inf 0
kva-12288 12288 526 0 7 26 0 26 26 0 inf 0
kva-16384 16384 11 0 8 1 0 1 1 0 inf 0
kva-20480 20480 17 0 10 2 1 1 2 0 inf 0
kva-24576 24576 1 0 1 1 1 0 1 0 inf 0
kva-28672 28672 10 0 0 2 0 2 2 0 inf 0
kva-32768 32768 3 0 2 1 0 1 1 0 inf 0
kva-36864 36864 77 0 0 11 0 11 11 0 inf 0
kva-65536 65536 2 0 0 1 0 1 1 0 inf 0
kva-8192 8192 27 0 7 1 0 1 1 0 inf 0
llentrypl 280 1 0 0 1 0 1 1 0 inf 0
lwppl 1024 182 0 15 45 0 45 45 0 inf 0
mbpl 520 1032 0 579 73 6 67 68 0 inf 2
mclpl 2048 1043 0 592 242 12 230 234 0 261831 4
msdosnopl 208 3 0 2 3 2 1 1 0 inf 0
namecache 192 885618 0 75876 38560 0 38560 38560 0 inf 0
pcache 2688 92 0 4 88 0 88 88 0 inf 0
pcachecpu 128 651 0 0 21 0 21 21 0 inf 0
pcglarge 1088 740 0 629 56 19 37 46 0 inf 0
pcgnormal 320 200584 0 197957 3795 1976 1819 3020 0 inf 1193
pdict16 80 262 0 216 1 0 1 1 0 inf 0
pdict32 96 11 0 1 1 0 1 1 0 inf 0
pdppl 4096 122 0 87 115 80 35 114 0 inf 0
phpool-128 72 26 0 0 1 0 1 1 0 inf 0
phpool-256 88 8 0 0 1 0 1 1 0 inf 0
phpool-64 64 313108 0 175253 4007 577 3430 4007 0 inf 0
piperd 256 19 0 13 2 1 1 2 0 inf 0
pipewr 256 52 0 45 4 1 3 4 0 inf 0
pmappl 512 122 0 87 15 2 13 15 0 inf 0
procpl 896 50 0 16 12 1 11 12 0 inf 0
proparay 56 117 0 0 2 0 2 2 0 inf 0
propdict 56 447 0 107 5 0 5 5 0 inf 0
propnmbr 64 47 0 3 1 0 1 1 0 inf 0
propstng 72 742 0 558 4 0 4 4 0 inf 0
pvpage 4096 831 0 40 819 28 791 813 0 inf 0
rtentpl 320 28 0 2 3 0 3 3 0 inf 0
scxspl 264 30 0 30 1 0 1 1 1 inf 1
sigacts 3088 113 0 79 111 77 34 110 0 inf 0
socket 592 141 0 21 23 1 22 23 0 inf 0
synpl 320 1 0 1 1 1 0 1 0 inf 0
tcpcbpl 832 65 0 62 1 0 1 1 0 inf 0
thplthrd 72 3 0 2 1 0 1 1 0 inf 0
tmpfs_dirent 56 5 0 1 1 0 1 1 0 inf 0
tmpfs_node 232 7 0 1 1 0 1 1 0 inf 0
uarea 24576 49 0 15 47 13 34 46 0 inf 0
uareasys 24576 141 0 8 141 8 133 141 0 inf 0
ufsdir 272 32 0 24 1 0 1 1 0 inf 0
uhcixfer 376 3 0 2 1 0 1 1 0 inf 0
uhcixfer 376 3 0 2 1 0 1 1 0 inf 0
uhcixfer 376 3 0 2 1 0 1 1 0 inf 0
vcachepl 640 923969 0 86108 139653 0 139653 139653 0 inf 9
vmembt 64 16546 0 4964 237 0 237 237 0 inf 0
vmmpepl 192 27152 0 16682 523 24 499 499 0 inf 0
wapbldealloc 40 8 0 8 1 1 0 1 0 inf 0
wapblentrypl 48 84 0 84 2 1 1 1 0 inf 1
wapblinopl 40 176 0 175 1 0 1 1 0 inf 0
Totals 7506830 0 1718506 767003 233357 533646
In use 2459547K, total allocated 2576816K; utilization 95.4%
>How-To-Repeat:
Have disk with lot of data (46GB in my case). Clone one disk to another using dump/restore with snapshot:
newfs -O 2 /dev/dk8
mkdir /mnt/target
mount -o async /dev/dk8 /mnt/target
df -h / /mnt/target
Filesystem Size Used Avail %Cap Mounted on
/dev/dk1 58G 46G 9.5G 82% /
/dev/dk8 197G 4.0K 187G 0% /mnt/target
# commands below will trigger panic after a while (around 20GB copied):
cd /mnt/target
dump -0uaX -f - / | restore rf -
Note: my NetBSD VM has lot of RAM (16GB) and cores (8) - I guess that lowering RAM will trigger panic sooner - need to test it.
Note: the system is otherwise idle while doing dump/restore (so snapshot device should not fill).
>Fix:
N/A
Even using SCSI emulation for target disk will cause similar panic - this time not uvm_fault
but rather "pool_get: [buf16k] pool page empty" - but again FFS snapshot is involved in stacktrace:
[ 2161.615264] panic: kernel diagnostic assertion "(ph->ph_nmissing < pp->pr_itemsperpage)" failed: file "/usr/src/sys/kern/subr_pool.c", l[0/1368]
pool_get: [buf16k] pool page empty
[ 2161.625407] cpu1: Begin traceback...
[ 2161.625407] vpanic() at netbsd:vpanic+0x171
[ 2161.625407] kern_assert() at netbsd:kern_assert+0x4b
[ 2161.625407] pool_get() at netbsd:pool_get+0x7c9
[ 2161.625407] allocbuf() at netbsd:allocbuf+0x113
[ 2161.625407] getblk() at netbsd:getblk+0x18c
[ 2161.625407] bio_doread() at netbsd:bio_doread+0x1d
[ 2161.625407] breadn() at netbsd:breadn+0x8b
[ 2161.625407] ffs_snapshot_read() at netbsd:ffs_snapshot_read+0x1b2
[ 2161.625407] VOP_READ() at netbsd:VOP_READ+0x42
[ 2161.625407] vn_rdwr() at netbsd:vn_rdwr+0xf1
[ 2161.625407] fss_bs_io() at netbsd:fss_bs_io+0x89
[ 2161.625407] fss_bs_thread() at netbsd:fss_bs_thread+0x50f
[ 2161.625407] cpu1: End traceback...
Home |
Main Index |
Thread Index |
Old Index