kern/60795: LFS userland cleaner and umount can enter deadlock

["clare%csel.org@localhost via gnats" <gnats-admin%NetBSD.org@localhost>]

>Number:         60795
>Category:       kern
>Synopsis:       LFS userland cleaner and umount can enter deadlock
>Confidential:   no
>Severity:       serious
>Priority:       medium
>Responsible:    kern-bug-people
>State:          open
>Class:          sw-bug
>Submitter-Id:   net
>Arrival-Date:   Sat Sep 26 06:20:00 +0000 2026
>Originator:     Shinichi Doyashiki
>Release:        11.99.8 around 2026-09-26 14:00 JST
>Organization:
at home
>Environment:
etBSD kanade2.hq.csel.org 11.99.8 NetBSD 11.99.8 (KANADE2_12) #21: Sat Sep 26 14:10:54 JST 2026  clare%kanade2.hq.csel.org@localhost:/export/netbsd/debuglfs/src/sys/arch/amd64/compile/KANADE2_12 amd64

>Description:
LFS deadlock was observed while cycling mount/dd/rm/umount test.

db{0}> ps
PID     LID S CPU     FLAGS       STRUCT LWP *               NAME WAIT
(snip)
19034 19034 3   1         0   ffff9bff037be800       sshd-session mutex
21479 21479 3   1   1000000   ffff9bff0365e000             umount rwlock
22182 22182 3   1         0   ffff9bff0365e400       lfs_cleanerd fstchg
(snip)
0     21074 3   1       240   ffff9bff01c77000         lfs_writer mutex
(snip)
0    >  102 1   3       201   ffff9bfef726bc00             idle/3
(snip)
0    >   96 1   2       201   ffff9bfef717b400             idle/2
(snip)
0    >   25 1   1       201   ffff9c062fa54c00             idle/1
(snip)
0        12 3   0       200   ffff9c0630ab1800             lfsswq mutex
0        11 3   0       200   ffff9c0630ab1400             lfscwq lfscwq
(snip)
0    >    2 1   0       201   ffff9c0633b1d000             idle/0
0         0 3   0       200   ffffffff814648c0            swapper uvm

the umount process

db{0}> bt /a ffff9bff0365e000
trace: pid 21479 lid 21479 at 0xffffcc094754cc00
sleepq_block() at netbsd:sleepq_block+0xea
turnstile_block() at netbsd:turnstile_block+0x24b
rw_vector_enter() at netbsd:rw_vector_enter+0x161
lfs_prelock() at netbsd:lfs_prelock+0x86
lfs_seglock() at netbsd:lfs_seglock+0x16
lfs_segwrite() at netbsd:lfs_segwrite+0x71
lfs_flushfiles() at netbsd:lfs_flushfiles+0x1a4
lfs_unmount() at netbsd:lfs_unmount+0x27
VFS_UNMOUNT() at netbsd:VFS_UNMOUNT+0x22
dounmount() at netbsd:dounmount+0xa9
sys_unmount() at netbsd:sys_unmount+0x101
syscall() at netbsd:syscall+0x9d
--- syscall (number 22) ---
netbsd:syscall+0x9d:

the lfs_cleanerd (userland) process

db{0}> bt /a ffff9bff0365e400
trace: pid 22182 lid 22182 at 0xffffcc09486d4950
sleepq_block() at netbsd:sleepq_block+0xea
cv_wait() at netbsd:cv_wait+0xca
fstrans_start() at netbsd:fstrans_start+0x14c
VOP_BMAP() at netbsd:VOP_BMAP+0x48
lfs_strategy() at netbsd:lfs_strategy+0x3b3
VOP_STRATEGY() at netbsd:VOP_STRATEGY+0x64
bio_doread() at netbsd:bio_doread+0x5f
bread() at netbsd:bread+0x18
ino_func_setclean() at netbsd:ino_func_setclean+0x2f7
lfs_parse_pseg() at netbsd:lfs_parse_pseg+0x87f
lfs_rewrite_segments() at netbsd:lfs_rewrite_segments+0x3f6
lfs_fcntl() at netbsd:lfs_fcntl+0x15b3
VOP_FCNTL() at netbsd:VOP_FCNTL+0x6e
sys_fcntl() at netbsd:sys_fcntl+0x461
syscall() at netbsd:syscall+0x9d
--- syscall (number 92) ---
netbsd:syscall+0x9d:

the lfs_writer thread

db{0}> bt /a ffff9bff01c77000
trace: pid 0 lid 21074 at 0xffffcc094a0d4e60
sleepq_block() at netbsd:sleepq_block+0xea
turnstile_block() at netbsd:turnstile_block+0x24b
mutex_vector_enter() at netbsd:mutex_vector_enter+0x13f
cv_timedwait() at netbsd:cv_timedwait+0xde
lfs_writerd() at netbsd:lfs_writerd+0x3ad

db{0}> show locks
PID     LID CPU       STRUCT LWP *              OWNER BITS LOCK
19034 19034   1   ffff9bff037be800   ffff9bff0365e000    2 netbsd:vfs_suspend_lock
21479 21479   1   ffff9bff0365e000                 20    3 ffff9bff04a112b8
0     21074   1   ffff9bff01c77000   ffff9bff0365e000    2 netbsd:lfs_lock
0        12   0   ffff9c0630ab1800   ffff9bff0365e000    2 netbsd:lfs_lock

db{0}> call fstrans_dump(0)
Fstrans locks by lwp:
[ 1371.1778959] Fstrans state by mount:
[ 1371.1778959] /testlfs         owner 0xffff9bff0365e000 state suspended
0

>How-To-Repeat:
#!/bin/sh
#
# LFS umount while cleaner test 2
#
DISK=/dev/dk4
RDISK=/dev/rdk4

newfs_lfs -f4k -b32k $RDISK
mkdir -p /testlfs
count=0
while true; do
  count=`expr $count + 1`
  echo "===> MOUNT $count"
  if mount_lfs -u $DISK /testlfs; then
    echo "===> GOOD mount"
  else
    echo "===> BAD mount"
    exit 1
  fi
  echo "===> try file access"
  {
    cd /testlfs
    dd if=/dev/zero of=testfile1.bin bs=1m count=1
    #dd if=/dev/zero of=testfile2.bin bs=1m count=1
    #dd if=/dev/zero of=testfile3.bin bs=1m count=1
    #dd if=/dev/zero of=testfile4.bin bs=1m count=1
    #dd if=/dev/zero of=testfile5.bin bs=1m count=1
    #dd if=/dev/zero of=testfile6.bin bs=1m count=1
    #dd if=/dev/zero of=testfile7.bin bs=1m count=1
    #dd if=/dev/zero of=testfile8.bin bs=1m count=1
    rm testfile*.bin
    cd ..
  }
  echo "===> try umount"
  if umount /testlfs; then
    echo "===> GOOD umount"
  else
    echo "===> BAD umount"
    exit 1
  fi
  if fsck_lfs -nf $RDISK; then
    echo "===> GOOD fsck_lfs"
  else
    echo "===> BAD fsck_lfs"
    exit 1
  fi
done

>Fix:
unknown