kern/38642: hang during unmount of mfs:/tmp during reboot

["Greg A. Woods" <woods%planix.com@localhost>]

>Number:         38642
>Category:       kern
>Synopsis:       hang during unmount of mfs:/tmp during reboot
>Confidential:   no
>Severity:       critical
>Priority:       medium
>Responsible:    kern-bug-people
>State:          open
>Class:          sw-bug
>Submitter-Id:   net
>Arrival-Date:   Mon May 12 18:50:00 +0000 2008
>Originator:     Greg A. Woods
>Release:        NetBSD 4.99.62 2008/05/12
>Organization:
Planix, Inc.; Toronto, Ontario; Canada
>Environment:
System: NetBSD 4.99.62 (GENERIC)
Architecture: i386
Machine: i386
>Description:

        The system hangs consistently during rebot at the point where it
        goes to unmount the MFS /tmp.  Sometimes it gives a lock error
        during the backtrace in DDB, like the following, and sometimes
        only after trying to reboot from DDB, as in the second example.

        This all worked fine up until a couple of days ago (may 5? 7?).

historically# reboot -q
syncing disks... 1 done
unmounting file systems...
unmounting /kern (kernfs)...ok,
unmounting /tmp (mfs:156)...[halt sent]
Stopped in pid 0.15 (system) at netbsd:breakpoint+0x4:  popl    %ebp
db{1}> bt
breakpoint(0,3f8,5,2,c0ba0320,c301950c,d132df6c,c30af21e,c30b0000,800) at 
netbsd:breakpoint+0x4
comintr(c3019400,d1321c50,deadbeef,1,46353148,9,0,0,0,0) at netbsd:comintr+0x575
DDB lost frame for netbsd:Xintr_ioapic_edge1+0xa9, trying 0xd132df74
Xintr_ioapic_edge1() at netbsd:Xintr_ioapic_edge1+0xa9
--- interrupt ---
--- switch to interrupt stack ---
x86_pause(1,64,c01aaa60,0,104,c0ba0320,d1321d40,c04c2d63,0,cc6b09c0) at 
netbsd:x86_pause+0x2
nd6_timer(0,cc6b09c0,7,d1325000,d1317074,d1325068,d1325870,d1326070,d1326870,c01aaa60)
 at netbsd:nd6_timer+0x4e
callout_softclock(0,0,0,0,0,0,0,0,0,0) at netbsd:callout_softclock+0x243
softint_dispatch(cc6a9280,2,0,0,0,0,d1321d90,d1321d28,cc6b09c0,28) at 
netbsd:softint_dispatch+0xb7
DDB lost frame for netbsd:Xsoftintr+0x3d, trying 0xd1321d88
Xsoftintr() at netbsd:Xsoftintr+0x3d
--- interrupt ---
Reader / writer lock error: rw_vector_enter: assertion failed: !cpu_intr_p()

lock address : 0x00000000c0bb0944 type     :     sleep/adaptive
shared holds :                  0 exclusive:                  0
shares wanted:                  0 exclusive:                  0
current cpu  :                  1 last held:                  1
current lwp  : 0x00000000cc6b09c0 last held: 000000000000000000
last locked  : 0x00000000c044a449 unlocked : 0x00000000c044d97b
initialized  : 0x00000000c04497ca
owner/count  : 000000000000000000 flags    : 0x0000000000000008

Turnstile chain at 0xc0bb2240.
=> No active turnstile for this lock.

panic: LOCKDEBUG
Stopped in pid 0.15 (system) at netbsd:breakpoint+0x4:  popl    %ebp
db{1}> x/I 0x00000000c04497ca
netbsd:uvm_map_setup+0x7a:      movl    $0xc0a5ee3e,%eax
db{1}> x/I 0x00000000c044a449
netbsd:vm_map_lock+0x69:        movl    0x8(%ebx),%edx
db{1}> x/I 0x00000000c044d97b
netbsd:uvmspace_fork+0x2cb:     movl    0x8(%ebp),%edx
db{1}> 



historically# reboot -q
syncing disks... 4 done
unmounting file systems...
unmounting /kern (kernfs)...ok,
unmounting /tmp (mfs:156)...[halt sent]
Stopped in pid 1174.1 (reboot) at       netbsd:breakpoint+0x4:  popl    %ebp
db{1}> trace
breakpoint(0,3f8,5,0,d1324000,c301850c,d1324068,c30ae142,c30af000,800) at 
netbsd:breakpoint+0x4
comintr(c3018400,d213fa0c,deadbeef,1,46353148,9,0,0,0,0) at netbsd:comintr+0x575
DDB lost frame for netbsd:Xintr_ioapic_edge1+0xa9, trying 0xd132cf74
Xintr_ioapic_edge1() at netbsd:Xintr_ioapic_edge1+0xa9
--- interrupt ---
--- switch to interrupt stack ---
Xspllower(0,c0520fbc,0,0,0,d1dacc48,4,c04a6552,0,c040dde8) at 
netbsd:Xspllower+0xf
mutex_vector_enter(d20b8ce0,c040ded2,0,d1dacc48,0,d20b8c3c,d213fb6c,c040df0a,d20b8ce0,20002)
 at netbsd:mutex_vector_enter+0x202
vn_lock(d20b8ce0,20002,1,0,0,0,0,d20b8c3c,d1edd000,d20b8c3c) at 
netbsd:vn_lock+0x9c
ffs_sync(d1edd000,1,cc6a5d80,0,0,d1edd000,d213fbbc,c051ef6d,d1edd000,1) at 
netbsd:ffs_sync+0x1da
VFS_SYNC(d1edd000,1,cc6a5d80,101a,80000,d20b8b98,1,d1edd000,d1edd0f8,d1cdf000) 
at netbsd:VFS_SYNC+0x2c
dounmount(d1edd000,80000,d2261380,0,d213fbf8,d2261380,0,d2261380,d213fd00,c0af0180)
 at netbsd:dounmount+0xfd
vfs_unmountall(d2261380,0,0,c048d4dd,cc69a238,0,d213fc2c,c05945bc,0,d2261380) 
at netbsd:vfs_unmountall+0xa0
vfs_shutdown(0,d2261380,0,0,d213fd00,0,d213fcdc,c04c5284,0,0) at 
netbsd:vfs_shutdown+0x8d
cpu_reboot(0,0,0,0,0,0,ffffffff,0,d226a400,8000000) at netbsd:cpu_reboot+0x19c
sys_reboot(d2261380,d213fd00,d213fd28,804d78c,804d000,d2272dd0,1,0,0,806d735) 
at netbsd:sys_reboot+0x74
syscall(d213fd48,b3,ab,1f,1f,806d72c,bfbffb49,bfbfea28,8073658,1) at 
netbsd:syscall+0xab
db{1}> reboot
Mutex error: lockdebug_wantlock: acquiring sleep lock from interrupt context

lock address : 0x00000000c0bb158c type     :     sleep/adaptive
shared holds :                  0 exclusive:                  0
shares wanted:                  0 exclusive:                  0
current cpu  :                  1 last held:                  0
current lwp  : 0x00000000d2261380 last held: 000000000000000000
last locked  : 0x00000000c04c6a1c unlocked : 0x00000000c04c6a8e
initialized  : 0x00000000c04c7a85
owner field  : 000000000000000000 wait/spin:                0/0

Turnstile chain at 0xc0bb13a0.
=> No active turnstile for this lock.

panic: LOCKDEBUG
Stopped in pid 1174.1 (reboot) at       netbsd:breakpoint+0x4:  popl    %ebp
db{1}> x/I 0x00000000c04c7a85
netbsd:config_init+0x35:        movl    $0xc0a6a470,%eax
db{1}> x/I 0x00000000c04c6a1c
netbsd:config_devlink+0xec:     movl    netbsd:alldevs_nwrite,%eax
db{1}> x/I 0x00000000c04c6a8e
netbsd:config_devlink+0x15e:    addl    $0x1c,%esp
db{1}> reboot
rebooting...


>How-To-Repeat:

        use MFS /tmp and reboot with "reboot -q" (to avoid sending
        SIGTERM to mount_mfs)

        swap    /tmp    mfs     rw,-s=512m,nodev,nosuid


>Fix:

        unknown