Re: hard hangs sometime after 22nd April and mutex problems

[Geoff Wing <gcw%pobox.com@localhost>]

On Sunday 2008-04-27 15:28 +0100, Andrew Doran output:
:On Sun, Apr 27, 2008 at 11:21:32AM +1000, Geoff Wing wrote:
:> On i386 SMP I've been getting hard hangs - no response, pings, can't break
:> into ddb - since sometime after the 22nd April.
:Can you try running with LOCKDEBUG+DIAGNOSTIC please?

Righteo,
it now seems to always die before the rc scripts are finished though at
different times.  One trace had more problems after hitting panic/cpu_reboot
and went through a sequence which panic'd/cpu_reboot'd again (#0/1 & #15/16)
but another didn't.

Regards,
Geoff

[160 column output follows]

#0  0xc036c553 in cpu_reboot (howto=260, bootstr=0x0) at 
../../../../arch/i386/i386/machdep.c:881
#1  0xc02dfaf2 in panic (fmt=0xc05073cb "LOCKDEBUG") at 
../../../../kern/subr_prf.c:260
#2  0xc02d8cea in lockdebug_abort1 (ld=0xcdd95e80, lk=0xc05df740, 
func=0xc04639a7 "mutex_vector_enter",
    msg=0xc05033c8 "assertion failed: MUTEX_OWNER(mtx->mtx_owner) == 
curthread", dopanic=true) at ../../../../kern/subr_lockdebug.c:802
#3  0xc02d9773 in lockdebug_abort (lock=0xcdde37cc, ops=0xc05594d0, 
func=0xc04639a7 "mutex_vector_enter",
    msg=0xc05033c8 "assertion failed: MUTEX_OWNER(mtx->mtx_owner) == 
curthread") at ../../../../kern/subr_lockdebug.c:846
#4  0xc02b4be3 in mutex_abort (mtx=0x0, func=0xc04639a7 "mutex_vector_enter", 
msg=0xc05033c8 "assertion failed: MUTEX_OWNER(mtx->mtx_owner) == curthread")
    at ../../../../kern/kern_mutex.c:325
#5  0xc02b54a7 in mutex_vector_enter (mtx=0xcdde37cc) at 
../../../../kern/kern_mutex.c:712
#6  0xc0326c3b in genfs_do_putpages (vp=0xcdde37cc, startoff=0, 
endoff=9223372036854771712, origflags=<value optimized out>, busypg=0x0)
    at ../../../../miscfs/genfs/genfs_io.c:935
#7  0xc03271e8 in genfs_putpages (v=0xce44d84c) at 
../../../../miscfs/genfs/genfs_io.c:744
#8  0xc03231b8 in VOP_PUTPAGES (vp=0xcdde37cc, offlo=0, offhi=0, flags=17) at 
../../../../kern/vnode_if.c:1763
#9  0xc0230f57 in ffs_full_fsync (vp=0xcdde37cc, flags=0) at 
../../../../ufs/ffs/ffs_vnops.c:350
#10 0xc0231071 in ffs_fsync (v=0xce44d978) at 
../../../../ufs/ffs/ffs_vnops.c:258
#11 0xc0323bad in VOP_FSYNC (vp=0xcdde37cc, cred=0xcc065f00, flags=0, offlo=0, 
offhi=0) at ../../../../kern/vnode_if.c:804
#12 0xc022e52e in ffs_sync (mp=0xcdd33000, waitfor=2, cred=0xcc065f00) at 
../../../../ufs/ffs/ffs_vfsops.c:1397
#13 0xc031992e in VFS_SYNC (mp=0xcdd33000, a=2, b=0xcc065f00) at 
../../../../kern/vfs_subr2.c:1102
#14 0xc031ef78 in sys_sync (l=0xce4016e0, v=0x0, retval=0x0) at 
../../../../kern/vfs_syscalls.c:783
#15 0xc0317e97 in vfs_shutdown () at ../../../../kern/vfs_subr.c:1810
#16 0xc036c5d8 in cpu_reboot (howto=256, bootstr=0x0) at 
../../../../arch/i386/i386/machdep.c:867
#17 0xc02dfaf2 in panic (fmt=0xc054bbec "kernel %sassertion \"%s\" failed: file 
\"%s\", line %d") at ../../../../kern/subr_prf.c:260
#18 0xc044471e in __kernassert (t=0xc04e3258 "diagnostic ", f=0xc050afa4 
"../../../../kern/uipc_socket.c", l=599, e=0xc04e7526 "solocked(so)")
    at ../../../../../../lib/libkern/__assert.c:50
#19 0xc0305b8b in sofree (so=0xc2c0b5d4) at ../../../../kern/uipc_socket.c:599
#20 0xc0305c6f in soabort (so=0xc2c0b5d4) at ../../../../kern/uipc_socket.c:721
#21 0xc0305d17 in soclose (so=0xc2c0b94c) at ../../../../kern/uipc_socket.c:661
#22 0xc02eff5c in soo_close (fp=0xce3425c0) at ../../../../kern/sys_socket.c:255
#23 0xc02a3686 in closef (fp=0xce3425c0) at ../../../../kern/kern_descrip.c:757
#24 0xc02a382f in fd_close (fd=3) at ../../../../kern/kern_descrip.c:648
#25 0xc02a3a1e in fd_free () at ../../../../kern/kern_descrip.c:1491
#26 0xc02a9fe1 in exit1 (l=0xce4016e0, rv=256) at 
../../../../kern/kern_exit.c:279
#27 0xc02aa7d8 in sys_exit (l=0xce4016e0, uap=0xce44dd00, retval=0xce44dd28) at 
../../../../kern/kern_exit.c:183
#28 0xc036fcb2 in syscall (frame=0xce44dd48) at 
../../../../arch/i386/i386/syscall.c:109
#29 0xc01004fd in syscall1 ()