Re: Test-bed time-out

[Chuck Silvers <chuq%chuq.com@localhost>]

On Thu, May 10, 2012 at 10:42:15PM +0300, Andreas Gustafsson wrote:
> Chuck,
> 
> Here's a kernel stack trace from a system in the locked-up state:
> 
>   #0  VOP_FSYNC (vp=0xc1a096e0, cred=0xc170f9c0, flags=17, 
>       offlo=-4560837987435091108, offhi=-4488417570814164992)
>       at 
> /tmp/bracket/build/2012.05.07.21.09.29-i386-debug/src/sys/kern/vnode_if.c:648
>   #1  0xc03a2b92 in ffs_sync (mp=0xc1b5f000, waitfor=1, cred=0xc170f9c0)
>       at 
> /tmp/bracket/build/2012.05.07.21.09.29-i386-debug/src/sys/ufs/ffs/ffs_vfsops.c:1735
>   #2  0xc0b3dc56 in VFS_SYNC (mp=0xc1b5f000, a=1, b=0xc170f9c0)
>       at 
> /tmp/bracket/build/2012.05.07.21.09.29-i386-debug/src/sys/kern/vfs_subr.c:1057
>   #3  0xc0b3a950 in dounmount (mp=0xc1b5f000, flags=0, l=0xc1ef3a80)
>       at 
> /tmp/bracket/build/2012.05.07.21.09.29-i386-debug/src/sys/kern/vfs_mount.c:823
>   #4  0xc0b3f7c7 in sys_unmount (l=0xc1ef3a80, uap=0xc8d2ecec, 
> retval=0xc8d2ed14)
>       at 
> /tmp/bracket/build/2012.05.07.21.09.29-i386-debug/src/sys/kern/vfs_syscalls.c:562
>   #5  0xc0a0a4cb in sy_call (sy=0xc0f70028, l=0xc1ef3a80, uap=0xc8d2ecec, 
>       rval=0xc8d2ed14)
>       at 
> /tmp/bracket/build/2012.05.07.21.09.29-i386-debug/src/sys/sys/syscallvar.h:61
>   #6  0xc0a0a7fe in syscall (frame=0xc8d2ed48)
>       at 
> /tmp/bracket/build/2012.05.07.21.09.29-i386-debug/src/sys/arch/x86/x86/syscall.c:169
>   #7  0xc01005d6 in Xsyscall ()

thanks for the info.  I tried to reproduce this myself but didn't have any luck.
I ran that one test program in a loop for 1/2 hour (several thousand cycles)
but it never got stuck.  that was on a real system, not qemu.  also, amd64.

that stack looks like the unmount thread wasn't sleeping, can you confirm
that the thread is continuing to run even though the unmount doesn't complete?
if that's the case, could you try to narrow down what it's doing?
I usually do this by setting ddb breakpoints on various functions that are
called from whatever function I've followed it down to so far,
that usually doesn't take too long to find something interesting.

or since it's i386, you could use dtrace too.

it's not obvious how the change you identified could have caused this,
all that does is change the I/O priority of background file data write I/Os.

-Chuck