lib/42093: rump rename race testcase crashes

To: lib-bug-people%netbsd.org@localhost,gnats-admin%netbsd.org@localhost,netbsd-bugs%netbsd.org@localhost
Subject: lib/42093: rump rename race testcase crashes
From: njoly%pasteur.fr@localhost
Date: Sat, 19 Sep 2009 09:20:01 +0000 (UTC)

>Number:         42093
>Category:       lib
>Synopsis:       rump rename race testcase crashes
>Confidential:   no
>Severity:       serious
>Priority:       medium
>Responsible:    lib-bug-people
>State:          open
>Class:          sw-bug
>Submitter-Id:   net
>Arrival-Date:   Sat Sep 19 09:20:00 +0000 2009
>Originator:     Nicolas Joly
>Release:        NetBSD 5.99.18
>Organization:
Insitut Pasteur
>Environment:
System: NetBSD lanfeust.sis.pasteur.fr 5.99.18 NetBSD 5.99.18 (LANFEUST) #0: 
Sat Sep 19 01:00:25 CEST 2009 
njoly%lanfeust.sis.pasteur.fr@localhost:/local/src/NetBSD/obj.amd64/sys/arch/amd64/compile/LANFEUST
 amd64
Architecture: x86_64
Machine: amd64
>Description:
Noticed that ffs/t_renamerace ATF testcase sometimes crash. Trying to debug it,
i made the following rump small code which do crash too.

The crashes do only appear when logging is enabled (MNT_LOG), and are not so 
uncommom.
On 10 runs; i get 6 success, 3 crashes with `Resource deadlock avoided' and 1
with `segmentation fault'.

njoly@lanfeust [rump/ffs]> make ffs_rename
cc -g -O0 -Wall -Werror   -lrumpfs_ffs -lrumpvfs -o ffs_rename ffs_rename.c 

I currently do see 2 different symptoms when running it:

1) 
njoly@lanfeust [rump/ffs]> ./ffs_rename 
panic: rumpuser fatal failure 11 (Resource deadlock avoided)
zsh: abort (core dumped)  ./ffs_rename
njoly@lanfeust [rump/ffs]> gdb ffs_rename ffs_rename.core 
GNU gdb 6.5
[...]
Core was generated by `ffs_rename'.
Program terminated with signal 6, Aborted.
#0  0x00007f7ffd7e0c0a in _lwp_kill () from /usr/lib/libc.so.12
(gdb) bt
#0  0x00007f7ffd7e0c0a in _lwp_kill () from /usr/lib/libc.so.12
#1  0x00007f7ffd7e057e in abort () at 
/local/src/NetBSD/src/lib/libc/stdlib/abort.c:74
#2  0x00007f7ffd103662 in rumpuser_rw_enter (rw=0x7f7ffb2011a0, iswrite=<value 
optimized out>)
    at 
/local/src/NetBSD/src/lib/librumpuser/../../sys/rump/librump/rumpuser/rumpuser_pth.c:353
#3  0x00007f7ffda2ab6a in vlockmgr (vl=0x7f7ffb20f2a8, flags=0)
    at 
/local/src/NetBSD/src/lib/librumpvfs/../../sys/rump/../kern/vfs_subr.c:2832
#4  0x00007f7ffd538d29 in VOP_LOCK (vp=0x7f7ffb20f1a0, flags=6)
    at /local/src/NetBSD/src/lib/librump/../../sys/rump/../kern/vnode_if.c:1380
#5  0x00007f7ffda24c38 in vn_lock (vp=0x7f7ffb20f1a0, flags=2)
    at 
/local/src/NetBSD/src/lib/librumpvfs/../../sys/rump/../kern/vfs_vnops.c:765
#6  0x00007f7ffda2ec4c in vget (vp=0x7f7ffb20f1a0, flags=65538)
    at 
/local/src/NetBSD/src/lib/librumpvfs/../../sys/rump/../kern/vfs_subr.c:1310
#7  0x00007f7ffdc26bee in ufs_ihashget (dev=65024, inum=5, flags=2)
    at 
/local/src/NetBSD/src/sys/rump/fs/lib/libffs/../../../../ufs/ufs/ufs_ihash.c:153
#8  0x00007f7ffdc0eeaf in ffs_vget (mp=0x7f7ffc6b3000, ino=5, 
vpp=0x7f7ffb7ff8b0)
    at 
/local/src/NetBSD/src/sys/rump/fs/lib/libffs/../../../../ufs/ffs/ffs_vfsops.c:1717
#9  0x00007f7ffdc0c07a in ufs_lookup (v=<value optimized out>)
    at 
/local/src/NetBSD/src/sys/rump/fs/lib/libffs/../../../../ufs/ufs/ufs_lookup.c:541
#10 0x00007f7ffd539c71 in VOP_LOOKUP (dvp=0x7f7ffc6ad400, vpp=0x6, cnp=<value 
optimized out>)
    at /local/src/NetBSD/src/lib/librump/../../sys/rump/../kern/vnode_if.c:130
#11 0x00007f7ffdc17599 in wapbl_ufs_rename (v=<value optimized out>)
    at 
/local/src/NetBSD/src/sys/rump/fs/lib/libffs/../../../../ufs/ufs/ufs_wapbl.c:358
#12 0x00007f7ffdc14403 in ufs_rename (v=<value optimized out>)
    at 
/local/src/NetBSD/src/sys/rump/fs/lib/libffs/../../../../ufs/ufs/ufs_vnops.c:976
#13 0x00007f7ffd539074 in VOP_RENAME (fdvp=0x7f7ffc6ad400, fvp=0x6, fcnp=<value 
optimized out>, 
    tdvp=0x7f7ffd7e0c0a, tvp=0x1, tcnp=0x8080808080808080)
    at /local/src/NetBSD/src/lib/librump/../../sys/rump/../kern/vnode_if.c:998
#14 0x00007f7ffda25ef6 in do_sys_rename (from=<value optimized out>, 
to=0x4011ba "/mnt/rename2.test", 
    seg=UIO_USERSPACE, retain=0)
    at 
/local/src/NetBSD/src/lib/librumpvfs/../../sys/rump/../kern/vfs_syscalls.c:3395
#15 0x00007f7ffd552a94 in rump_sys_rename (from=<value optimized out>, 
to=<value optimized out>)
    at 
/local/src/NetBSD/src/lib/librump/../../sys/rump/librump/rumpkern/rump_syscalls.c:878
#16 0x0000000000400e86 in func2 (arg=0x0) at ffs_rename.c:37
#17 0x00007f7ffd30a812 in pthread__create_tramp (cookie=<value optimized out>)
    at /local/src/NetBSD/src/lib/libpthread/pthread.c:476
#18 0x00007f7ffd76e690 in ___lwp_park50 () from /usr/lib/libc.so.12
#19 0x00007f7ffb800000 in ?? ()
#20 0x0000000111110001 in ?? ()
#21 0x0000000033330003 in ?? ()
#22 0x0000000000000000 in ?? ()

2)
njoly@lanfeust [rump/ffs]> ./ffs_rename
zsh: segmentation fault (core dumped)  ./ffs_rename
njoly@lanfeust [rump/ffs]> gdb ffs_rename ffs_rename.core
GNU gdb 6.5
[...]
Core was generated by `ffs_rename'.
Program terminated with signal 11, Segmentation fault.
#0  0x00007f7ffdc0b198 in ufs_makedirentry (ip=0x7f7ffb416100, 
    cnp=0x7f7ffb7ffb70, newdirp=0x7f7ffb51a040)
    at 
/local/src/NetBSD/src/sys/rump/fs/lib/libffs/../../../../ufs/ufs/ufs_lookup.c:735
735             if (FSFMT(ITOV(ip)))
(gdb) bt
#0  0x00007f7ffdc0b198 in ufs_makedirentry (ip=0x7f7ffb416100, 
    cnp=0x7f7ffb7ffb70, newdirp=0x7f7ffb51a040)
    at 
/local/src/NetBSD/src/sys/rump/fs/lib/libffs/../../../../ufs/ufs/ufs_lookup.c:735
#1  0x00007f7ffdc17f3b in wapbl_ufs_rename (v=<value optimized out>)
    at 
/local/src/NetBSD/src/sys/rump/fs/lib/libffs/../../../../ufs/ufs/ufs_wapbl.c:443
#2  0x00007f7ffdc14403 in ufs_rename (v=<value optimized out>)
    at 
/local/src/NetBSD/src/sys/rump/fs/lib/libffs/../../../../ufs/ufs/ufs_vnops.c:976
#3  0x00007f7ffd539074 in VOP_RENAME (fdvp=0x7f7ffc6ad400, fvp=0x7f7ffb501811, 
    fcnp=<value optimized out>, tdvp=0x0, tvp=0x4, tcnp=0xe2)
    at /local/src/NetBSD/src/lib/librump/../../sys/rump/../kern/vnode_if.c:998
#4  0x00007f7ffda25ef6 in do_sys_rename (from=<value optimized out>, 
    to=0x4011ba "/mnt/rename2.test", seg=UIO_USERSPACE, retain=0)
    at 
/local/src/NetBSD/src/lib/librumpvfs/../../sys/rump/../kern/vfs_syscalls.c:3395
#5  0x00007f7ffd552a94 in rump_sys_rename (from=<value optimized out>, 
    to=<value optimized out>)
    at 
/local/src/NetBSD/src/lib/librump/../../sys/rump/librump/rumpkern/rump_syscalls.c:878
#6  0x0000000000400e86 in func2 (arg=0x0) at ffs_rename.c:37
#7  0x00007f7ffd30a812 in pthread__create_tramp (cookie=<value optimized out>)
    at /local/src/NetBSD/src/lib/libpthread/pthread.c:476
#8  0x00007f7ffd76e690 in ___lwp_park50 () from /usr/lib/libc.so.12
#9  0x00007f7ffb800000 in ?? ()
#10 0x0000000111110001 in ?? ()
#11 0x0000000033330003 in ?? ()
#12 0x0000000000000000 in ()
(gdb) p ip
$1 = (struct inode *) 0x7f7ffb416100
(gdb) p ip->i_vnode
$2 = (struct vnode *) 0x7f7ffb40f070
(gdb) p ip->i_vnode->v_mount
$3 = (struct mount *) 0x0
(gdb) p ip->i_vnode->v_mount->mnt_iflag
Cannot access memory at address 0x64


njoly@lanfeust [rump/ffs]> cat ffs_rename.c

#include <rump/rump_syscalls.h>
#include <rump/rump.h>

#include <ufs/ufs/ufsmount.h>

#include <err.h>
#include <fcntl.h>
#include <pthread.h>
#include <stdlib.h>
#include <string.h>
#include <unistd.h>

#define IMAGE "image.ffs"
#define DEVICE "/dev/device.ffs"
#define MNTDIR "/mnt"
#define FILE1 MNTDIR "/rename1.test"
#define FILE2 MNTDIR "/rename2.test"


static int quit = 0;


static void *func1(void *arg) {
  int fd;

  while (quit != 1) {
    fd = rump_sys_open(FILE1, O_WRONLY|O_CREAT|O_TRUNC, 0666);
    rump_sys_unlink(FILE1);
    rump_sys_close(fd); }

  return NULL; }

static void *func2(void *arg) {

  while (quit != 1) {
    rump_sys_rename(FILE1, FILE2); }

  return NULL; }


int main() {
  int res;
  pthread_t th1, th2;
  struct ufs_args args;

  (void)unlink(IMAGE);
  res = system("newfs -F -s 10000 " IMAGE " >/dev/null");
  if (res == -1)
    err(1, "system failed");

  res = rump_init();
  if (res != 0)
    err(1, "rump_init failed");

  res = rump_etfs_register(DEVICE, IMAGE, RUMP_ETFS_BLK);
  if (res != 0)
    err(1, "rump_etfs_register failed");

  res = rump_sys_mkdir(MNTDIR, 0777);
  if (res != 0)
    err(1, "rump_sys_mkdir failed");

  memset(&args, 0x0, sizeof(args));
  args.fspec = DEVICE;

  res = rump_sys_mount(MOUNT_FFS, MNTDIR, MNT_LOG, &args, sizeof(args));
  if (res == -1)
    err(1, "rump_sys_mount failed");

  res = pthread_create(&th1, NULL, func1, NULL);
  if (res != 0)
    errx(1, "pthread_create failed");
  res = pthread_create(&th2, NULL, func2, NULL);
  if (res != 0)
    errx(1, "pthread_create failed");

  sleep(10);
  quit = 1;

  res = pthread_join(th2, NULL);
  if (res != 0)
    errx(1, "pthread_join failed");
  res = pthread_join(th1, NULL);
  if (res != 0)
    errx(1, "pthread_join failed");

  res = rump_sys_unmount(MNTDIR, 0);
  if (res == -1)
    err(1, "rump_sys_unmount failed");

  res = rump_etfs_remove(DEVICE);
  if (res == -1)
    err(1, "rump_etfs_remove failed");

  res = unlink(IMAGE);
  if (res == -1)
    err(1, "unlink failed");

  return 0; }

>How-To-Repeat:
Compile the provided testcase, run it multiple times and see it fail.
>Fix:

Follow-Ups:
- Re: kern/42093: rump rename race testcase crashes
  - From: Nicolas Joly
- Re: lib/42093: rump rename race testcase crashes
  - From: Manuel Bouyer
- Re: lib/42093: rump rename race testcase crashes
  - From: Nicolas Joly

Prev by Date: port-amd64/42091: ld devices can't be set up with more than 2^31 sectors
Next by Date: PR/38188 CVS commit: src/sys/fs/puffs
Previous by Thread: port-amd64/42091: ld devices can't be set up with more than 2^31 sectors
Next by Thread: Re: lib/42093: rump rename race testcase crashes
Indexes:

Home | Main Index | Thread Index | Old Index