kern/56546: cgd tests fail randomly

[gson%gson.org@localhost (Andreas Gustafsson)]

>Number:         56546
>Category:       kern
>Synopsis:       cgd tests fail randomly
>Confidential:   no
>Severity:       serious
>Priority:       medium
>Responsible:    kern-bug-people
>State:          open
>Class:          sw-bug
>Submitter-Id:   net
>Arrival-Date:   Sun Dec 12 15:25:00 +0000 2021
>Originator:     Andreas Gustafsson
>Release:        NetBSD-current
>Organization:
  
>Environment:
System: NetBSD
Architecture: i386
Machine: i386
>Description:

On some testbeds, various cgd test cases are randomly failing with the
error message "panic: rumpuser fatal failure 22 (Invalid argument)".
For example:

  http://releng.netbsd.org/b5reports/sparc64/2021/2021.12.10.01.18.29/test.html#dev_cgd_t_cgd_blowfish_cgd_bf_cbc_448_encblkno1
  https://www.gson.org/netbsd/bugs/build/i386-laptop/2021/2021.12.11.11.13.30/test.html#dev_cgd_t_cgd_aes_cgd_aes_cbc_128_encblkno8

Based on the "0xdead" in ptm_magic, I'm guessing it's trying to lock a freed mutex:

  (gdb) bt
  #0  0xb996f847 in _lwp_kill () from /usr/lib/libc.so.12
  #1  0xb996f7d6 in raise (s=6)
      at /tmp/build/2021.12.12.11.18.46-i386-debug-laptop/src/lib/libc/gen/raise.c:48
  #2  0xb996fe06 in abort ()
      at /tmp/build/2021.12.12.11.18.46-i386-debug-laptop/src/lib/libc/stdlib/abort.c:74
  #3  0xb9a395de in rumpuser_mutex_enter_nowrap (mtx=0xb94914c0)
      at /tmp/build/2021.12.12.11.18.46-i386-debug-laptop/src/lib/librumpuser/rumpuser_pth.c:211
  #4  0xb9a3967a in rumpuser_mutex_enter (mtx=0xb94914c0)
      at /tmp/build/2021.12.12.11.18.46-i386-debug-laptop/src/lib/librumpuser/rumpuser_pth.c:196
  #5  0xb9b18a85 in mutex_enter (mtx=0xb9752e90)
      at /tmp/build/2021.12.12.11.18.46-i386-debug-laptop/src/lib/librump/../../sys/rump/librump/rumpkern/locks.c:166
  #6  0xb9c49aa4 in cgd_process (wk=0xb5ca6fb0, arg=0x0)
      at /tmp/build/2021.12.12.11.18.46-i386-debug-laptop/src/sys/rump/dev/lib/libcgd/../../../../dev/cgd.c:1574
  #7  0xb9abbae8 in workqueue_runlist (list=0xb9463a94, list=0xb9463a94, wq=0xb9463a40)
      at /tmp/build/2021.12.12.11.18.46-i386-debug-laptop/src/lib/librump/../../sys/rump/../kern/subr_workqueue.c:105
  #8  workqueue_worker (cookie=<optimized out>, cookie@entry=0xb9463a40)
      at /tmp/build/2021.12.12.11.18.46-i386-debug-laptop/src/lib/librump/../../sys/rump/../kern/subr_workqueue.c:135
  #9  0xb9b1bd04 in threadbouncer (arg=0xb9752ec0)
      at /tmp/build/2021.12.12.11.18.46-i386-debug-laptop/src/lib/librump/../../sys/rump/librump/rumpkern/threads.c:90
  #10 0xb9a26e8b in pthread__create_tramp (cookie=0xb6ff4000)
      at /tmp/build/2021.12.12.11.18.46-i386-debug-laptop/src/lib/libpthread/pthread.c:561
  #11 0xb9863b10 in __mknod50 () from /usr/lib/libc.so.12
  (gdb) frame 4
  #4  0xb9a3967a in rumpuser_mutex_enter (mtx=0xb94914c0)
      at /tmp/build/2021.12.12.11.18.46-i386-debug-laptop/src/lib/librumpuser/rumpuser_pth.c:196
  196	in /tmp/build/2021.12.12.11.18.46-i386-debug-laptop/src/lib/librumpuser/rumpuser_pth.c
  (gdb) print /x *mtx
  $4 = {pthmtx = {ptm_magic = 0xdead0003, ptm_errorcheck = 0x1, ptm_pad1 = {0x0,
        0x0, 0x0}, {ptm_ceiling = 0x0, ptm_unused = 0x0}, ptm_pad2 = {0x0, 0x0,
        0x0}, ptm_owner = 0x0, ptm_waiters = 0x0, ptm_recursed = 0x0,
      ptm_spare2 = 0x0}, owner = 0x0, flags = 0x3}

>How-To-Repeat:

cd /usr/tests/dev/cgd/
sysctl -w kern.defcorename="/tmp/%n.core"
i=0; while echo && echo $i && atf-run t_cgd_aes:cgd_aes_cbc_128_encblkno8; do i=$(expr $i + 1); done
gdb ./t_cgd_aes /tmp/t_cgd_aes.core

>Fix: