kern/54897: ipsec tests now fail randomly on real hardware

[gson%gson.org@localhost (Andreas Gustafsson)]

>Number:         54897
>Category:       kern
>Synopsis:       ipsec tests now fail randomly on real hardware
>Confidential:   no
>Severity:       serious
>Priority:       high
>Responsible:    kern-bug-people
>State:          open
>Class:          sw-bug
>Submitter-Id:   net
>Arrival-Date:   Sun Jan 26 16:35:00 +0000 2020
>Originator:     Andreas Gustafsson
>Release:        NetBSD-current, source adate >= 2020.01.02.15.42.27
>Organization:

>Environment:
System: NetBSD
Architecture: x86_64
Machine: amd64
>Description:

Various ipsec tests are now failing randomly on real hardware where
they used to relibably pass.  The problem started with these commits:

  2020.01.02.15.42.26 thorpej src/sys/compat/common/kern_info_43.c 1.38
  2020.01.02.15.42.26 thorpej src/sys/compat/common/kern_time_50.c 1.34
  2020.01.02.15.42.26 thorpej src/sys/compat/netbsd32/netbsd32_sysctl.c 1.41
  2020.01.02.15.42.26 thorpej src/sys/external/bsd/drm2/include/linux/ktime.h 1.8
  2020.01.02.15.42.26 thorpej src/sys/fs/nfs/common/nfs_lock.c 1.3
  2020.01.02.15.42.27 thorpej src/sys/kern/init_main.c 1.517
  2020.01.02.15.42.27 thorpej src/sys/kern/init_sysctl.c 1.223
  2020.01.02.15.42.27 thorpej src/sys/kern/kern_rndq.c 1.96
  2020.01.02.15.42.27 thorpej src/sys/kern/kern_tc.c 1.54
  2020.01.02.15.42.27 thorpej src/sys/kern/kern_time.c 1.203
  2020.01.02.15.42.27 thorpej src/sys/miscfs/fdesc/fdesc_vnops.c 1.131
  2020.01.02.15.42.27 thorpej src/sys/miscfs/kernfs/kernfs.h 1.41
  2020.01.02.15.42.27 thorpej src/sys/miscfs/kernfs/kernfs_vnops.c 1.162
  2020.01.02.15.42.27 thorpej src/sys/miscfs/procfs/procfs_linux.c 1.80
  2020.01.02.15.42.27 thorpej src/sys/nfs/nfs_serv.c 1.178
  2020.01.02.15.42.27 thorpej src/sys/rump/librump/rumpkern/cons.c 1.9
  2020.01.02.15.42.27 thorpej src/sys/rump/librump/rumpkern/emul.c 1.195
  2020.01.02.15.42.27 thorpej src/sys/rump/librump/rumpkern/rump.c 1.339
  2020.01.02.15.42.27 thorpej src/sys/sys/kernel.h 1.32
  2020.01.02.15.42.27 thorpej src/sys/sys/timevar.h 1.40

At least amd64 and i386 are affected.  Log output from a
representative recent failure on amd64 is at:

  http://www.gson.org/netbsd/bugs/build/amd64-baremetal/2020/2020.01.25.00.12.42/test.html#net_ipsec_t_ipsec_tcp_ipsec_tcp_ipv4mappedipv6_esp_rijndaelcbc

>How-To-Repeat:

I used the following script to reproduce the issue on i386:

  set -x
  cd /usr/tests/net/ipsec
  nfail=0
  for i in $(seq 100)
  do
      atf-run t_ipsec_tunnel >>log 2>&1 || nfail=$(expr $nfail + 1)
  done
  echo $nfail

This printed 0 on a system built from source date 2020.01.02.14.33.55
(immediately before the above commits), and 23 on a system built from
source date 2020.01.02.15.42.27 (immediately after the above commits).
In other words, before the commits, the t_ipsec_tunnel test program
failed 0 times out of 100, and after them, it failed 23 times out of
100.

>Fix: