kern/55408: systat vm hogs CPU and RAM after a few hours of system uptime

[bernd.sieker%posteo.net@localhost]

>Number:         55408
>Category:       kern
>Synopsis:       systat vm hogs CPU and RAM after a few hours of system uptime
>Confidential:   no
>Severity:       serious
>Priority:       medium
>Responsible:    kern-bug-people
>State:          open
>Class:          sw-bug
>Submitter-Id:   net
>Arrival-Date:   Tue Jun 23 11:15:00 +0000 2020
>Originator:     bernd.sieker%posteo.net@localhost
>Release:        NetBSD 9.0_STABLE
>Organization:
n/a
>Environment:
System: NetBSD niob.bersie.loc 9.0_STABLE NetBSD 9.0_STABLE (NIOB) #8: Wed Mar 18 18:04:09 CET 2020 bernd%niob.bersie.loc@localhost:/usr/source/netbsd-9/src/sys/arch/amd64/compile/NIOB amd64
Architecture: x86_64
Machine: amd64

dmesg:
[     1.000000] NetBSD 9.0_STABLE (NIOB) #8: Wed Mar 18 18:04:09 CET 2020
[     1.000000]         bernd%niob.bersie.loc@localhost:/usr/source/netbsd-9/src/sys/arch/amd64/compile/NIOB
[     1.000000] total memory = 40950 MB
[     1.000000] avail memory = 39737 MB
[     1.000000] timecounter: Timecounters tick every 10.000 msec
[     1.000000] Kernelized RAIDframe activated
[     1.000000] running cgd selftest aes-xts-256 aes-xts-512 done
[     1.000000] timecounter: Timecounter "i8254" frequency 1193182 Hz quality 100
[     1.000003] Sun Microsystems SUN FIRE X2270 M2 (REV 50     )
	[...]
[     1.000003] cpu0: Intel(R) Xeon(R) CPU           X5675  @ 3.07GHz, id 0x206c2
[     1.000003] cpu0: package 0, core 0, smt 0
	[...]
[     1.000003] cpu23: Intel(R) Xeon(R) CPU           X5675  @ 3.07GHz, id 0x206c2
[     1.000003] cpu23: package 1, core 10, smt 1
	[...]
[     5.013327] raid0: RAID Level 5
[     5.013327] raid0: Components: /dev/dk0 /dev/dk1 /dev/dk2
[     5.013327] raid0: Total Sectors: 15625976960 (7629871 MB)
[     5.013327] raid0: GPT GUID:
	[...]

ldd:
/usr/bin/systat:
        -lutil.7 => /usr/lib/libutil.so.7
        -lc.12 => /usr/lib/libc.so.12
        -lcurses.8 => /usr/lib/libcurses.so.8
        -lterminfo.1 => /usr/lib/libterminfo.so.1
        -lm.0 => /usr/lib/libm.so.0
        -lkvm.6 => /usr/lib/libkvm.so.6

ident:
/usr/bin/systat:
     $NetBSD: crt0.S,v 1.4 2018/11/26 17:37:46 joerg Exp $
     $NetBSD: crt0-common.c,v 1.23 2018/12/28 20:12:35 christos Exp $
     $NetBSD: crti.S,v 1.1 2010/08/07 18:01:35 joerg Exp $
     $NetBSD: crtbegin.S,v 1.2 2010/11/30 18:37:59 joerg Exp $
     $NetBSD: bufcache.c,v 1.29 2018/12/26 01:47:37 sevan Exp $
     $NetBSD: cmds.c,v 1.30 2016/10/22 22:02:55 christos Exp $
     $NetBSD: cmdtab.c,v 1.26 2018/12/26 01:47:37 sevan Exp $
     $NetBSD: convtbl.c,v 1.2 2016/08/02 17:53:46 scole Exp $
     $NetBSD: disks.c,v 1.19 2014/03/08 20:51:20 jdc Exp $
     $NetBSD: df.c,v 1.4 2005/12/24 21:14:50 matt Exp $
     $NetBSD: fetch.c,v 1.13 2009/04/13 23:20:27 lukem Exp $
     $NetBSD: globalcmds.c,v 1.15 2009/11/01 22:08:14 dsl Exp $
     $NetBSD: icmp.c,v 1.13 2014/06/03 22:22:41 joerg Exp $
     $NetBSD: ifcmds.c,v 1.2 2016/08/02 17:53:46 scole Exp $
     $NetBSD: ifstat.c,v 1.4 2016/08/05 07:22:17 christos Exp $
     $NetBSD: iostat.c,v 1.39 2019/01/25 15:31:11 christos Exp $
     $NetBSD: ip.c,v 1.18 2014/06/03 22:22:41 joerg Exp $
     $NetBSD: keyboard.c,v 1.26 2015/08/23 18:33:15 mrg Exp $
     $NetBSD: main.c,v 1.55 2019/01/25 15:31:11 christos Exp $
     $NetBSD: mbufs.c,v 1.16 2012/11/23 01:43:10 christos Exp $
     $NetBSD: netcmds.c,v 1.21 2005/02/26 22:12:33 dsl Exp $
     $NetBSD: netstat.c,v 1.30 2018/05/03 07:13:48 maxv Exp $
     $NetBSD: pigs.c,v 1.33 2012/11/23 03:46:35 christos Exp $
     $NetBSD: ps.c,v 1.38 2019/02/03 10:48:47 mrg Exp $
     $NetBSD: swap.c,v 1.20 2008/05/30 02:29:37 mrg Exp $
     $NetBSD: tcp.c,v 1.16 2014/06/03 22:22:41 joerg Exp $
     $NetBSD: vmstat.c,v 1.87 2019/06/30 19:57:23 he Exp $
     $NetBSD: utmpentry.c,v 1.18 2015/11/21 15:01:43 christos Exp $
     $NetBSD: syscall.c,v 1.10 2019/01/25 15:31:11 christos Exp $
     $NetBSD: syscalls.c,v 1.313 2018/08/10 21:47:16 pgoyette Exp $
     $NetBSD: ip6.c,v 1.16 2014/06/03 22:22:41 joerg Exp $
     $NetBSD: crtend.S,v 1.1 2010/08/07 18:01:34 joerg Exp $
     $NetBSD: crtn.S,v 1.1 2010/08/07 18:01:35 joerg Exp $

>Description:
After the system has been running for a couple of hours with only
a few mostly idle services (dhcpd, named, nfs, smbd, afpd, nginx),
"systat vm" reports "can't get buffers: Cannot allocate memory"
and starts growing and using more and more CPU time.  At the default
display interval of one second it eventually reaches more than 30%
CPU time.

Typical truncated top output:

  PID USERNAME PRI NICE   SIZE   RES STATE      TIME   WCPU    CPU COMMAND
  162 bernd     84    0   154M  121M ttyraw/1  40:44 31.54% 31.54% sysstat

For the first few hours after system boot time, size of systat is
less than 20 MB and CPU usage is mostly below top's "idle" threshold,
i. e.  less than 0.005%, displayed as "0.00%", and ocassionally
just above 1%.

The kernel is a GENERIC-derived non-modular custom-built kernel with nvmm enabled and
some unused hardware drivers removed.

It also happens with the unmodified GENERIC 9.0 kernel downloaded from
ftp://ftp.netbsd.org/pub/NetBSD/NetBSD-9.0/amd64/binary/kernel/netbsd-GENERIC.gz

It did not happen on NetBSD 8.x. A memory-upgrade from 8 to 40 GB
co-incided with the upgrade from NetBSD 8.1 to 9.0.

>How-To-Repeat:
Boot NetBSD-9.0/amd64, start "top" and "systat vm" and wait for a
day. Observere systat CPU and memory consumption and an error
message by systat.

>Fix:
Unknown. I currently know of no workaround.

>Unformatted:
 	cvs last updated to netbsd-9 branch on 2020-03-18T1905
 	also happens with stock 9.0 GENERIC kernel