kern/55409: systat vm hogs CPU and RAM after a few hours of system uptime

[bernd.sieker%posteo.net@localhost]

>Number:         55409
>Category:       kern
>Synopsis:       systat vm hogs CPU and RAM after a few hours of system uptime
>Confidential:   no
>Severity:       serious
>Priority:       medium
>Responsible:    kern-bug-people
>State:          open
>Class:          sw-bug
>Submitter-Id:   net
>Arrival-Date:   Tue Jun 23 11:15:00 +0000 2020
>Originator:     bernd.sieker%posteo.net@localhost
>Release:        NetBSD 9.0_STABLE
>Organization:
n/a
>Environment:
	
System: NetBSD niob.bersie.loc 9.0_STABLE NetBSD 9.0_STABLE (NIOB) #8: Wed Mar 18 18:04:09 CET 2020 bernd%niob.bersie.loc@localhost:/usr/source/netbsd-9/src/sys/arch/amd64/compile/NIOB amd64

Architecture: x86_64
Machine: amd64

dmesg:
[     1.000000] NetBSD 9.0_STABLE (NIOB) #8: Wed Mar 18 18:04:09 CET 2020
[     1.000000]         bernd%niob.bersie.loc@localhost:/usr/source/netbsd-9/src/sys/arch/amd64/compile/NIOB
[     1.000000] total memory = 40950 MB
[     1.000000] avail memory = 39737 MB
[     1.000000] timecounter: Timecounters tick every 10.000 msec
[     1.000000] Kernelized RAIDframe activated
[     1.000000] running cgd selftest aes-xts-256 aes-xts-512 done
[     1.000000] timecounter: Timecounter "i8254" frequency 1193182 Hz quality 100
[     1.000003] Sun Microsystems SUN FIRE X2270 M2 (REV 50     )
	[...]
[     1.000003] cpu0: Intel(R) Xeon(R) CPU           X5675  @ 3.07GHz, id 0x206c2
[     1.000003] cpu0: package 0, core 0, smt 0
	[...]
[     1.000003] cpu23: Intel(R) Xeon(R) CPU           X5675  @ 3.07GHz, id 0x206c2
[     1.000003] cpu23: package 1, core 10, smt 1
	[...]
[     5.013327] raid0: RAID Level 5
[     5.013327] raid0: Components: /dev/dk0 /dev/dk1 /dev/dk2
[     5.013327] raid0: Total Sectors: 15625976960 (7629871 MB)
[     5.013327] raid0: GPT GUID: f0afab14-0e17-4f15-a465-4099134f1979
	[...]>Description:

ldd:
/usr/bin/systat:
        -lutil.7 => /usr/lib/libutil.so.7
        -lc.12 => /usr/lib/libc.so.12
        -lcurses.8 => /usr/lib/libcurses.so.8
        -lterminfo.1 => /usr/lib/libterminfo.so.1
        -lm.0 => /usr/lib/libm.so.0
        -lkvm.6 => /usr/lib/libkvm.so.6

ident:
/usr/bin/systat:
     $NetBSD: crt0.S,v 1.4 2018/11/26 17:37:46 joerg Exp $
     $NetBSD: crt0-common.c,v 1.23 2018/12/28 20:12:35 christos Exp $
     $NetBSD: crti.S,v 1.1 2010/08/07 18:01:35 joerg Exp $
     $NetBSD: crtbegin.S,v 1.2 2010/11/30 18:37:59 joerg Exp $
     $NetBSD: bufcache.c,v 1.29 2018/12/26 01:47:37 sevan Exp $
     $NetBSD: cmds.c,v 1.30 2016/10/22 22:02:55 christos Exp $
     $NetBSD: cmdtab.c,v 1.26 2018/12/26 01:47:37 sevan Exp $
     $NetBSD: convtbl.c,v 1.2 2016/08/02 17:53:46 scole Exp $
     $NetBSD: disks.c,v 1.19 2014/03/08 20:51:20 jdc Exp $
     $NetBSD: df.c,v 1.4 2005/12/24 21:14:50 matt Exp $
     $NetBSD: fetch.c,v 1.13 2009/04/13 23:20:27 lukem Exp $
     $NetBSD: globalcmds.c,v 1.15 2009/11/01 22:08:14 dsl Exp $
     $NetBSD: icmp.c,v 1.13 2014/06/03 22:22:41 joerg Exp $
     $NetBSD: ifcmds.c,v 1.2 2016/08/02 17:53:46 scole Exp $
     $NetBSD: ifstat.c,v 1.4 2016/08/05 07:22:17 christos Exp $
     $NetBSD: iostat.c,v 1.39 2019/01/25 15:31:11 christos Exp $
     $NetBSD: ip.c,v 1.18 2014/06/03 22:22:41 joerg Exp $
     $NetBSD: keyboard.c,v 1.26 2015/08/23 18:33:15 mrg Exp $
     $NetBSD: main.c,v 1.55 2019/01/25 15:31:11 christos Exp $
     $NetBSD: mbufs.c,v 1.16 2012/11/23 01:43:10 christos Exp $
     $NetBSD: netcmds.c,v 1.21 2005/02/26 22:12:33 dsl Exp $
     $NetBSD: netstat.c,v 1.30 2018/05/03 07:13:48 maxv Exp $
     $NetBSD: pigs.c,v 1.33 2012/11/23 03:46:35 christos Exp $
     $NetBSD: ps.c,v 1.38 2019/02/03 10:48:47 mrg Exp $
     $NetBSD: swap.c,v 1.20 2008/05/30 02:29:37 mrg Exp $
     $NetBSD: tcp.c,v 1.16 2014/06/03 22:22:41 joerg Exp $
     $NetBSD: vmstat.c,v 1.87 2019/06/30 19:57:23 he Exp $
     $NetBSD: utmpentry.c,v 1.18 2015/11/21 15:01:43 christos Exp $
     $NetBSD: syscall.c,v 1.10 2019/01/25 15:31:11 christos Exp $
     $NetBSD: syscalls.c,v 1.313 2018/08/10 21:47:16 pgoyette Exp $
     $NetBSD: ip6.c,v 1.16 2014/06/03 22:22:41 joerg Exp $
     $NetBSD: crtend.S,v 1.1 2010/08/07 18:01:34 joerg Exp $
     $NetBSD: crtn.S,v 1.1 2010/08/07 18:01:35 joerg Exp $

>Description:
After the system has been running for a couple of hours with only
a few mostly idle services (smbd, afpd, nginx), "systat vm" reports
"can't get buffers: Cannot allocate memory" and starts growing and
using more and more CPU time.  At the default display interval of
1 second it eventually reaches more than 30% CPU time.

  PID USERNAME PRI NICE   SIZE   RES STATE      TIME   WCPU    CPU COMMAND
  162 bernd     84    0   154M  121M ttyraw/1  40:44 31.54% 31.54% sysstat

The kernel is a GENERIC-derived non-modular custom-built kernel
with nvmm enabled and some unused hardware drivers removed.

This happens with and without the Linux emulation subsystem enabled
and/or running. It also happens on a freshly installed stock 9.0
and also on systems with fewer (4) CPU cores, so it does not seem
to be caused by some kernel tables being too small for 24 CPUs.

It also does not depend on nvmm being enabled or active.


>How-To-Repeat:
Install a new 9.0 or 9.0_STABLE system on amd64, let it run for a
while, having "systat vm" running, and watch system load in "top".

>Fix:
Unknown.

>Unformatted:
 	cvs last updated to netbsd-9 branch on 2020-03-18T1905