kern/54209: NetBSD 8 large memory system performance extremely low

[kardel%netbsd.org@localhost]

>Number:         54209
>Category:       kern
>Synopsis:       NetBSD 8 large memory performance extremely low
>Confidential:   no
>Severity:       serious
>Priority:       medium
>Responsible:    kern-bug-people
>State:          open
>Class:          sw-bug
>Submitter-Id:   net
>Arrival-Date:   Thu May 16 14:40:00 +0000 2019
>Originator:     Frank Kardel
>Release:        NetBSD 8.0_STABLE
>Organization:
	
>Environment:
	
	
System: NetBSD sf2 8.0_STABLE NetBSD 8.0_STABLE (GENERIC) #4: Wed Apr 24 15:54:41 CEST 2019 kardel@pip:/src/NetBSD/n8/src/obj.amd64/sys/arch/amd64/compile/GENERIC amd64
Architecture: x86_64
Machine: amd64
>Description:
	This is an observation recited from memory to document as a PR for further investigation.
	System is a 16 CPU 382 GB server with 1 NVME disk and 6 SSDs.
	Using the default vm configuration the system gets very unresponsive and performs
	almost no IO and user cpu with following workload:
		5 larger (~10GB) java processes
			These process where performing computational and database work (postgresql 10 - DB is shared between java processes).
		2 rsyncs perfoming local disk/disk copies
	Very slow response time for shell input.
	We observed 100% system time, almost no I/O (for DB or the rsyncs), also free memory was stuck for long times at 380Mb.
	Inactive memory was at 7Gb instead of the expected 2/3 main memory.
	Lots of memory starvation was happening.
	The memory starvation effects on a larger memory system seem to be a bit counterintuative.
>How-To-Repeat:
	see description
>Fix:
	Performance improved radically giving responsiveness, I/O performance and throughput at lower system time
	when setting following vm tunables in order to force more aggressive memory reclaimation.

	    vm.anonmin=10
	    vm.filemin=10
	    vm.execmin=5
	    vm.anonmax=30
	    vm.filemax=20
	    vm.execmax=30
	    vm.inactivepct=40

>Unformatted: