Re: "zfs send" freezes system (was: Re: pgdaemon high CPU consumption)

[Chuck Silvers <chuq%chuq.com@localhost>]

On Tue, Jul 19, 2022 at 08:46:07AM +0200, Matthias Petermann wrote:
> Hello,
> 
> On 13.07.22 12:30, Matthias Petermann wrote:
> 
> > I can now confirm that reverting the patch also solved my problem. Of
> > course I first fell into the trap, because I had not considered that the
> > ZFS code is loaded as a module and had only changed the kernel. As a
> > result, it looked at first as if this would not help. Finally it did...I
> > am now glad that I can use a zfs send again in this way. This previously
> > led reproducibly to a crash, whereby I could not make backups. This is
> > critical for me and I would like to support tests regarding this.
> > 
> > In contrast to the PR, there are hardly any xcalls in my use case -
> > however, my system only has 4 CPU cores, 2 of which are physical.
> > 
> > 
> > Many greetings
> > Matthias
> > 
> 
> Roundabout one week after removing the patch, my system with ZFS is behaving
> "normally" for the most part and the freezes have disappeared. What is the
> recommended way given the 10 branch? If it is not foreseeable that the basic
> problem can be solved shortly, would it also be an option to withdraw the
> patch in the sources to get at least a stable behavior? (Not only) on the
> sidelines, I would still be interested in whether this "zfs send" problem
> occurs in general, or whether certain hardware requirements have a favorable
> effect on it.
> 
> Kind regards
> Matthias


hi, sorry for the delay in getting to this.

what is happening here is that the pagedaemon is hitting the check for
"uvm_km_va_starved_p()", which tries to keep the usage of kernel memory
below 90% of the virtual space available for kernel memory.  the checks that
I changed (effectively removed for 64-bit kernels) in that previous patch
tried to keep the ARC kernel memory usage below 75% of the kernel virtual space.
on other OSs that support ZFS, the kernel allocates enough virtual space for
the kernel be able to allocate almost all of RAM for itself if it wants,
but on netbsd we have this calculation in kmeminit_nkmempages():


#if defined(KMSAN)
	npages = (physmem / 8);
#elif defined(PMAP_MAP_POOLPAGE)
	npages = (physmem / 4);
#else
	npages = (physmem / 3) * 2;
#endif /* defined(PMAP_MAP_POOLPAGE) */

#ifndef NKMEMPAGES_MAX_UNLIMITED
	if (npages > NKMEMPAGES_MAX)
		npages = NKMEMPAGES_MAX;
#endif



this limits the amount of kernel memory to 1/4 of RAM on 64-bit platforms.
PMAP_MAP_POOLPAGE is for accessing pool objects that are smaller than a page
using a direct-mapped region of virtual addresses.  all 64-bit kernels can
do this... though it looks like sparc64 doesn't do this for such pool allocations
even though it could?  weird.

most 64-bit kernels also define NKMEMPAGES_MAX_UNLIMITED to indicate that
no arbitrary fixed limit should be imposed on kernel memory usage.
though again not all platforms that could define this actually do.
this time it's the mips kernels that don't enable this one.

for ZFS, the memory used for the ARC cache is allocated through pools
but the allocation sizes are almost all larger than a page,
so basically none of these allocations will be able to use the direct map,
and instead they will all have to allocate kernel virtual space.
I don't think it makes sense for the kernel to arbitrarily limit
the ZFS ARC cache to 1/4 of RAM just because that's how much virtual space
is made available for kernel memory mappings, so instead I think we should
increase the size of the kernel virtual space on 64-bit kernels to support
mapping all of RAM, something like the attached patch.

however even with this change, reading an bunch of data into the ZFS ARC
still results in the system hanging, this time due to running out of
physical memory.  there are other mechanisms that ZFS also uses to try to
control its memory usage, and some part of that is apparently not
working either.  I'm continuing to look into this.

-Chuck

Index: src/sys/uvm/uvm_km.c
===================================================================
RCS file: /home/chs/netbsd/cvs/src/sys/uvm/uvm_km.c,v
retrieving revision 1.160
diff -u -p -r1.160 uvm_km.c
--- src/sys/uvm/uvm_km.c	13 Mar 2021 15:29:55 -0000	1.160
+++ src/sys/uvm/uvm_km.c	26 Jul 2022 20:24:14 -0000
@@ -237,6 +237,8 @@ kmeminit_nkmempages(void)
 #ifndef NKMEMPAGES_MAX_UNLIMITED
 	if (npages > NKMEMPAGES_MAX)
 		npages = NKMEMPAGES_MAX;
+#else
+	npages = physmem;
 #endif
 
 	if (npages < NKMEMPAGES_MIN)