Re: port-evbarm/56944: ZFS heavy usage on NetBSD running in Mac M1 VM results in kernel thread running away and filesystem hang

[Chuck Silvers <chuq%chuq.com@localhost>]

The following reply was made to PR port-evbarm/56944; it has been noted by GNATS.

From: Chuck Silvers <chuq%chuq.com@localhost>
To: gnats-bugs%netbsd.org@localhost
Cc: 
Subject: Re: port-evbarm/56944: ZFS heavy usage on NetBSD running in Mac M1
 VM results in kernel thread running away and filesystem hang
Date: Thu, 28 Jul 2022 03:46:18 -0700

 --MVVFUAH6qIqjAxhF
 Content-Type: text/plain; charset=us-ascii
 Content-Disposition: inline
 
 On Thu, Jul 28, 2022 at 12:10:02AM +0000, Tobias Nygren wrote:
 > The following reply was made to PR port-evbarm/56944; it has been noted by GNATS.
 > 
 > From: Tobias Nygren <tnn%NetBSD.org@localhost>
 > To: gnats-bugs%netbsd.org@localhost
 > Cc: 
 > Subject: Re: port-evbarm/56944: ZFS heavy usage on NetBSD running in Mac M1
 >  VM results in kernel thread running away and filesystem hang
 > Date: Thu, 28 Jul 2022 02:06:50 +0200
 > 
 >  I could easily reproduce the bug on a Raspberry Pi 4 with a bolted-on
 >  ahcisata adapter. With 3090M free memory it does not seem general
 >  memory shortage is the reason pgdaemon keeps getting woken. So you can
 >  disregard the previous patch. It looks rather that the reason is kva
 >  fragmentation, leak or mismanagement.
 
 I just posted about this on current-users... the problem is not fragmentation
 or a leak, though I suppose you could say that it's mismanagement.
 ZFS kind of assumes that on 64-bit kernels, kernel virtual space
 will not be the limiting factor for allocating kernel memory.
 however, netbsd currently allocates much less kernel virtual space than
 the system has RAM, so kernel virtual space can be the limiting factor
 for memory allocations even on 64-bit kernels.  increasing the
 kernel virtual space on 64-bit kernels removes this problem,
 but exposes others.
 
 your observation that we don't initialize zfs_arc_free_target is good,
 I hadn't found that yet before you mentioned it.  when I change that
 check to use the pagedaemon's free target then the "zfs send" test
 no longer triggers any obvious bad behavior.
 
 
 >  One possible problem here is ARC tries to maintain 1/16th of the kernel
 >  heap free:
 >  
 >  https://nxr.netbsd.org/xref/src/external/cddl/osnet/dist/uts/common/fs/zfs/arc.c#3990
 >  
 >  But UVM tries to maintain 1/10th free:
 >  
 >  https://nxr.netbsd.org/xref/src/sys/uvm/uvm_km.c#902
 >  
 >  This means UVM will notice and start to rectify kva shortage before ARC,
 >  when it really needs to be the other way around.
 
 with the arbitrary limit on kernel virtual space removed and
 zfs_arc_free_target fixed, this doesn't appear to be a problem in practice.
 I suspect this is because enough kernel memory is accessed via the direct map
 rather than being mapped in the kernel heap that the system always runs out
 of free pages before it runs out of free kva.
 
 my current patch with both of these changes is attached.
 
 -Chuck
 
 --MVVFUAH6qIqjAxhF
 Content-Type: text/plain; charset=us-ascii
 Content-Disposition: attachment; filename="diff.zfs-mem.1"
 
 Index: src/external/cddl/osnet/dist/uts/common/fs/zfs/arc.c
 ===================================================================
 RCS file: /home/chs/netbsd/cvs/src/external/cddl/osnet/dist/uts/common/fs/zfs/arc.c,v
 retrieving revision 1.21
 diff -u -p -r1.21 arc.c
 --- src/external/cddl/osnet/dist/uts/common/fs/zfs/arc.c	4 May 2022 15:49:55 -0000	1.21
 +++ src/external/cddl/osnet/dist/uts/common/fs/zfs/arc.c	28 Jul 2022 10:35:20 -0000
 @@ -288,6 +288,7 @@ int arc_procfd;
  #define	freemem		uvm_availmem(false)
  #define	minfree		uvmexp.freemin
  #define	desfree		uvmexp.freetarg
 +#define	zfs_arc_free_target desfree
  #define	lotsfree	(desfree * 2)
  #define	availrmem	desfree
  #define	swapfs_minfree	0
 @@ -387,7 +388,6 @@ int zfs_arc_grow_retry = 0;
  int zfs_arc_shrink_shift = 0;
  int zfs_arc_p_min_shift = 0;
  uint64_t zfs_arc_average_blocksize = 8 * 1024; /* 8KB */
 -u_int zfs_arc_free_target = 0;
  
  /* Absolute min for arc min / max is 16MB. */
  static uint64_t arc_abs_min = 16 << 20;
 @@ -395,6 +395,8 @@ static uint64_t arc_abs_min = 16 << 20;
  boolean_t zfs_compressed_arc_enabled = B_TRUE;
  
  #if defined(__FreeBSD__) && defined(_KERNEL)
 +u_int zfs_arc_free_target = 0;
 +
  static int sysctl_vfs_zfs_arc_free_target(SYSCTL_HANDLER_ARGS);
  static int sysctl_vfs_zfs_arc_meta_limit(SYSCTL_HANDLER_ARGS);
  static int sysctl_vfs_zfs_arc_max(SYSCTL_HANDLER_ARGS);
 Index: src/sys/uvm/uvm_km.c
 ===================================================================
 RCS file: /home/chs/netbsd/cvs/src/sys/uvm/uvm_km.c,v
 retrieving revision 1.160
 diff -u -p -r1.160 uvm_km.c
 --- src/sys/uvm/uvm_km.c	13 Mar 2021 15:29:55 -0000	1.160
 +++ src/sys/uvm/uvm_km.c	26 Jul 2022 20:24:14 -0000
 @@ -237,6 +237,8 @@ kmeminit_nkmempages(void)
  #ifndef NKMEMPAGES_MAX_UNLIMITED
  	if (npages > NKMEMPAGES_MAX)
  		npages = NKMEMPAGES_MAX;
 +#else
 +	npages = physmem;
  #endif
  
  	if (npages < NKMEMPAGES_MIN)
 
 --MVVFUAH6qIqjAxhF--