Re: kern/57558: pgdaemon 100% busy - no scanning (ZFS case)

[Frank Kardel <kardel%netbsd.org@localhost>]

The following reply was made to PR kern/57558; it has been noted by GNATS.

From: Frank Kardel <kardel%netbsd.org@localhost>
To: gnats-bugs%netbsd.org@localhost, kern-bug-people%netbsd.org@localhost,
 gnats-admin%netbsd.org@localhost, netbsd-bugs%netbsd.org@localhost
Cc: 
Subject: Re: kern/57558: pgdaemon 100% busy - no scanning (ZFS case)
Date: Fri, 26 Apr 2024 17:34:12 +0200

 see below.
 
 On 04/26/24 16:20, Michael van Elst wrote:
 > The following reply was made to PR kern/57558; it has been noted by GNATS.
 >
 > From: mlelstv%serpens.de@localhost (Michael van Elst)
 > To: gnats-bugs%netbsd.org@localhost
 > Cc:
 > Subject: Re: kern/57558: pgdaemon 100% busy - no scanning (ZFS case)
 > Date: Fri, 26 Apr 2024 14:19:37 -0000 (UTC)
 >
 >   kardel%netbsd.org@localhost (Frank Kardel) writes:
 >   
 >   >Observed behavior:
 >   >pagedaemon runs at 100% (alreads at 550 minutes CPU and counting)
 >   
 >   I have local patches to avoid spinning. As the page daemon keeps
 >   data structures locked and runs at maximum priority, it prevents
 >   other tasks to release resources. That's only a little help,
 >   if the page daemon really cannot free anything, the system is still
 >   locked up to some degree.
 In this situation it spins only because the KVA starvation condition is 
 met. That triggers
 the pooldrain thread which in turn attempts to drain the pools. The ZFS 
 pools
 call into arc.c:hdr_recl() which triggers the arc_reclaim_thread. In 
 this scenario
 arc_availablemem() returns a positive value and thus the reclaim thread 
 does not
 reclaim anything.
 So the pagedaemon loops but nothing improves as long
 as arc_availmem returns positive values. At this point the zfs 
 statistics lists
 large amounts of evictable data.
 >   
 >   The main reason, that the page daemon cannot free memory is
 >   that vnodes are not drained. This keeps the associated pools
 >   busy and buffers allocated by the file cache.
 Well, this may not be the reason in this case - there is not even an attempt
 made to let the reclaim thread evict data and drain pools. It doesn't 
 get that far.
 >   
 >   Of course, if ZFS isn't throttled (and it would be less so, if
 >   others make room), it would just expunge the rest of the system
 >   data, so any improvement here just shifts the problem.
 Well ZFS currently likes to eat up all pool memory in this situation.
 >   
 >   >Once uvm_availmem() falls below uvmexp.freetarg the pagedaemon unwedges
 >   >from its tight loop as ZFS finally gives up its stranglehold on pool
 >   >memory..
 >   
 >   I locally added some of the arc tunables to experiment with the
 >   free_target value. The calculation of arc_c_max in arc.c also
 >   doesn't agree with the comments.
 >   
 >   Later ZFS versions did change a lot in this area. Anything we
 >   correct, might need to be redone, when we move to a newer ZFS
 >   code base.
 >   
 Yes, but currently we seem to have a broken ZFS at least on large memory 
 environments. Effects I observed
 are:
      - this bug: the famous looping pagedaemon
      - PR kern/58198: ZFS can lead to UVM kills (no swap, out of swap)
      - I am still trying to find out what causes the whole system to 
 slow down to a crawl. but that that point the system is unusable to 
 gather any information.
 
 So something needs to be done.