Re: aarch64 reliability issues

[Greg Troxel <gdt%lexort.com@localhost>]

Lloyd Parkes <lloyd%must-have-coffee.gen.nz@localhost> writes:

> On Sun, 2025-05-18 at 07:49 +0000, Michael van Elst wrote:
>> 
>> ZFS has its own ideas on when to allocate and free memory, which will
>> also change if or when we get to update the code.
>> 
>
> ZFS is supposed to limit itself to 50% of RAM, but it has a certain
> reputation which is why I mentioned that I'm using it.
>
> vmstat -m and vmstat -mW look to be the right tools for finding out
> what memory pools are growing. I would be inclined to record those
> numbers in something like Prometheus, mostly because Prometheus makes
> pretty graphs.

Yes, but the real issue with our ZFS code has been diagnosed.  ZFS uses
lots of RAM for caching and mostly frees it just fine on memory
pressure.  If you unmount and unload zfs it is all freed, or at least
there's no evidence that it's not.

The problems are:

  There's no pressure mechanism to free vnodes from zfs under memory
  pressure, and those vnodes still hold refs to other memory.

  There is highly likely a deadlock between freeing memory used for zfs
  and getting dirty pages in mmaped files on zfs back to disk.  FreeBSD
  has a different thread for these things, and NetBSD doesn't.

The workaround is to set kern.maxvnodes low. and to be gentle on the
combination of dirty pages and memory pressure.  Pro Tip: Don't run
syncthing on GB of stuff while doing heavy builds.

I am using maxvnodes = 300000 on a machine with 32GB of RAM, and that
makes it mostly stable, rather than mean time to crash of sub-week.

The fix for the deadlock is awaiting round tuits.