Tuning ZFS memory usage on NetBSD - call for advice

[Matthias Petermann <mp%petermann-it.de@localhost>]

Hello all,

under [1] is described in the section "Memory usage", which requirements 
ZFS has for the memory.

It further mentions that the tunables that exist in FreeBSD do not exist 
in NetBSD. Especially for the size of the ARC there seems to be no limit 
for NetBSD:

"vfs.zfs.arc_max - Upper size of the ARC. The default is all RAM but 1 
GB, or 5/8 of all RAM, whichever is more. Use a lower value if the 
system runs any other daemons or processes that may require memory. 
Adjust this value at runtime with sysctl(8) and set it in 
/boot/loader.conf or /etc/sysctl.conf."

So far so good... I have here the concrete case that I use ZFS ZVOLS as 
backend storage for virtual machines (Qemu/nvmm). The host has 8192 MB 
RAM available, of which are allocated to the VMs:

* net (512 MB RAM)
* iot (1024 MB RAM)
* mail (512 MB RAM)
* app (2048 MB RAM)

This should leave 4096 MB for the host - while ZFS would claim 5120 MB 
(5/8) of the available RAM. On this, after a while, the value under 
"File" in top increases to over 3 GB, with the consequence that the 
system starts swapping to the swap partition.

This raises the following questions for me:

- how can I investigate the composition of the amount of memory 
displayed under "File" in top more precisely?

- are there any hidden tuning possibilities for ZFS in NetBSD (maybe 
boot parameters etc.) or compile-time settings?

- what kind of memory can basically be swapped out? Only memory of the 
processes (e.g. RAM of the Qemu VMs) or also parts of the ZFS ARC?

- Which value does ZFS use to determine the ARC upper limit? The 
physical RAM or the physical RAM + swap? Background to the question: in 
my example, would I perhaps be better off disabling swap?

Kind regards
Matthias

Btw, sorry for the cross-posting. The host is running on a NetBSD 
9.3_STABLE, however the topic seems relevant for current as well and I 
would have the possibility to test or compare on current as well.


[1] https://wiki.netbsd.org/zfs/
[2] https://docs.freebsd.org/en/books/handbook/zfs/#zfs-advanced