Re: Tuning ZFS memory usage on NetBSD - call for advice

[Lloyd Parkes <lloyd%must-have-coffee.gen.nz@localhost>]

It might not be ZFS related. But it could be.

Someone else reported excessive, ongoing, increasing "File" usage a 
while back and I was somewhat dismissive because they were running a 
truckload of apps at the same time (not in VMs).

I did manage to reproduce his problem on an empty non-ZFS NetBSD system, 
so there is definitely something going on where "File" pages are not 
getting reclaimed when there is pressure on the memory system.

I haven't got around to looking into it any deeper though.

BTW the test was to copy a single large file (>1TB?) from SATA storage 
to USB storage. Since the file is held open for the duration of the copy 
(I used dd IIRC) this might end up exercising many of the same code 
paths as a VM accessing a disk image.

Cheers,
Lloyd

On 31/08/22 22:52, Matthias Petermann wrote:
> Hello all,
>
> under [1] is described in the section "Memory usage", which requirements 
> ZFS has for the memory.
>
> It further mentions that the tunables that exist in FreeBSD do not exist 
> in NetBSD. Especially for the size of the ARC there seems to be no limit 
> for NetBSD:
>
> "vfs.zfs.arc_max - Upper size of the ARC. The default is all RAM but 1 
> GB, or 5/8 of all RAM, whichever is more. Use a lower value if the 
> system runs any other daemons or processes that may require memory. 
> Adjust this value at runtime with sysctl(8) and set it in 
> /boot/loader.conf or /etc/sysctl.conf."
>
> So far so good... I have here the concrete case that I use ZFS ZVOLS as 
> backend storage for virtual machines (Qemu/nvmm). The host has 8192 MB 
> RAM available, of which are allocated to the VMs:
>
> * net (512 MB RAM)
> * iot (1024 MB RAM)
> * mail (512 MB RAM)
> * app (2048 MB RAM)
>
> This should leave 4096 MB for the host - while ZFS would claim 5120 MB 
> (5/8) of the available RAM. On this, after a while, the value under 
> "File" in top increases to over 3 GB, with the consequence that the 
> system starts swapping to the swap partition.
>
> This raises the following questions for me:
>
> - how can I investigate the composition of the amount of memory 
> displayed under "File" in top more precisely?
>
> - are there any hidden tuning possibilities for ZFS in NetBSD (maybe 
> boot parameters etc.) or compile-time settings?
>
> - what kind of memory can basically be swapped out? Only memory of the 
> processes (e.g. RAM of the Qemu VMs) or also parts of the ZFS ARC?
>
> - Which value does ZFS use to determine the ARC upper limit? The 
> physical RAM or the physical RAM + swap? Background to the question: in 
> my example, would I perhaps be better off disabling swap?
>
> Kind regards
> Matthias
>
> Btw, sorry for the cross-posting. The host is running on a NetBSD 
> 9.3_STABLE, however the topic seems relevant for current as well and I 
> would have the possibility to test or compare on current as well.
>
>
> [1] https://wiki.netbsd.org/zfs/
> [2] https://docs.freebsd.org/en/books/handbook/zfs/#zfs-advanced