Re: misc/60958 ((Probaby) avoidable stability problems due to default memory management behavior on systems with a lot of RAM, ZFS, and no swap)

[Ellie <el%horse64.org@localhost>]

Happy to elaborate, and sorry if some of my speculation is wrong.

For what it's worth, I was in the #NetBSD chat at the time and shared 
details about the process, so some of this is based on the input of 
others. I'm really not that terribly smart...

Let me provide more details where I can:

On 11/3/26 17:49, riastradh%NetBSD.org@localhost wrote:
> Synopsis: (Probaby) avoidable stability problems due to default memory management behavior on systems with a lot of RAM, ZFS, and no swap
>
> State-Changed-From-To: open->feedback
> State-Changed-By: riastradh%NetBSD.org@localhost
> State-Changed-When: Tue, 03 Nov 2026 17:49:14 +0000
> State-Changed-Why:
> There's a lot of speculation and analysis here, and not a lot of
> specific evidence.  Can you share some more details like:
>
> 1. exactly what observations led you to say `When memory runs out for
>     the first and second time [...] ZFS evicts a lot of cache'?

I had "top" open and was talking about the process in the chat, while 
waiting for the OOM to confirm my hypothesis and observing what was 
going on as best as I could.

During the unfolding of events, I observed the only major process taking 
up a lot of memory (Firefox) continuing to grow, and free memory (shown 
in the top right of "top") going to zero then suddenly jumping to 8GB.

Another such jump happened from zero free memory back to around 4GB. At 
the same time, "Swap / Pools" went down - I didn't check if by the exact 
same amount, but from around 12G to less than 1G at the end.

> 2. exactly what observations led you to say `ZFS will evict so much of
>     its own memory that it seems to start to break down'?

This was my best guess because after the second jump from zero memory to 
around 4GB free, I could no longer launch any processes:

https://horse64.org/100_7111.JPG

For example I had a terminal open, I typed "dmesg" which I still saw as 
a word appearing in the terminal, but after pressing enter, I just get a 
blank line and nothing ever happened. See in the photo!

Firefox's window stopped redrawing, but generally X remained responsive 
at first and I saw "top" continuing to run and update the remaining 
memory count.

That this is related to ZFS causing trouble for the I/O subsystem after 
evicting so much memory was a mere guess and I could be 100% wrong.

> 3. output of top(1) at various stages?

You can see the output of "top" and "dmesg" being stuck on this photo 
that I took at that point:

https://horse64.org/100_7111.JPG

Firefox had multiple processes listed that took over 30GB at that point.

> 4. output of the following commands at various stages?
>     - echo ps | crash
>     - echo ps/w | crash
>     - echo show all locks | crash
>     - echo show all tstiles | crash

I assume that once "dmesg" no longer works, I wouldn't be able to run 
these anymore.

I could collect those at earlier stages before memory runs out, perhaps 
before & after the first possibly-ZFS-eviction, would that help?
> 5. if you kill firefox, do the symptoms continue?

The system stays perfectly alive without Firefox, for many hours. Once 
"dmesg" hangs I wouldn't know how to kill Firefox. Before that, there 
aren't really any symptoms. The first big perhaps-ZFS-eviction had no 
noticeable effect other than the suspicious change in numbers in "top".

> 6. if you can't start crash, try switching to the first virtual
>     terminal ttyE0 with C-A-F1, drop into ddb with C-A-ESC, and take a
>     photo of the first page of output of the same commands (ps, ps/w,
>     show all locks, show all tstiles) -- don't go wild recording
>     everything, mostly just want to see a few wchans to find what
>     processes are waiting on

I couldn't start anything at that point, SSH login doesn't even work 
anymore once "dmesg" also no longer does. I highly doubt the TTY login 
would still work at that stage.


> 7. if you drop into ddb and type `call wakeup(uvmexp+0x10)' and then
>     `continue', do the symptoms continue?
>
> (For context: I have been running zfs (on cgd) and firefox with
> hundreds of tabs on laptops with 32-64GB of RAM for years, and have not
> encountered these symptoms.)

I wonder, had you ever firefox take up all your memory? Perhaps you just 
didn't let the right web apps sit in idle, or something.

Also, people in chat were guessing that no swap might make a big 
difference in how recoverable this situation.

Regards,

Ellie