Re: misc/60958 ((Probaby) avoidable stability problems due to default memory management behavior on systems with a lot of RAM, ZFS, and no swap)

["Ellie via gnats" <gnats-admin%NetBSD.org@localhost>]

The following reply was made to PR misc/60958; it has been noted by GNATS.

From: Ellie <el%horse64.org@localhost>
To: gnats-bugs%netbsd.org@localhost, misc-bug-people%netbsd.org@localhost,
 netbsd-bugs%netbsd.org@localhost, gnats-admin%netbsd.org@localhost, riastradh%NetBSD.org@localhost
Cc: 
Subject: Re: misc/60958 ((Probaby) avoidable stability problems due to default
 memory management behavior on systems with a lot of RAM, ZFS, and no swap)
Date: Tue, 3 Nov 2026 19:42:47 +0000

 Happy to elaborate, and sorry if some of my speculation is wrong.
 
 For what it's worth, I was in the #NetBSD chat at the time and shared 
 details about the process, so some of this is based on the input of 
 others. I'm really not that terribly smart...
 
 Let me provide more details where I can:
 
 On 11/3/26 17:49, riastradh%NetBSD.org@localhost wrote:
 > Synopsis: (Probaby) avoidable stability problems due to default memory management behavior on systems with a lot of RAM, ZFS, and no swap
 > 
 > State-Changed-From-To: open->feedback
 > State-Changed-By: riastradh%NetBSD.org@localhost
 > State-Changed-When: Tue, 03 Nov 2026 17:49:14 +0000
 > State-Changed-Why:
 > There's a lot of speculation and analysis here, and not a lot of
 > specific evidence.  Can you share some more details like:
 > 
 > 1. exactly what observations led you to say `When memory runs out for
 >     the first and second time [...] ZFS evicts a lot of cache'?
 
 I had "top" open and was talking about the process in the chat, while 
 waiting for the OOM to confirm my hypothesis and observing what was 
 going on as best as I could.
 
 During the unfolding of events, I observed the only major process taking 
 up a lot of memory (Firefox) continuing to grow, and free memory (shown 
 in the top right of "top") going to zero then suddenly jumping to 8GB.
 
 Another such jump happened from zero free memory back to around 4GB. At 
 the same time, "Swap / Pools" went down - I didn't check if by the exact 
 same amount, but from around 12G to less than 1G at the end.
 
 > 
 > 2. exactly what observations led you to say `ZFS will evict so much of
 >     its own memory that it seems to start to break down'?
 
 This was my best guess because after the second jump from zero memory to 
 around 4GB free, I could no longer launch any processes:
 
 https://horse64.org/100_7111.JPG
 
 For example I had a terminal open, I typed "dmesg" which I still saw as 
 a word appearing in the terminal, but after pressing enter, I just get a 
 blank line and nothing ever happened. See in the photo!
 
 Firefox's window stopped redrawing, but generally X remained responsive 
 at first and I saw "top" continuing to run and update the remaining 
 memory count.
 
 That this is related to ZFS causing trouble for the I/O subsystem after 
 evicting so much memory was a mere guess and I could be 100% wrong.
 
 > 
 > 3. output of top(1) at various stages?
 
 You can see the output of "top" and "dmesg" being stuck on this photo 
 that I took at that point:
 
 https://horse64.org/100_7111.JPG
 
 Firefox had multiple processes listed that took over 30GB at that point.
 
 > 
 > 4. output of the following commands at various stages?
 >     - echo ps | crash
 >     - echo ps/w | crash
 >     - echo show all locks | crash
 >     - echo show all tstiles | crash
 
 I assume that once "dmesg" no longer works, I wouldn't be able to run 
 these anymore.
 
 I could collect those at earlier stages before memory runs out, perhaps 
 before & after the first possibly-ZFS-eviction, would that help?
 > 
 > 5. if you kill firefox, do the symptoms continue?
 
 The system stays perfectly alive without Firefox, for many hours. Once 
 "dmesg" hangs I wouldn't know how to kill Firefox. Before that, there 
 aren't really any symptoms. The first big perhaps-ZFS-eviction had no 
 noticeable effect other than the suspicious change in numbers in "top".
 
 > 
 > 6. if you can't start crash, try switching to the first virtual
 >     terminal ttyE0 with C-A-F1, drop into ddb with C-A-ESC, and take a
 >     photo of the first page of output of the same commands (ps, ps/w,
 >     show all locks, show all tstiles) -- don't go wild recording
 >     everything, mostly just want to see a few wchans to find what
 >     processes are waiting on
 
 I couldn't start anything at that point, SSH login doesn't even work 
 anymore once "dmesg" also no longer does. I highly doubt the TTY login 
 would still work at that stage.
 
 
 > 
 > 7. if you drop into ddb and type `call wakeup(uvmexp+0x10)' and then
 >     `continue', do the symptoms continue?
 > 
 > (For context: I have been running zfs (on cgd) and firefox with
 > hundreds of tabs on laptops with 32-64GB of RAM for years, and have not
 > encountered these symptoms.)
 
 I wonder, had you ever firefox take up all your memory? Perhaps you just 
 didn't let the right web apps sit in idle, or something.
 
 Also, people in chat were guessing that no swap might make a big 
 difference in how recoverable this situation.
 
 Regards,
 
 Ellie