Re: misc/60958 ((Probaby) avoidable stability problems due to default memory management behavior on systems with a lot of RAM, ZFS, and no swap)

["Taylor R Campbell via gnats" <gnats-admin%NetBSD.org@localhost>]

The following reply was made to PR misc/60958; it has been noted by GNATS.

From: Taylor R Campbell <riastradh%NetBSD.org@localhost>
To: Ellie <el%horse64.org@localhost>
Cc: gnats-bugs%netbsd.org@localhost, misc-bug-people%netbsd.org@localhost,
	netbsd-bugs%netbsd.org@localhost, gnats-admin%netbsd.org@localhost
Subject: Re: misc/60958 ((Probaby) avoidable stability problems due to default memory management behavior on systems with a lot of RAM, ZFS, and no swap)
Date: Tue, 3 Nov 2026 20:20:31 +0000

 > Date: Tue, 3 Nov 2026 19:42:47 +0000
 > From: Ellie <el%horse64.org@localhost>
 > 
 > During the unfolding of events, I observed the only major process taking 
 > up a lot of memory (Firefox) continuing to grow, and free memory (shown 
 > in the top right of "top") going to zero then suddenly jumping to 8GB.
 
 Which columns were you counting?
 
 SIZE is the virtual address space size, and doesn't mean much:
 processes can map large swaths of virtual address space without ever
 needing backing pages allocated for it.
 
 RES is the resident set size, i.e., amount of RAM actually used as
 backing pages in the process's address space.  This means more than
 SIZE for RAM usage -- but it may be double-counted, e.g. if two
 processes have the same file (such as the multi-gigabyte libxul.so)
 mapped into their address space and in use.  So the RES column doesn't
 sum to the systemwide RAM in use.
 
 > Another such jump happened from zero free memory back to around 4GB. At 
 > the same time, "Swap / Pools" went down - I didn't check if by the exact 
 > same amount, but from around 12G to less than 1G at the end.
 
 FYI, you should expect to only rarely have substantial amounts of
 `Free' memory in top(1) -- e.g., shortly after memory-intensive
 processes have exited or large cached files have been deleted, but
 before the pages have been reused to cache file data.
 
 It is almost always better for the system to use RAM to cache file
 data indefinitely than to discard the file data promptly to `free' the
 RAM:
 
 - If the system discards the file data, and someone asks for the file
   data again, the system has to read it back from disk, which is
   costly.
 
 - If the system keeps the file data, and someone asks for more RAM,
   the system can just ditch the file data and recycle the RAM
   immediately, which is almost free (except possibly for zeroing the
   pages).
 
 So the system does not aim to keep the `Free' memory high; it only
 aims to free RAM on demand, generally by finding inactive cached file
 data to discard.
 
 > https://horse64.org/100_7111.JPG
 
 404?
 
 > For example I had a terminal open, I typed "dmesg" which I still saw as 
 > a word appearing in the terminal, but after pressing enter, I just get a 
 > blank line and nothing ever happened. See in the photo!
 
 Does ^T respond in that case?
 
 > > 4. output of the following commands at various stages?
 > >     - echo ps | crash
 > >     - echo ps/w | crash
 > >     - echo show all locks | crash
 > >     - echo show all tstiles | crash
 > 
 > I assume that once "dmesg" no longer works, I wouldn't be able to run 
 > these anymore.
 > 
 > I could collect those at earlier stages before memory runs out, perhaps 
 > before & after the first possibly-ZFS-eviction, would that help?
 
 If you notice anything hanging, you could try that.  If it doesn't
 work, the equivalent commands are available in ddb, which runs in the
 kernel even if userland is stuck.
 
 > > 6. if you can't start crash, try switching to the first virtual
 > >     terminal ttyE0 with C-A-F1, drop into ddb with C-A-ESC, and take a
 > >     photo of the first page of output of the same commands (ps, ps/w,
 > >     show all locks, show all tstiles) -- don't go wild recording
 > >     everything, mostly just want to see a few wchans to find what
 > >     processes are waiting on
 > 
 > I couldn't start anything at that point, SSH login doesn't even work 
 > anymore once "dmesg" also no longer does. I highly doubt the TTY login 
 > would still work at that stage.
 
 You don't need to interact with the login process.  The logic of
 switching virtual terminals mostly happens in the kernel, and the
 logic of entering ddb with C-A-ESC is entirely in the kernel.