Re: Unexpected out of memory kills when running parallel find instances over millions of files

[Reinoud Zandijk <reinoud%dropje.13thmonkey.org@localhost>]

Hi,

On Thu, Oct 19, 2023 at 11:20:02AM +0200, Mateusz Guzik wrote:
> Running 20 find(1) instances, where each has a "private" tree with
> million of files runs into trouble with the kernel killing them (and
> others):
> [   785.194378] UVM: pid 1998.1998 (find), uid 0 killed: out of swap
> [   785.194378] UVM: pid 2010.2010 (find), uid 0 killed: out of swap
> [   785.224675] UVM: pid 1771.1771 (top), uid 0 killed: out of swap
> [   785.285291] UVM: pid 1960.1960 (zsh), uid 0 killed: out of swap
> [   785.376172] UVM: pid 2013.2013 (find), uid 0 killed: out of swap
> [   785.416572] UVM: pid 1760.1760 (find), uid 0 killed: out of swap
> [   785.416572] UVM: pid 1683.1683 (tmux), uid 0 killed: out of swap
> 
> This should not be happening -- there is tons of reusable RAM as
> virtually all of the vnodes getting here are immediately recyclable.
> 
> $elsewhere I got a report of a workload with hundreds of millions of
> files which get walked in parallel -- a number high enough that it
> does not fit in RAM on boxes which run it. Out of curiosity I figured
> I'll check how others are doing on the front, but key is that this is
> not a made up problem.

I can second that. I have had UVM killing my X11 when visiting millions of
files; it might have been using rump but I am not sure.

What struck me was that swap was maxed out but systat showed something like
40gb as `File'. I haven't looked at the Meta percentage but it wouldn't
surpise me if that was also high. Just some random snippet:

Anon      2847644   4%   zero             
Exec       378096    %   wired   11540    
File     41386972  63%   inact14002564    
Meta      8566989    %   bufs  2397524    

If a process really starts to take memory I'd say reduce the number of cached
inodes and buffered file data first before starting to swap out.

> 
> I'm running NetBSD 10, kernel built from this commit at top of the tree:
> Author: andvar <andvar%NetBSD.org@localhost>
> Date:   Sat Oct 14 08:05:25 2023 +0000
> 
>     fix various typos in comments and documentation, mainly in word "between".
> 
> Specs are 24 cores, 24G of RAM and ufs2 with noatime. swap is *not* configured.
> 
> Test generates 20 separate trees, each has 1000 directories with 1000
> files (or 20 million files in total + some dirs).
> 
> Repro instructions are here:
> https://people.freebsd.org/~mjg/.junk/fstree.tgz

I'll take a look in it.

Cheers,
Reinoud