Re: Unexpected out of memory kills when running parallel find instances over millions of files

[mlelstv%serpens.de@localhost (Michael van Elst)]

mjguzik%gmail.com@localhost (Mateusz Guzik) writes:

>Running 20 find(1) instances, where each has a "private" tree with
>million of files runs into trouble with the kernel killing them (and
>others):
>[   785.194378] UVM: pid 1998.1998 (find), uid 0 killed: out of swap


>This should not be happening -- there is tons of reusable RAM as
>virtually all of the vnodes getting here are immediately recyclable.

While vnodes would be recyclable, they hardly get recycled unless
an filesystem object is deleted or the filesystem is unmounted.

>Specs are 24 cores, 24G of RAM and ufs2 with noatime. swap is *not* configured.

Without swap, the kernel also has no chance to evict process pages
to grow the vnode cache further.

You can try to avoid that situation by reducing the amount of cached
vnodes by setting kern.maxvnodes with sysctl. That value would need
to be more dynamic to actually exercise pressure when memory runs short.

N.B. it's possible for the system to lock up too. You can change maxvnodes
through ddb by setting the kernel variable desiredvnodes. There is a good
chance that the system recovers.