ethan@geocast.com (Ethan Solomita) wrote:
> 	Soft Updates is particularly susceptible to eating memory during a
> large series of removes, but that has always led to panics for me, not
> lockups. In particular, the kmem_map runs out of virtual address space.
> This has been improved greatly with Kirk McKusick's latest changes. In
> my testing, it requires three different processes to be doing huge file
> removals in parallel before it'll eat an unbounded amount of memory.
> This is still a bug, but it's much more tolerable for most people.

Some news admins could get very seriously hosed by this. If you're running
classic storage method (one-file-per-article, not CNFS), and you're doing
it on multiple volumes instead of one huge RAID or striped set (and there
were reasons to do this, which may still be valid), then you've probably
spent an hour optimizing news.daily to run simultaneous "fastrm"s on each of
the disk spindles simultaneously. (fastrm is a program that unlink()s lots
of files much more efficiently than rm.) Until we switched to a RAID system
about 18 months ago, we had news on three volumes, and each night we'd fastrm
on all three, 40-80k files per volume. I imagine that there are probably
people still doing things that way.

Come to think of it, I wonder if fastrm by itself might not trigger this
problem, even with the latest changes?

/a

---
Alexis Rosen
PANIX Public Access Unix & Internet, NYC.
alexis@panix.com