Re: Using mmap(2) in sort(1) instead of temp files

[Robert Elz <kre%munnari.OZ.AU@localhost>]

    Date:        Thu, 4 Apr 2024 10:05:17 -0400 (EDT)
    From:        Mouse <mouse%Rodents-Montreal.ORG@localhost>
    Message-ID:  <202404041405.KAA01317%Stone.Rodents-Montreal.ORG@localhost>

  | Actually, if you mmap it PROT_WRITE and MAP_PRIVATE, you could go right
  | ahead.  But that'll cost RAM or swap space when the COW fault happens.

And the overheads from doing that (unless the file has very long records,
every block in the file would likely end up being copied, for a large file)
would drawf whatever small saving avoiding the buffering (in kernel, and
probably in stdio though that could be avoided) would gain.

  | It also works only when the input file fits into VM; to rephrase part
  | of what I wrote yesterday on tech-kern, sorting a file bigger than 4G
  | on a 32-bit port shouldn't break.

Agreed.   Aside from anything else sort(1) needs the ability to merge
already sorted files to handle the -m option, what it is doing using
temp files is simply setting up that scenario, in the case where the
file is big enough that it cannot simply be sorted in core.

  | (well, MAP_ANON).  Yes, but that has issues.  The size of an mmap()ped
  | area is fixed, set at map time, whereas file sizes grow dynamically.  I
  | suspect that trying to use mmap instead of temp files would amount to
  | implementing a rudimentary ramfs.

Yes, in cases where temp files are actually needed, using mmap() is a
very minor gain indeed - the buffering cost might be saved, but sorting
a large file is a cpu costly endeavour (lots of comparisons, lots of times
even with the best sorting algorithms available) so when temp files are
needed in the first place (large input files) the saving is liklely to be
a few ms in an operation which takes minutes of cpu time (or more).
Not worth the bother.

  | Furthermore, if the dataset fits in RAM, I'd say you shouldn't be using
  | the temporary-space paradigm at all; just slurp it in and sort it in
  | core.

Also agreed.   Whether the "slurp" means fgets(), read(), or mmap() doesn't
matter a lot for this, provided additional hidden copies are also being
made (ie: you want the data set to fit in RAM, as well as in the VM space,
as soon as paging/swapping starts, using temp files is likely better).
But to be sure about that, someone would need to benchmark it.

Certainly, simply believing "mmap() must be faster, therefore better"
is folly.    There are applications for which it wins, and others where
it doesn't - my gut feeling is that sorting is one where it doesn't,
or at least not enough in the cases that matter to justify the extra
code complexity - saving a little when sorting a small file, which might
be possible, isn't good enough IMO.

I'd simply forget the whole thing, unless someone is motivated enough to
really benchmark various implementations for sorting LARGE files (say
several TB) and post the results.

kre