Re: getting creative with cp (was Re: cp -n diff)

[Lourival Vieira Neto <lourival.neto%gmail.com@localhost>]

Hi David,

On Sat, May 14, 2011 at 5:37 PM, David Holland 
<dholland-tech%netbsd.org@localhost> wrote:
> On Tue, May 10, 2011 at 04:24:30PM -0300, Lourival Vieira Neto wrote:
>  > Sorry for the thread hijacking, but, coincidentally, I did an
>  > experiment with cp using Lunatik [1] to speed up file copying by
>  > avoiding context switches. I based this experiment on the
>  > CompositeCalls idea [2] and I used the following Lua script:
>  >
>  > function copy(fd, fdw, nbytes)
>  >         while true do
>  >                 local nr = read(fd, nbytes)
>  >                 if not nr then break end
>  >                 write(fdw, nr)
>  >         end
>  > end
>  >
>  > So,  I wrote a C program which uses the Lunatik ioctl [3] and the copy
>  > function to copy a file instead of use read/write system calls. As
>  > result, I have speedups (in relation to cp(1)) of around 20% with
>  > files until ~500M. However, with big files (>= 1G), I have speedups
>  > lower than 9%. I didn't understand this behavior yet.
>
> That's basically sendfile, which we don't have because we believe that
> UVM's page loaning makes it mostly superfluous. IIUC.
>
> What size copy buffer did you get that speedup with?

I'm using MAXBSIZE for the buffer, which is allocated with kmem(9)
(only one time).

>  > total memory = 1978 MB
>  > avail memory = 1933 MB
>
> That's probably why 1G matters.

In fact, my experiment was extremely biased: copy script was rewriting
a file and cp(1) was creating a new one (w/ O_CREAT | O_TRUNC). This
was the reason of the speedup =/.

In order to perform a -better- less biased experiment, now I'm
comparing only the read/write loops and I have a more linear result.
However, with "real" files I have only speedups from 3.5% to 9%. When
I copy bytes from /dev/zero to /dev/null, I have speedups around
20/30%.

I don't know why I have these poor speedups for "real" files. Anyway,
I will cleanup my code and measures and post them soon.

Cheers,
--
Lourival Vieira Neto