Re: getting creative with cp (was Re: cp -n diff)

[Lourival Vieira Neto <lourival.neto%gmail.com@localhost>]

On Fri, May 6, 2011 at 2:14 PM, David Young <dyoung%pobox.com@localhost> wrote:
> On Tue, May 03, 2011 at 10:08:12PM +0200, Marc Balmer wrote:
>> I am sure as a creative developers community we can think of many other
>> useful, non-POSIX additions to cp.
>
> We do have a creative developer community, and I think that our
> developers are open to the idea that we should strive for the most
> useful, modern system that fits with the UNIX design philosophy.  UNIX
> is more than forty years old, and it may sometimes require more serious
> renovations than whittling away at command options.  Here are a handful
> of possible renovations that this cp discussion has brought to mind:
>
> A) De-composition / refactoring of tools: can cp, or can cp, ln, mv, and
>   rm, divide into two or more useful parts?  For example, does it help
>   to think of cp as a two-part process: 1) compare two directories
>   or trees, 2) resolve differences between the directories/trees by
>   creating/destroying links or by making copies.
>
>   In the first part of the process, are there useful ways to compare
>   files than by pathname?  Content?
>
>   Can we create one or more useful tools for each part of the
>   process?
>
> B) Speed it up: couldn't cp copy large directory trees much, much faster
>   if it copied files concurrently instead of sequentially, thus giving
>   the disk scheduler a lot more requests to chew on?  (See "Deceptive
>   idleness.")
>
> C) Speed it up some more: what if the kernel allowed for "lazy"
>   copies to be made when the source & destination were on the same
>   filesystem, and the filesystem supported it?  After a copy, source
>   and destination would share data blocks, which would be copied on any
>   subsequent write.  Yes, this would take a lot of kernel & filesystem
>   hacking.
>
> Dave
>
> --
> David Young             OJC Technologies
> dyoung%ojctech.com@localhost      Urbana, IL * (217) 344-0444 x24
>

Hi folks,

Sorry for the thread hijacking, but, coincidentally, I did an
experiment with cp using Lunatik [1] to speed up file copying by
avoiding context switches. I based this experiment on the
CompositeCalls idea [2] and I used the following Lua script:

function copy(fd, fdw, nbytes)
        while true do
                local nr = read(fd, nbytes)
                if not nr then break end
                write(fdw, nr)
        end
end

So,  I wrote a C program which uses the Lunatik ioctl [3] and the copy
function to copy a file instead of use read/write system calls. As
result, I have speedups (in relation to cp(1)) of around 20% with
files until ~500M. However, with big files (>= 1G), I have speedups
lower than 9%. I didn't understand this behavior yet.

[1] http://mail-index.netbsd.org/tech-kern/2010/10/05/msg008900.html
[2] Ballesteros, F. J. et al. Using interpreted CompositeCalls to
improve operating system services. Software Practice and Experience.
2000.
[3] http://netbsd-soc.sourceforge.net/projects/luakern/

Cheers,
-- 
Lourival Vieira Neto

P.S.: $ dmesg
NetBSD 5.99.38 (GENERIC) #0: Fri Jul 30 10:52:31 UTC 2010
        
builds%b7.netbsd.org@localhost:/home/builds/ab/HEAD/i386/201007300000Z-obj/home/builds/ab/HEAD/src/sys/arch/i386/co
mpile/GENERIC
total memory = 1978 MB
avail memory = 1933 MB
(...)
cpu0 at mainbus0 apid 0: Intel 686-class, 1296MHz, id 0x1067a
(...)
wd0 at atabus0 drive 0: <WDC WD2500BEVT-22A23T0>
wd0: drive supports 16-sector PIO transfers, LBA48 addressing
wd0: 232 GB, 484521 cyl, 16 head, 63 sec, 512 bytes/sect x 488397168 sectors
wd0: drive supports PIO mode 4, DMA mode 2, Ultra-DMA mode 6 (Ultra/133)
wd0(ahcisata0:0:0): using PIO mode 4, DMA mode 2, Ultra-DMA mode 6
(Ultra/133) (using DMA)