Re: Another rfe: "cp" this time

[Pádraig Brady]

On 04/27/2012 03:32 PM, Bruce Korb wrote:
> Our corporate infrastructure only allows CIFS shares.  No FTP access.
> No scp access.  Nope.  Gotta be CIFS.  One of these is on the other
> side of the pond.  The pipe is fairly fat, but very, very long.
> Sometimes, like yesterday, after 8 hours of copying, I was only
> 80% complete and I had to shut down and go home, other times, the
> operation has been interrupted.  So I am proposing two options
> for "cp":  --resume and --parallel
> I'll let you guess what --resume does, --parallel would open the
> source file several times and have multiple threads reading and
> writing different parts of the file.
> 
> I can either add this to cp, or roll my own toy.  What say you?

Yes, cp is bad for high latency links as it has a
32KiB buffer which it serially reads to and writes from.

Ideally you would have a process on each side of the link,
streaming to each other like:

  ssh -C remote tar -c . | tar -xp

Your suggestion of parallelizing the reads on the local
host might help, especially when the processing of each
block takes time. I.E. it would be more beneficial for
gzip than for cp. You could get much the equivalent parallelization
for cp by using tar to continuously read and buffer like:

  tar -c /cifs/ | (cd dest && tar -xp)

This would be better for the local system,
as you would have a single process writing to
the dest, rather than multiple writers competing
for that resource.

Note the latency/throughput of the above, is dictated by
the pipe buffer in the kernel which is 1M on my system
(/proc/sys/fs/pipe-max-size).

You might get more control over the that,
as well as some feedback by putting something
like `pv` between the two tars.

cheers,
Pádraig.