Re: [Duplicity-talk] duplicity and partial rsync transfers

[Eliot Moss]

On 8/31/2011 8:04 AM, address@hidden wrote:
> maybe we should add a --rsync-options switch for advanced users?

That's one angle -- potentially dangerous, but advanced users proceed
at their own risk, eh?

I tried a quick and dirty insertion of --partial-dir=.rsync-partial in
the rsync back-end command (self.cmd).  This adds the flag in all cases,
which may not make sense for cases other than get/put.

It seems to work (i.e., does not break duplicity) and to help *sometimes*.
Let me lay out when it helps.  Let us call the case where duplicity
stops and the user needs to re-run duplicity a 'restart'. And let us call
the case where the back-end transmission is interrupted, duplicity is
still running, and it tries the back end again a (back-end) 'retry'.

In the restart case, --partial-dir will be less helpful since the
encrypted volume file from the new run of duplicity may differ from
the volume file that was partially sent.  However, if they share a
significant common prefix, which (assuming I understand encryption
right) can happen if the cleartext files have a significant common
prefix, then it may be of some use.

In the retry case, --partial-dir can be very helpful. However, I
have found that it is possible for the sending side to see the
connection drop, and start a retry, while the receiving side has
not yet noticed the drop. (Different operating systems, etc., may
take longer to notice and report a connection drop.) In that case,
the old receiving rsync has not stashed the partially transmitted
file into the --partial-dir directory yet, so the new transfer
starts either from scratch, or in the case of multiple failures,
some previous --partial-dir version of the file.

So, --partial-dir helps for retry, provided the receiving remote
has terminated. I am not sure if there is any simple way to
terminate a previous attempt (to force the partial file into
the --partial-dir directory), so this may be the best we can do.
Still, for large files, which are more likely to be interrupted,
it seems to be of some use, and easy to apply.

If it is desired to make it optional, we might do so via an
environment variable (say, PARTIAL_DIR). Now rsync already has
an environment variable it looks for, for specifying the
directory, but setting that environment variable does not by
itself enable the partial transfer feature. So my suggestion
would be to use a different variable to control whether
duplicity will do this in the rsync back end, and to have the
rsync back end python logic perhaps use the rsync environment
variable, if it is set, or supply the default .rsync-partial
if it is not.

I'm not much of a python hacker, but I could try to develop
a patch for your consideration and tidying ...

Thoughts?    Eliot