[Duplicity-talk] Re: are periodic full backups necessary?

[Adam Megacz]

Peter Schuller <address@hidden> writes:
> rdiff-backup can do what it does efficiently because it is running
> on boths ends of the pipe. It does not need to transfer the entire
> file (n - 1, nor n) in either direction thanks to its use of the
> rsync algorithm.

I wonder if there's some way to do this using a different approach.
For example, consider how "git" works when used on top of an encrypted
filesystem.  The actual storage could be remote, and one would not
need to trust the storage service.

Because there is a level of indirection between the directory tree and
the actual blocks ("blobs" in git terms) of data which comprise the
files, it's possible to replace "literal bytes XYZ" with "diff against
other block ABC" in yesterday's directory tree.  I believe git calls
this operation "packing".

I think the operations you'd need to be able to perform on the store
are:

  1. Upload new encrypted blob plus encrypted signature
  2. Upload encrypted directory tree (contains pointers to blobs)
  3. Download encrypted signature for blob
  4. Download portion of encrypted directory tree
  5. Replace subtree of encrypted directory tree (ie re-pack)
  6. (Garbage collection) delete encrypted blob or tree

The problem with literally using git for this is that (among other
problems) it can't cope with multi-gigabyte files.  I also don't know
if it's binary diffing facilities are as good as rdiff's (I doubt it).

But I think the general approach has merit.

  - a