Introducing chop-backup

[Ludovic Courtès]

Ladies & Gentlemen,

I am pleased to announce the ultimate encrypted backup system, modestly
called ‘chop-backup’.  Here’s a demo:

--8<---------------cut here---------------start------------->8---
$ ./utils/chop-backup my-backup guile2/
(() 
"tree_indexer:chk_block_fetcher:chk_index_handle:100:aes256,cbc,sha1:6w3z3fjdysc2v2idyycpowtxktjhwebsfbbtstjlgnrftxw5ceca====,jhdatnqk6hytmwsdilu2vsmwsj6yys2w/4a"
 ("zlib_unzip_filter"))

$ ./utils/chop-backup --list=recursive my-backup \
     '(() 
"tree_indexer:chk_block_fetcher:chk_index_handle:100:aes256,cbc,sha1:6w3z3fjdysc2v2idyycpowtxktjhwebsfbbtstjlgnrftxw5ceca====,jhdatnqk6hytmwsdilu2vsmwsj6yys2w/4a"
 ("zlib_unzip_filter"))'
drwxr-xr-x     4096 Sep 16 18:24     chop
    -rw-r--r--    17553 Sep 15 18:29     choppers.go
    -rw-r--r--     6831 Apr  5 14:32     indexers.scm

    [...]

    -rw-r--r--     8849 Mar 31 22:55     cipher.scm
    -rw-r--r--    20558 Jul 11 22:24     internal.scm
drwxr-xr-x     4096 Sep 15 18:29     srfi
    -rw-r--r--     2348 Dec  8 10:23     srfi-64.scm
    -rw-r--r--    33471 Mar 13 14:53     srfi-64.upstream.scm
    -rw-r--r--     2718 Sep 15 18:29     srfi-64.go
-rw-r--r--    34404 Apr  5 23:22     encrypted-data.gdbm
-rw-r--r--      736 Sep 16 20:38     indexers.log

[...]

$ ./utils/chop-backup --restore my-backup \
  '(() 
"tree_indexer:chk_block_fetcher:chk_index_handle:100:aes256,cbc,sha1:6w3z3fjdysc2v2idyycpowtxktjhwebsfbbtstjlgnrftxw5ceca====,jhdatnqk6hytmwsdilu2vsmwsj6yys2w/4a"
 ("zlib_unzip_filter"))' restore-dir

$ ls restore-dir/
block-indexers.log  chop.go       chop.scm    encrypted-data.gdbm  filters.log  
indexers.log  Makefile.am  objects.log  stores.log   test-suite.log  
test-suite.scm.log
chop/               choppers.log  cipher.log  example.scm          hash.log     
Makefile      Makefile.in  srfi/        streams.log  test-suite.scm
--8<---------------cut here---------------end--------------->8---

The basic functionality is there:

  - Recursive encrypted backup of directories, with cached mapping from
    inode/device to index so that files that haven’t changed since the
    last backup need not be read.  The backup operation returns a
    “tuple”, which is basically a tuple as returned by ‘chop-archiver’
    but augmented with filter info.

  - Directory listing of a snapshot, or of a subdirectory with a
    snapshot.

  - Restoration of a whole file system tree, preserving permission bits
    and symlinks.

  - Restoration of individual files, through ‘--show’.

It will eventually allow the version history of a directory to be
navigated (currently all the versions are kept but the pointer to the
previous version isn’t stored.)

Unlike ‘chop-archiver’, it’s really meant to be
“end-user-oriented”–i.e., it should make the right choices by default.
For instance, it will eventually automatically choose the right storage
pipeline according to the type of file being stored–e.g., anchor-based
chopper + zlib for text files, large fixed-size blocks and no encryption
for JPEGs.  Likewise, it will probably store the latest tuples in the
cache so that they don’t have to be passed on the command-line.

Anyway, I find it pretty cool and would welcome feedback!  :-)

Thanks,
Ludo’.