[Duplicity-talk] Large amount of small changing files

[Remy van Elst]

I have a few servers with a huge amount (+4 million) small (< 5 MB)
.TIFF files which change weekly. (custom geo-data mapping application)

I want to backup this to an Openstack swift objectstore. The commandline
I use is:

duplicity --asynchronous-upload --verbosity 5 --log-file
/var/log/duplicity.log --volsize 250 --tempdir="/root/tmp"
--file-prefix="fs2." --name="fs2." --exclude-device-files
--exclude-globbing-filelist=/etc/duplcity/exclude.conf
--full-if-older-than="14D" --no-encryption  / swift://fs2-backup

Duplicity version is 0.7, however 0.6.24 also had this.

The files change while the backup is running.

The backup takes multiple days to complete, it varies between 7 and 13.
The chain never is complete so only full backups are made. The swift
container does have data, albeit a huge signatures file. Sometimes the
backup breaks because the signatures file is > 5 GB (swift limit), but I
also have broken backup chains with a 700 MB signature file.

Broken means that restoring out of the backup always fails.

LVM snapshots are not possible on this machine, since some idiot has
chosen to install them with one big ext4 partition without lvm.

I want to have a chain of 14 dayly incremental backups and 1 full
backup, saving 4 full backups. The cronjob runs at 4AM in the morning.

What can I do to get regular, working duplicity backup in this situation?

0x1B7F88DC.asc
Description: application/pgp-keys

smime.p7s
Description: S/MIME Cryptographic Signature