Re: [Duplicity-talk] Recovering from an interruption during backup

[Thomas Laubrock]

Hi Jakob,

There are some countermeasures in duplicity to help at least with some issues. Tuning  "--backend-retry-delay"  "--num-retries" "--concurrency" to recover from failures and speed up.
But I think in your case you should review your setup.
Duplicity is a file backup, but not a file sync solution. Even if duplicity is able to recover from all interruptions while running one backup over 3 weeks, it could cause inconsistencies, because duplicity is not doing a snapshot when it starts but reading each file at the time when it is placed in a volume. Means if there are file changes while the backup is running and files depend on each other, they may not align to each other.

You should consider a different setup like:
Do a backup to a local drive. Could be something like an USB3 disk, which is fast enough to complete timely. In a 2nd step sync the content of the USB disk to s3. There are tools out for this. So each volume can be transferred/retried separately, which should be a good granularity. Even if USB disks are know to be unreliable, having a local copy helps to speed up restore if the disk survived the disaster.

An other option is to split backups into parts. E.g. I have a huge photo collection and create a dedicated backup per year, as photos of each year are stored in a separate folder. This allows to restart those backups more granularly. I also use the option `--skip-if-no-change` as most of the folders doesn't change or very rarely. As there is not advantage to do another full backup if nothing has changed, is saves a lot of unnecessary s3 transfers/costs.

I have created a wrapper for duplicity https://github.com/poggenpower/duplicity-backup/ to support this use case if separate backups per folder. It will also trigger full backups based on the amount of incremental backup made and not by time (it is still on my task list to implement this option in duplicity itself).

 Sorry no easy button ‾\_(ツ)_/‾

Bye

  Thomas

On 15 August 2024 07:02:10 CEST, Jakob Bohm via Duplicity-talk <duplicity-talk@nongnu.org> wrote:

Dear Duplicity community.

As I am backing up lots of data with Duplicity, I find that the biggest
backup volume takes over 3 weeks to do a full backup (starting a new
chain), seemingly bottlenecked by the processing speed of AWS storage
or the speed of the local router.

This creates a special problem: If anything interrupts this 3 week long
upload process (Such as: Router failure, AWS outage, process interrupt
or local machine crash), there is no clear way to continue the backup
without starting over the entire backup .  It would be really nice if
there was a clear way to make the backup continue where it left off,
as long as at least some of the local Duplicity state directories
survive the interruption.

Enjoy

Jakob