Hi Jakob,
There are some countermeasures in duplicity to help at least with some issues. Tuning
"--backend-retry-delay" "--num-retries" "--concurrency" to recover from
failures and speed up.
But I think in your case you should review your setup.
Duplicity is a file backup, but not a file sync solution. Even if duplicity is
able to recover from all interruptions while running one backup over 3 weeks,
it could cause inconsistencies, because duplicity is not doing a snapshot when
it starts but reading each file at the time when it is placed in a volume.
Means if there are file changes while the backup is running and files depend on
each other, they may not align to each other.
You should consider a different setup like:
Do a backup to a local drive. Could be something like an USB3 disk, which is
fast enough to complete timely. In a 2nd step sync the content of the USB disk
to s3. There are tools out for this. So each volume can be transferred/retried
separately, which should be a good granularity. Even if USB disks are know to
be unreliable, having a local copy helps to speed up restore if the disk
survived the disaster.
An other option is to split backups into parts. E.g. I have a huge photo
collection and create a dedicated backup per year, as photos of each year are
stored in a separate folder. This allows to restart those backups more
granularly. I also use the option `--skip-if-no-change` as most of the folders
doesn't change or very rarely. As there is not advantage to do another full
backup if nothing has changed, is saves a lot of unnecessary s3 transfers/costs.
I have created a wrapper for duplicity
https://github.com/poggenpower/duplicity-backup/ to support this use case if
separate backups per folder. It will also trigger full backups based on the
amount of incremental backup made and not by time (it is still on my task list
to implement this option in duplicity itself).
Sorry no easy button ‾\_(ツ)_/‾
Bye
Thomas
On 15 August 2024 07:02:10 CEST, Jakob Bohm via Duplicity-talk
<duplicity-talk@nongnu.org> wrote:
Dear Duplicity community. As I am backing up lots of data with Duplicity, I
find that the biggest backup volume takes over 3 weeks to do a full backup
(starting a new chain), seemingly bottlenecked by the processing speed of AWS
storage or the speed of the local router. This creates a special problem: If
anything interrupts this 3 week long upload process (Such as: Router failure,
AWS outage, process interrupt or local machine crash), there is no clear way to
continue the backup without starting over the entire backup . It would be
really nice if there was a clear way to make the backup continue where it left
off, as long as at least some of the local Duplicity state directories survive
the interruption. Enjoy Jakob
_______________________________________________
Duplicity-talk mailing list
Duplicity-talk@nongnu.org
https://lists.nongnu.org/mailman/listinfo/duplicity-talk