Re: [Duplicity-talk] Scalability of duplicity

[Mikko Ohtamaa]

Hi,

Sorry about the scant environment details. I'm running duplicity 0.7.06 on Ubuntu 16.04.3 with 63GB of RAM and 12TB of disk space. The lscpu command says it has 48 CPUs. Big honkin' server. The command 'gpg --version' displays gpg (GnuPG) 1.4.20. The destination host where duplicity puts the backup files is running Ubuntu 16.04.3 with 8GB of RAM and 4.9TB of disk space. The lscpu command says it has 4 CPUs. 31 days of backups should need about 2.7TB of storage without compression and encryption.

If there is no CPU activity and Duplicity is still running it might be that the underlying network  protocol has stalled. Are you moving files over SFTP or SCP? If the underlying network hangs and does not properly timeout then it might cause symptoms like this.

You can use strace system command the check what a process is waiting if it does nothing.

Hope this gives clues,

-Mikko

 

Read the duplicity change log down to the 2015-12-07 line that says 'Prep for 0.7.06'. Didn't see anything specifically there (I didn't look at the bug DB) describing a fix for this problem. I could try running the latest duplicity, 0.7.14, to see if the problem goes away.

What version of duplicity are you running?

Tom Ekberg
Senior Computer Specialist, Lab Medicine
University of Washington Medical Center
1959 NE Pacific St, MS 357110
Seattle WA 98195
work: (206) 598-8544
email: address@hidden

On Thu, 5 Oct 2017, Philip Jocks wrote:

Date: Thu, 5 Oct 2017 11:06:31 +0200
From: Philip Jocks <address@hidden>
To: Discussion about duplicity backup <address@hidden>
Cc: Tom Ekberg <address@hidden>
Subject: Re: [Duplicity-talk] Scalability of duplicity

Hej,

Now for my questions:

 Has anyone run duply/duplicity with 90GB of binary files?

yes, we do run it with backup sets >300GB, haven't had any issues with stalling backups in the last few years as far as I remember. Running on FreeBSD, though.
Which version of duplicity are you running?

Cheers,

Philip

_______________________________________________
Duplicity-talk mailing list
address@hidden
https://lists.nongnu.org/mailman/listinfo/duplicity-talk

--

Mikko Ohtamaa
http://opensourcehacker.com
http://twitter.com/moo9000