Re: [Duplicity-talk] Why do I have a 19GB ~/.cache/duplicity folder?

[Kenneth Loafman]

Tim,

Good testing!

Please report this bug at https://bugs.launchpad.net/duplicity

...Thanks,
...Ken

Tim Riemenschneider wrote:
> Timothee Besset schrieb:
>> I'm finding that some systems that were backed up by duplicity are
>> running out of disk space after I upgraded the duplicity version. More
>> specifically, I am seeing 19GB worth of data in a ~/.cache/duplicity folder?
>>
>> I can't afford to have 20GB of disk occupied by my backup software. I
>> haven't found much information on the nature of that directory, except
>> that it's an addition in 0.6? Should it really be growing that big? Is
>> there a limit to how much it'll grow? Is it a case of backup biting it's
>> own tail like we had backups of /tmp trying to backup duplicity's own
>> temporary files at one point?
>>
>>   
> The problem seems to be, that duplicity does not cleanup signature-files
> when removing old backups, so the signatures occupy more and more space.
> 
> To reproduce, I did some backups (locally), some with a forced full
> backup and some incementals. Afterwards the backup looks like this:
> 
> $ PASSPHRASE="foo" duplicity collection-status file:///tmp/duptest
> Local and Remote metadata are synchronized, no sync needed.
> Last full backup date: Wed Feb 10 16:51:46 2010
> Collection Status
> -----------------
> Connecting with backend: LocalBackend
> Archive dir: /home/tim/.cache/duplicity/dfd6cdfea9f4dc13e6e57c08d55d2a93
> 
> Found 2 secondary backup chains.
> Secondary chain 1 of 2:
> -------------------------
> Chain start time: Wed Feb 10 16:50:16 2010
> Chain end time: Wed Feb 10 16:50:44 2010
> Number of contained backup sets: 4
> Total number of contained volumes: 4
>  Type of backup set:                            Time:      Num volumes:
>                 Full         Wed Feb 10 16:50:16 2010                 1
>          Incremental         Wed Feb 10 16:50:36 2010                 1
>          Incremental         Wed Feb 10 16:50:41 2010                 1
>          Incremental         Wed Feb 10 16:50:44 2010                 1
> -------------------------
> 
> Secondary chain 2 of 2:
> -------------------------
> Chain start time: Wed Feb 10 16:51:01 2010
> Chain end time: Wed Feb 10 16:51:39 2010
> Number of contained backup sets: 4
> Total number of contained volumes: 4
>  Type of backup set:                            Time:      Num volumes:
>                 Full         Wed Feb 10 16:51:01 2010                 1
>          Incremental         Wed Feb 10 16:51:36 2010                 1
>          Incremental         Wed Feb 10 16:51:38 2010                 1
>          Incremental         Wed Feb 10 16:51:39 2010                 1
> -------------------------
> 
> 
> Found primary backup chain with matching signature chain:
> -------------------------
> Chain start time: Wed Feb 10 16:51:46 2010
> Chain end time: Wed Feb 10 16:52:04 2010
> Number of contained backup sets: 2
> Total number of contained volumes: 2
>  Type of backup set:                            Time:      Num volumes:
>                 Full         Wed Feb 10 16:51:46 2010                 1
>          Incremental         Wed Feb 10 16:52:04 2010                 1
> -------------------------
> No orphaned or incomplete backup sets found.
> $ ls /tmp/duptest/
> duplicity-full.20100210T155016Z.manifest.gpg
> duplicity-full.20100210T155016Z.vol1.difftar.gpg
> duplicity-full.20100210T155101Z.manifest.gpg
> duplicity-full.20100210T155101Z.vol1.difftar.gpg
> duplicity-full.20100210T155146Z.manifest.gpg
> duplicity-full.20100210T155146Z.vol1.difftar.gpg
> duplicity-full-signatures.20100210T155016Z.sigtar.gpg
> duplicity-full-signatures.20100210T155101Z.sigtar.gpg
> duplicity-full-signatures.20100210T155146Z.sigtar.gpg
> duplicity-inc.20100210T155016Z.to.20100210T155036Z.manifest.gpg
> duplicity-inc.20100210T155016Z.to.20100210T155036Z.vol1.difftar.gpg
> duplicity-inc.20100210T155036Z.to.20100210T155041Z.manifest.gpg
> duplicity-inc.20100210T155036Z.to.20100210T155041Z.vol1.difftar.gpg
> duplicity-inc.20100210T155041Z.to.20100210T155044Z.manifest.gpg
> duplicity-inc.20100210T155041Z.to.20100210T155044Z.vol1.difftar.gpg
> duplicity-inc.20100210T155101Z.to.20100210T155136Z.manifest.gpg
> duplicity-inc.20100210T155101Z.to.20100210T155136Z.vol1.difftar.gpg
> duplicity-inc.20100210T155136Z.to.20100210T155138Z.manifest.gpg
> duplicity-inc.20100210T155136Z.to.20100210T155138Z.vol1.difftar.gpg
> duplicity-inc.20100210T155138Z.to.20100210T155139Z.manifest.gpg
> duplicity-inc.20100210T155138Z.to.20100210T155139Z.vol1.difftar.gpg
> duplicity-inc.20100210T155146Z.to.20100210T155204Z.manifest.gpg
> duplicity-inc.20100210T155146Z.to.20100210T155204Z.vol1.difftar.gpg
> duplicity-new-signatures.20100210T155016Z.to.20100210T155036Z.sigtar.gpg
> duplicity-new-signatures.20100210T155036Z.to.20100210T155041Z.sigtar.gpg
> duplicity-new-signatures.20100210T155041Z.to.20100210T155044Z.sigtar.gpg
> duplicity-new-signatures.20100210T155101Z.to.20100210T155136Z.sigtar.gpg
> duplicity-new-signatures.20100210T155136Z.to.20100210T155138Z.sigtar.gpg
> duplicity-new-signatures.20100210T155138Z.to.20100210T155139Z.sigtar.gpg
> duplicity-new-signatures.20100210T155146Z.to.20100210T155204Z.sigtar.gpg
> 
> Now I remove the old backups:
> 
> $ PASSPHRASE="foo" duplicity remove-all-but-n-full 1 --force
> file:///tmp/duptest
> Local and Remote metadata are synchronized, no sync needed.
> Last full backup date: Wed Feb 10 16:51:46 2010
> Deleting backup sets at times:
> Wed Feb 10 16:50:16 2010
> Wed Feb 10 16:50:36 2010
> Wed Feb 10 16:50:41 2010
> Wed Feb 10 16:50:44 2010
> Wed Feb 10 16:51:01 2010
> Wed Feb 10 16:51:36 2010
> Wed Feb 10 16:51:38 2010
> Wed Feb 10 16:51:39 2010
> Warning, found the following local orphaned signature files:
> duplicity-new-signatures.20100210T155016Z.to.20100210T155036Z.sigtar.gz
> duplicity-new-signatures.20100210T155036Z.to.20100210T155041Z.sigtar.gz
> duplicity-new-signatures.20100210T155041Z.to.20100210T155044Z.sigtar.gz
> duplicity-new-signatures.20100210T155101Z.to.20100210T155136Z.sigtar.gz
> duplicity-new-signatures.20100210T155136Z.to.20100210T155138Z.sigtar.gz
> duplicity-new-signatures.20100210T155138Z.to.20100210T155139Z.sigtar.gz
> 
> Note the "locally orphaned files".
> When I look at the status:
> 
> $ PASSPHRASE="foo" duplicity collection-status file:///tmp/duptest
> Synchronizing remote metadata to local cache...
> Copying duplicity-full-signatures.20100210T155016Z.sigtar to local cache.
> Copying duplicity-full-signatures.20100210T155101Z.sigtar to local cache.
> Last full backup date: Wed Feb 10 16:51:46 2010
> Collection Status
> -----------------
> Connecting with backend: LocalBackend
> Archive dir: /home/tim/.cache/duplicity/dfd6cdfea9f4dc13e6e57c08d55d2a93
> 
> Found 0 secondary backup chains.
> 
> Found primary backup chain with matching signature chain:
> -------------------------
> Chain start time: Wed Feb 10 16:51:46 2010
> Chain end time: Wed Feb 10 16:52:04 2010
> Number of contained backup sets: 2
> Total number of contained volumes: 2
>  Type of backup set:                            Time:      Num volumes:
>                 Full         Wed Feb 10 16:51:46 2010                 1
>          Incremental         Wed Feb 10 16:52:04 2010                 1
> -------------------------
> No orphaned or incomplete backup sets found.
> 
> Note that duplicity did fetch the signatures of the 2 old (deleted!)
> full-backups
> 
> When looking at the target, it becomes clear that "duplicity remove..."
> did only remove the data (*vol*difftar-files), not the obsoleted signatures:
> 
> $ ls /tmp/duptest/
> duplicity-full.20100210T155146Z.manifest.gpg
> duplicity-full.20100210T155146Z.vol1.difftar.gpg
> duplicity-full-signatures.20100210T155016Z.sigtar.gpg
> duplicity-full-signatures.20100210T155101Z.sigtar.gpg
> duplicity-full-signatures.20100210T155146Z.sigtar.gpg
> duplicity-inc.20100210T155146Z.to.20100210T155204Z.manifest.gpg
> duplicity-inc.20100210T155146Z.to.20100210T155204Z.vol1.difftar.gpg
> duplicity-new-signatures.20100210T155016Z.to.20100210T155036Z.sigtar.gpg
> duplicity-new-signatures.20100210T155036Z.to.20100210T155041Z.sigtar.gpg
> duplicity-new-signatures.20100210T155041Z.to.20100210T155044Z.sigtar.gpg
> duplicity-new-signatures.20100210T155101Z.to.20100210T155136Z.sigtar.gpg
> duplicity-new-signatures.20100210T155136Z.to.20100210T155138Z.sigtar.gpg
> duplicity-new-signatures.20100210T155138Z.to.20100210T155139Z.sigtar.gpg
> duplicity-new-signatures.20100210T155146Z.to.20100210T155204Z.sigtar.gpg
> 
> These files are also present (in decrypted form) in the cache-dir. And
> they stay there (or are refetched from the backup-target).
> Only workaround: delete the unneeded signature-files on the target, then
> duplicity also removes them from the cache (but be carefull to only
> remove old and unneeded files):
> $ cd /tmp/duptest/
> :/tmp/duptest$ rm duplicity-full-signatures.20100210T155016Z.sigtar.gpg
> duplicity-full-signatures.20100210T155101Z.sigtar.gpg
> duplicity-new-signatures.20100210T1550*
> duplicity-new-signatures.20100210T155101Z.to.20100210T155136Z.sigtar.gpg
> duplicity-new-signatures.20100210T15513*
> :/tmp/duptest$ ls
> duplicity-full.20100210T155146Z.manifest.gpg
> duplicity-full.20100210T155146Z.vol1.difftar.gpg
> duplicity-full-signatures.20100210T155146Z.sigtar.gpg
> duplicity-inc.20100210T155146Z.to.20100210T155204Z.manifest.gpg
> duplicity-inc.20100210T155146Z.to.20100210T155204Z.vol1.difftar.gpg
> duplicity-new-signatures.20100210T155146Z.to.20100210T155204Z.sigtar.gpg
> :/tmp/duptest$ cd -
> ~$ PASSPHRASE="foo" duplicity collection-status file:///tmp/duptest
> Synchronizing remote metadata to local cache...
> Deleting local
> /home/tim/.cache/duplicity/dfd6cdfea9f4dc13e6e57c08d55d2a93/duplicity-full-signatures.20100210T155016Z.sigtar.gz
> (not authoritative at backend).
> Deleting local
> /home/tim/.cache/duplicity/dfd6cdfea9f4dc13e6e57c08d55d2a93/duplicity-full-signatures.20100210T155101Z.sigtar.gz
> (not authoritative at backend).
> Deleting local
> /home/tim/.cache/duplicity/dfd6cdfea9f4dc13e6e57c08d55d2a93/duplicity-new-signatures.20100210T155016Z.to.20100210T155036Z.sigtar.gz
> (not authoritative at backend).
> Deleting local
> /home/tim/.cache/duplicity/dfd6cdfea9f4dc13e6e57c08d55d2a93/duplicity-new-signatures.20100210T155036Z.to.20100210T155041Z.sigtar.gz
> (not authoritative at backend).
> Deleting local
> /home/tim/.cache/duplicity/dfd6cdfea9f4dc13e6e57c08d55d2a93/duplicity-new-signatures.20100210T155041Z.to.20100210T155044Z.sigtar.gz
> (not authoritative at backend).
> Deleting local
> /home/tim/.cache/duplicity/dfd6cdfea9f4dc13e6e57c08d55d2a93/duplicity-new-signatures.20100210T155101Z.to.20100210T155136Z.sigtar.gz
> (not authoritative at backend).
> Deleting local
> /home/tim/.cache/duplicity/dfd6cdfea9f4dc13e6e57c08d55d2a93/duplicity-new-signatures.20100210T155136Z.to.20100210T155138Z.sigtar.gz
> (not authoritative at backend).
> Deleting local
> /home/tim/.cache/duplicity/dfd6cdfea9f4dc13e6e57c08d55d2a93/duplicity-new-signatures.20100210T155138Z.to.20100210T155139Z.sigtar.gz
> (not authoritative at backend).
> Last full backup date: Wed Feb 10 16:51:46 2010
> Collection Status
> -----------------
> Connecting with backend: LocalBackend
> Archive dir: /home/tim/.cache/duplicity/dfd6cdfea9f4dc13e6e57c08d55d2a93
> 
> Found 0 secondary backup chains.
> 
> Found primary backup chain with matching signature chain:
> -------------------------
> Chain start time: Wed Feb 10 16:51:46 2010
> Chain end time: Wed Feb 10 16:52:04 2010
> Number of contained backup sets: 2
> Total number of contained volumes: 2
>  Type of backup set:                            Time:      Num volumes:
>                 Full         Wed Feb 10 16:51:46 2010                 1
>          Incremental         Wed Feb 10 16:52:04 2010                 1
> -------------------------
> No orphaned or incomplete backup sets found.
> 
> (Of course, one has to do full-backups from time to time in the first
> place, when you only ever do incrementals, ALL signature-files are still
> needed)
> 
> cu
> Tim
> 
> 
> 
> _______________________________________________
> Duplicity-talk mailing list
> address@hidden
> http://lists.nongnu.org/mailman/listinfo/duplicity-talk
>

signature.asc
Description: OpenPGP digital signature