Re: [Duplicity-talk] Simple full backup verify error

[Felix Natter]

hello Edgar,

thank you for your help.

It works fine with the nfs 'sync' mount option; it seems checksums on NFS shares
are not reliable because something is cached.

Maybe it would be good idea to mention this in the man page, in the
"Local file path" section: "Make sure to use the 'sync' mount option if your
local destination is on an nfs share"? Or maybe in a "Troubleshooting"
section or in https://duplicity.us/FAQ.html?

Many Thanks and Best Regards,

Felix

On 24.01.23 17:38, edgar.soldin--- via Duplicity-talk wrote:

> hey Felix,
>       
>       
>       see below
>       
>       
>       On 24.01.2023 17:16, Felix Natter via Duplicity-talk wrote:
>       
> > hello Edgar,
> >         
> >         
> >         thank you for your reply! My answers are inline.
> >         
> >         
> >         On 24.01.23 11:51, edgar.soldin--- via Duplicity-talk wrote:
> >         
> > > hey Felix,
> > >           
> > >           
> > >           see inline
> > >           
> > >           
> > >           On 24.01.2023 11:19, Felix Natter via Duplicity-talk wrote:
> > >           
> > > > hello Duplicity community,
> > > >             
> > > >             
> > > >             I am running a simple daily backup (in this case: just one
> > > >             full backup with just one volume):
> > > >             
> > > >             
> > > >             export DUPLICITY_OPTS="--verbosity=debug
> > > >             --tempdir=/repos/temp --archive-dir=/repos/.duplicity-cache
> > > >             --volsize=28000 --no-compression --encrypt-key MY_KEY"
> > > >             
> > > >             
> > > >             duplicity $DUPLICITY_OPTS --full-if-older-than 5D
> > > >             /repos/dumps.backup/ file:///mnt/FOO/daily/ |& tee
> > > >             /tmp/duplicity.log
> > > >             
> > > >           
> > >           
> > >           not sure that you need debug verbosity here already
> > >           
> > >         
> >         
> >         do I need to debug verify only or also fullbackup?
> >         
> >       
>       
>       i don't see any reason for you to raise log verbosity at all, as
>       duplicity seems to work fine and just reports actual problems with
>       your backup files.
>       
>       
> > >           
> > > > which succeeds. Immediately
> > > >             afterwards, I run a verify including compare:
> > > >             
> > > >             
> > > >             duplicity verify $DUPLICITY_OPTS --compare-data
> > > >             file:///mnt/FOO/daily/ /repos/dumps.backup/ |& tee
> > > >             /tmp/duplicity.log
> > > >             
> > > >             
> > > >             which fails with many of these:
> > > >             
> > > >             
> > > >             Registering (mktemp) temporary file
> > > >             /repos/temp/duplicity-t168pkjr-tempdir/mktemp-d9mbcjxz-1
> > > >             
> > > >             Invalid data - SHA1 hash mismatch for file:
> > > >             
> > > >               duplicity-full.20230124T082316Z.vol1.difftar.gpg
> > > >             
> > > >               Calculated hash: ccab08e9879d9598b83945c83233a9978d954c67
> > > >             
> > > >               Manifest hash: cf642123945b4f86b376815c8b8f2ce92e0f2688
> > > >             
> > > >             
> > > >             Registering (mktemp) temporary file
> > > >             /repos/temp/duplicity-t168pkjr-tempdir/mktemp-gz1sto53-2
> > > >             
> > > >             Invalid data - SHA1 hash mismatch for file:
> > > >             
> > > >               duplicity-full.20230124T082316Z.vol1.difftar.gpg
> > > >             
> > > >               Calculated hash: ccab08e9879d9598b83945c83233a9978d954c67
> > > >             
> > > >               Manifest hash: cf642123945b4f86b376815c8b8f2ce92e0f2688
> > > >             
> > > >             
> > > >             [...]
> > > >             
> > > >           
> > >           
> > >           duplicity creates checksums during backup and these do not
> > >           match anymore, vulgo they have been corrupted.
> > >           
> > >           
> > > > /mnt/FOO is an NFS mounted share. The
> > > >             target directory and cache directory are fresh
> > > >             
> > > >             (just deleted before the backup job). I haven't deleted the
> > > >             temp folder though.
> > > >             
> > > >             I am using the latest duplicity: 1.2.1.
> > > >             
> > > >             
> > > >             Here is more log output that occurs before the above:
> > > >             
> > > >             
> > > >             3 file(s) exists on backend
> > > >             
> > > >             3 file(s) exists in cache
> > > >             
> > > >             Extracting backup chains from list of files:
> > > >             ['duplicity-full.20230124T082316Z.vol1.difftar.gpg',
> > > >             'duplicity-full-signatures.20230124T082316Z.sigtar.gpg',
> > > >             'duplicity-full.20230124T082316Z.manifest.gpg']
> > > >             
> > > >             File duplicity-full.20230124T082316Z.vol1.difftar.gpg is not
> > > >             part of a known set; creating new set
> > > >             
> > > >             File duplicity-full-signatures.20230124T082316Z.sigtar.gpg
> > > >             is not part of a known set; creating new set
> > > >             
> > > >             Ignoring file (rejected by backup set)
> > > >             'duplicity-full-signatures.20230124T082316Z.sigtar.gpg'
> > > >             
> > > >             Processing local manifest
> > > > b'/repos/.duplicity-cache/956a308cc4cd0300f1ebbd0e0b7865c1/duplicity-full.20230124T082316Z.manifest'
> > > >             (4835)
> > > >             
> > > >             Found manifest volume 1
> > > >             
> > > >             Found 1 volumes in manifest
> > > >             
> > > >             File duplicity-full.20230124T082316Z.manifest.gpg is part of
> > > >             known set
> > > >             
> > > >             Found backup chain [Tue Jan 24 09:23:16 2023]-[Tue Jan 24
> > > >             09:23:16 2023]
> > > >             
> > > >             Local and Remote metadata are synchronized, no sync needed.
> > > >             
> > > >             Last full backup date: Tue Jan 24 09:23:16 2023
> > > >             
> > > >             Collection Status
> > > >             
> > > >             -----------------
> > > >             
> > > >             Connecting with backend: BackendWrapper
> > > >             
> > > >             Archive dir:
> > > >             /repos/.duplicity-cache/956a308cc4cd0300f1ebbd0e0b7865c1
> > > >             
> > > >             
> > > >             Found 0 secondary backup chain(s).
> > > >             
> > > >             
> > > >             Found primary backup chain with matching signature chain:
> > > >             
> > > >             -------------------------
> > > >             
> > > >             Chain start time: Tue Jan 24 09:23:16 2023
> > > >             
> > > >             Chain end time: Tue Jan 24 09:23:16 2023
> > > >             
> > > >             Number of contained backup sets: 1
> > > >             
> > > >             Total number of contained volumes: 1
> > > >             
> > > >               Type of backup set:                            Time:     
> > > >             Num volumes:
> > > >             
> > > >                              Full         Tue Jan 24 09:23:16
> > > >             2023                 1
> > > >             
> > > >             -------------------------
> > > >             
> > > >             No orphaned or incomplete backup sets found.
> > > >             
> > > >             Processing local manifest
> > > > b'/repos/.duplicity-cache/956a308cc4cd0300f1ebbd0e0b7865c1/duplicity-full.20230124T082316Z.manifest'
> > > >             (4835)
> > > >             
> > > >             Found manifest volume 1
> > > >             
> > > >             Found 1 volumes in manifest
> > > >             
> > > >             Processing local manifest
> > > > b'/repos/.duplicity-cache/956a308cc4cd0300f1ebbd0e0b7865c1/duplicity-full.20230124T082316Z.manifest'
> > > >             (4835)
> > > >             
> > > >             Found manifest volume 1
> > > >             
> > > >             Found 1 volumes in manifest
> > > >             
> > > >             Using temporary directory
> > > >             /repos/temp/duplicity-t168pkjr-tempdir
> > > >             
> > > >             Registering (mktemp) temporary file
> > > >             /repos/temp/duplicity-t168pkjr-tempdir/mktemp-d9mbcjxz-1
> > > >             
> > > >             
> > > >             So what am I missing?
> > > >             
> > > >           
> > >           
> > >           just tried locally with another key but essentially your
> > >           DUPLICITY_OPTS and duplicity 1.2.1 verified fine.
> > >           
> > >         
> >         We are using a total size of 22Gb, did you try with large files?
> >         
> >       
>       
>       no. i would if you can reproduce your issue with a local file
>       system target.
>       
>       
> > >           
> > >           how about you backup/verify to/from a local file:// target
> > >           first (e.g. /tmp/test.backup/) and work your way up from
> > >           there. this way you make sure everything is working and then
> > >           you can start debugging the NFS issues.
> > >           
> > >         
> >         
> >         Locally everything is ok! Can I add more than --verbosity=debug
> >         logging? Do the NFS mount
> >         
> >       
>       
>       well, then it is settled. not a duplicity issue. would surprise me
>       as well, as we have quite a user base and an issue like that would
>       likely have been reported by the lot of them.
>       
>       
> > parameters make a difference? The problem
> >         does not occur every time, i.e. sometimes it works.
> >         
> >         
> >         sidact-nas2:/volume1/export/Backup/svn    /mnt/NAS2-SVN    nfs
> >         rw,vers=3,soft,bg,intr    0    0
> >         
> >         
> >         I am now trying with the 'sync' nfs mount option.
> >         
> >       
>       
>       sorry, can't help you there. my nfs experience is limited. how
>       about switching protocols and use sftp:// or something else,
>       whatever your NAS supports as a workaround? check the duplicity
>       man page for supported backends.
>       http://duplicity.us/stable/duplicity.1.html#url-format
>       
>       
>       good luck.. ede/duply.net
>       
>       
>       _______________________________________________
>       
>       Duplicity-talk mailing list
>       
>       Duplicity-talk@nongnu.org
>       
>       https://lists.nongnu.org/mailman/listinfo/duplicity-talk
>       
>

--

SIDACT GmbH
Simulation Data Analysis and
Compression Technologies

Felix Natter
Software Developer

Auguststraße 29
53229 Bonn
Germany

Phone	:   +49 228 5348 0430
Direct	:   +49 228 4097 7118
Email	:   felix.natter@sidact.com
Web	:   http://www.sidact.com/

OpenPGP_signature
Description: OpenPGP digital signature