[Duplicity-talk] sigtar file truncated at 2048Mb

[Johann Haarhoff]

Hi

I am seeing a problem where my 
duplicity-full-signatures.20100729T163548Z.sigtar.gz file gets truncated at 
2048Mb (2147490841 bytes to be exact). I initially thought this was a Samba 
problem, since I am backing up to a CIFS share on the network, but I can 
reproduce it locally as well.

As a result only a small percentage of my files end up in the sigtar file and 
then subsequent backup runs want to add the missing files to the backup again, 
even though they are already present in the volumes.

When I run my backup script, duplicity prints nothing to STDERR, and the log 
file shows no error:

address@hidden:~$ ./backup.sh > /tmp/dupl.log 
address@hidden:~$

address@hidden:~$ tail -28 /tmp/dupl.log  
Deleting /tmp/duplicity-RucH-a-tempdir/mktemp-5O6joj-245
AsyncScheduler: task execution done (success: True)
Processed volume 245
Writing 
/net/kluis/backup/klaasvakie/duplicity_volumes/duplicity-full.20100729T163548Z.vol245.difftar.gz
Deleting /tmp/duplicity-RucH-a-tempdir/mktemp-Qcutoc-246
AsyncScheduler: task execution done (success: True)
Writing 
/net/kluis/backup/klaasvakie/duplicity_volumes/duplicity-full-signatures.20100729T163548Z.sigtar.gz
Writing 
/net/kluis/backup/klaasvakie/duplicity_volumes/duplicity-full.20100729T163548Z.manifest
--------------[ Backup Statistics ]--------------
StartTime 1280421349.61 (Thu Jul 29 18:35:49 2010)
EndTime 1280499112.37 (Fri Jul 30 16:11:52 2010)
ElapsedTime 77762.76 (21 hours 36 minutes 2.76 seconds)
SourceFiles 1213671
SourceFileSize 301803446232 (281 GB)
NewFiles 1213671
NewFileSize 301803446232 (281 GB)
DeletedFiles 0
ChangedFiles 0
ChangedFileSize 0 (0 bytes)
ChangedDeltaSize 0 (0 bytes)
DeltaEntries 1213671
RawDeltaSize 301753880276 (281 GB)
TotalDestinationSizeChange 256386110266 (239 GB)
Errors 0
-------------------------------------------------


I have ~1,200,000 files to backup up, and only ~500,000 are present in my 
sigtar:

address@hidden:~$ find . -iname "*" | wc -l     
1213722
address@hidden:~$ tar tfv 
.cache/duplicity/klaasvakie/duplicity-full-signatures.20100729T163548Z.sigtar.gz
 | wc -l
tar: Unexpected EOF in archive
tar: Error is not recoverable: exiting now
525850
address@hidden:~$


Other relevant details:

address@hidden:~$ duplicity -V    
duplicity 0.6.08b

address@hidden:~$ tar --version
tar (GNU tar) 1.16.1

address@hidden:~$ gzip --version
gzip 1.3.12

address@hidden:~$ ls /var/log/packages/ | grep rsync         
librsync-0.9.7-i386-1
rsync-3.0.4-i486-1

address@hidden:~$ cat ./backup.sh
#!/bin/bash

duplicity \
   --volsize=1000 \
   --name=klaasvakie \
   --tempdir=/tmp \
   -v5 \
   --asynchronous-upload \
   --no-encryption \
   --full-if-older-than=1M \
   --exclude /home/johann/movies \
   /home/johann \
   file:///net/kluis/backup/klaasvakie/duplicity_volumes

address@hidden:~$ ls -al .cache/duplicity/klaasvakie/
total 2099272
drwxr-xr-x 2 johann users        176 2010-07-30 16:34 ./
drwxr-xr-x 3 johann users         80 2010-07-29 18:35 ../
-rw------- 1 johann users 2147490841 2010-07-30 16:34 
duplicity-full-signatures.                                                      
                                                                       
20100729T163548Z.sigtar.gz
-rw------- 1 johann users      65080 2010-07-30 16:12 
duplicity-full.20100729T16                                                      
                                                                       
3548Z.manifest

address@hidden:~$ ls -al /net/kluis/backup/klaasvakie/duplicity_volumes/*sig*
-rw-rw-r-- 1 root users 2147490841 2010-07-30 16:16 
/net/kluis/backup/klaasvakie/duplicity_volumes/duplicity-full-signatures.20100729T163548Z.sigtar.gz

Has anyone seen this before? Any suggestions to try and find the problem?

Thanks in advance,

Johann