Re: [rdiff-backup-users] Reporting update bug for large files in rdiff-backup (>4Gb)

[Vadim Kouzmine]

Just in case, repeated it on Gentoo/rdiff-backup-1.0.1/python-2.4.2.

The same here:

python: ERROR: (rs_job_iter) internal error: job made no progress
[orig_in=13, orig_out=65536, final_in=13, final_out=65536]
UpdateError largefile_4096_MB.dat librsync error 107 while in patch
cycle
python: ERROR: (rs_job_iter) internal error: job made no progress
[orig_in=69754, orig_out=65536, final_in=69754, final_out=65536]
UpdateError largefile_5120_MB.dat librsync error 107 while in patch
cycle



On Tue, 2006-02-07 at 15:27 +0100, Gerard van Dijnsen wrote:
> Hi all,
> 
> a while ago I reported seeing librsync error 107 when updating large 
> files. The answer was to use the command rdiff to see if this was indeed 
> rdiff-backup related. At the time I was mighty busy, but now I have 
> taken the time to look into this more thorough and found out that it is 
> indeed a problem in rdiff-backup. I would like to flag this as a bug. 
> The following was carried out with rdiff-backup 1.0.3 with Debian. I 
> know this is not the latest and greatest, but according to the 
> changelist nothing even remotely resembling this problem has been fixed. 
> Here goes:
> 
> To start I created some files in a temporary directory:
> 
> $ mkdir /bigdisk/tmp
> $ cd /bigdisk/tmp
> $ for i in 512 1024 2048 4096 5120; do echo $i; dd if=/dev/urandom 
> of=largefile_${i}_MB.dat bs=1024K count=$i; done
> 
> This creates files with random data sized 512Mb, 1, 2 , 4 and 5 Gb. I 
> proceeded by making a backup of the temporary directory:
> 
> $ cd ..
> $ rdiff-backup tmp/ tmp2
> 
> This takes a while and succeeds. Everything is ok up to this point. The 
> mirror is created, no errors reported, the files are identical in both 
> directories. Now I changed the big files somewhat by adding some bytes:
> 
> $ cd tmp
> $ for i in *.dat; do echo bladiebla >> $i; done
> 
> Now restart the backup process:
> 
> $ cd ..
> $ rdiff-backup tmp/ tmp2
> 
> This again takes a while and then produces the following output:
> python: ERROR: (rs_job_iter) internal error: job made no progress 
> [orig_in=23, orig_out=65536, final_in=23, final_out=65536]
> UpdateError largefile_4096_MB.dat librsync error 107 while in patch cycle
> python: ERROR: (rs_job_iter) internal error: job made no progress 
> [orig_in=17533, orig_out=65536, final_in=17533, final_out=65536]
> UpdateError largefile_5120_MB.dat librsync error 107 while in patch cycle
> 
> sure enough, the files mentioned in the output are no longer in sync:
> 
> $ md5sum tmp/*.dat tmp2/*.dat
> cb79763db0567150a3962299d568ba81  tmp/largefile_1024_MB.dat
> ea52ed724707b474208ff08baabd2020  tmp/largefile_2048_MB.dat
> e2948b4fdbdd140392f166718e19f1fd  tmp/largefile_4096_MB.dat --> *
> 73aec8d4c7d0a57c3ecfe5e6be0755ae  tmp/largefile_5120_MB.dat --> *
> b86db4919db290010b8610b3a234035a  tmp/largefile_512_MB.dat
> cb79763db0567150a3962299d568ba81  tmp2/largefile_1024_MB.dat
> ea52ed724707b474208ff08baabd2020  tmp2/largefile_2048_MB.dat
> 38b463c451c667c91430e82cfb611ed3  tmp2/largefile_4096_MB.dat -->*
> 4f864e56beac1ce384fe66608ee98e85  tmp2/largefile_5120_MB.dat --> *
> b86db4919db290010b8610b3a234035a  tmp2/largefile_512_MB.dat
> 
> I have seen this behaviour on my servers on several occasion, and it 
> seems consistent. Also, this has nothing to do with ssh, because we are 
> making a local backup. However, the behaviour over ssh is exactly the 
> same. Now for good measure I used rdiff to see if this would generate a 
> similar error:
> 
> $ rdiff signature tmp2/largefile_4096_MB.dat signature.dat
> $ rdiff delta signature.dat tmp/largefile_4096_MB.dat delta.dat
> $ rdiff patch tmp2/largefile_4096_MB.dat delta.dat largefile_4096_MB.dat
> rdiff: ERROR: (rs_job_iter) internal error: job made no progress 
> [orig_in=23, orig_out=16000, final_in=23, final_out=16000]
> rdiff: ERROR: library internal error
> 
> It is notable that the error is given immediately after the 3rd command. 
> The output file is 0 bytes long. It may be worth to note that when I 
> tried this for files of the same size, but consisting of all zeros, no 
> problem occured. Perhaps the files are compressed during the rdiff 
> process and this has influence by making the datastream smaller?
> 
> Hopefully you can patch this. I dont have a clue where to start!
> 
> Sincerely
> Gerard van Dijnsen
> 
> 
> 
> 
> 
> 
> 
> 
> _______________________________________________
> rdiff-backup-users mailing list at address@hidden
> http://lists.nongnu.org/mailman/listinfo/rdiff-backup-users
> Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki
-- 
Vadim Kouzmine <address@hidden>