[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: [rdiff-backup-users] Failing on 11GB file - other large files workin
Re: [rdiff-backup-users] Failing on 11GB file - other large files working
Sun, 20 Jul 2003 19:54:02 -0700
>>>>> "Ty!" == ty <Ty!>
>>>>> wrote the following on Fri, 18 Jul 2003 12:10:26 -0600
Ty!> We've been doing nightly backups for a couple of weeks now
Ty!> without any problems, and then a user throws another kink at
Ty!> us. I've attached a trace below of the entire backup run, but
Ty!> the file it is failing on is an 11GB file, of which about 5GB
Ty!> shows up on the backup mirror.
Ty!> We regularly handle files larger than 2GB, so I thought we were
Ty!> done with the large file problems, and we run with
Ty!> --no-compression as we still have Python 2.2.1 and are limited
Ty!> to 2GB zip files if we don't disable compression.
Ty!> I'm wondering if there may still be a 32 bit offset index that
Ty!> is used when building the delta. Granted that I know very
Ty!> little about the inner workings of rdiff-backup (or librsync),
Ty!> so that's just a guess.
Ty!> Any ideas on how to get large files like this (this file will
Ty!> have very large block-level differences) to get backed up
I was just trying to test this. rdiff-backup worked OK when diffing a
12GB file when there were no changes, but the diff was only about
50MB. Then I tried diffing two totally different (both constructed
from /dev/urandom) 12GB files.
But I had to terminate---extrapolating from my experience it would
take literally a week straight to finish. I'm surprised you let the
process complete before killing it. At least on my system (Athlon
2000, 7200rpm ide HD) whether rdiff-backup can construct an 11GB diff
is immaterial because it would take impractically long.
I never realized how slow rdiff-backup could be in cases like this.
Hopefully it will help for the rdiff block size to be set as a
function of file size. Or it could be my python interface, I will
have to check.
Description: PGP signature