rdiff-backup-users
[Top][All Lists]
Advanced

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re[6]: [rdiff-backup-users] Verify times increasing


From: listserv . traffic
Subject: Re[6]: [rdiff-backup-users] Verify times increasing
Date: Tue, 24 Nov 2009 14:28:50 -0800

See inline...

>> I'm not sure what you're doing with your --verify...
>>
>> It *sounds* like you want a full CRC style check of the *current*
>> files after the backup is complete. (i.e. File X gets updated with a
>> delta, and you want to verify that file X is the same both on the
>> source and destination locations/drives.)

> Yes, although it's more of an internal consistency check within the  
> rdiff-backup repository itself. I'm looking for a way to quickly  
> verify the integrity my entire rdiff-backup repository.

> In my scenario the repository is synced to an external USB drive that
> gets rotated each day (i.e. each day I put yesterday's drive in  
> storage and bring a different drive out of storage to use for the next
> backup). I use rsync to transfer my rdiff-backup repository (which  
> gets updated daily) to the USB drive. Then I run rdiff-backup --verify-
> at-time to verify that the files on the USB drive are not corrupt. But
> lately this has been taking too long.

> Does that make sense?

Yes, and the USB connection may explain the longish verify times,
since it's somewhat slow, compared to a SATA drive connected directly
to the controller...

But I see that you want to verify the "local" RDiff repository to the
"of-line" one.

Not sure how to do that - I'd guess you could do it with some other
tools - not storing the hashes - just a full compare each time. (How
big is the repository? [I think you said, but I don't recall.]

---
But I'd guess your "local" repository isn't on the same disks as the
data, right?

If so, then it's probably not a huge deal if it takes 20 hours to
check the local repository against the remote. [Though I guess all
that disk channel activity might impact other disk through-put too...]

(Add a controller? Dunno...)

I use a similar system and I don't verify the local repository to the
remote, though perhaps I should. (I trust rsync to make sure they're
the same...since it's not just copying the files - it's doing hash
matches like RDiff...)

> Yesterday I introduced a utility called yafic into my backup scheme.  
> Yafic can do a full-repository verification. This works and it's much
> faster than rdiff-backup's --verify-at-time, but it's complicated to  
> setup and I have to ignore all the changes that happen each day when  
> rdiff-backup updates the repository. It would be nicer to have this  
> kind of verification built-in to rdiff-backup so I wouldn't have to  
> filter out all the new delta and metadata files. rdiff-backup knows  
> which files were added/changed/deleted and would not report those  
> changes like yafic does. With my proposed enhancement, rdiff-backup  
> would only report warnings or errors if any part of the repository  
> became corrupt.

>> If that's the case, I think you already get it. (It's "built-in" to
>> RDiff-backup.)

> This is good to know. Does this happen during the backup, or only  
> during --verify... ? I assume you're talking about something  
> equivalent to --verify-at-time 0B ? Of course, this would only verify
> the current mirror.

Well, it computes a hash for the whole file at the local and remote
ends, and if they don't match, it then sends the delta's to make them
match.

What you end up with in the end is a current file on the remote who's hash 
matches
the local file - by definition. I don't know if it checks the "local"
hash vs remote file again [i.e. a recalc of the remote file hash] at
the end to make sure they match, but that would be a redundant step -
tho, probably a good one.

I don't think it's documented per-se, but it's inherent in the protocol.

[But that's not really what you are aiming for in your check
above...so it sorta doesn't apply...]

BTW, is this on a windows platform? (Curious...) Ah, probably not
since yafic isn't... :)

-Greg

> BTW, is this documented? I'm going to feel stupid if it is, because I
> did not see it when I read the docs (multiple times) for rdiff-backup.

> ~ Daniel



-- 
Best regards,
 listserv                            mailto:address@hidden





reply via email to

[Prev in Thread] Current Thread [Next in Thread]