[rdiff-backup-users] Verify times increasing

[Daniel Miller]

Hi,

I've been using rdiff-backup to backup my server (~100 GB of data)  
since June of this year. I'm having a problem with the time needed to  
verify increasing in a seemingly uncontrollable manner. The setup is  
as follows:

Description                  original time (approx) / current time  
(approx) in hours

rdiff-backup server to a local firewire drive             3.8 / 4
rdiff-backup --verify-at-time 3D (local firewire drive)*  1.2 / 2.5
rsync local firewire drive to remote system               1.0 / 2.5
rdiff-backup --verify-at-time 1Y (run by remote system)** 2.3 / 9+

total backup time                                         8+  / 18+

* changed from 1Y to 7D in Aug. 2009 and then to 3D in Oct. 2009  
(these changes had a noticeable affect; see attached graph)
** changed from 1Y to 3M in Oct. 2009 due to the alarming rate of  
increase (this change had little to no affect on total verify time)

Notes:
- The original backup size was ~88 GB of data and there were  
(obviously) no deltas when I started.
- The current backup size is ~100 GB of data and there are now 77 GB  
of delta data (rdiff-backup is very space efficient, I might add).
- The first verify runs on the server and verifies a firewire drive  
attached locally to the server.
- The second verify runs on the remote system and verifies the data on  
an external hard disk attached to the remote system via USB. The drive  
is switched out each day. All drives used at the remote system are 1TB  
USB 2.0 drives (they are all the same make/model).

Ideology: I do the large verify every day on the remote system to make  
sure my backup history is not becoming corrupt (e.g. due to disk  
failure, etc.). Ideally I would like to verify the past year, but that  
will obviously take way too long to be possible with my setup.

Observations:
Despite reducing the amount of historical data that gets verified from  
one year to three months, the steadily increasing amount of time  
needed for the verify has not abated (i.e. it is still increasing at  
near the same rate as before). What would cause such a thing? The  
remote system which does the large verify is CPU bound during the  
verify, so it is not a disk performance problem as far as I can tell.  
I am surprised that the verify time did not level off when I reduced  
the amount of data being verified. I actually expected a decrease in  
time given that when I made the change from 1Y to 3M it had been  
verifying about 4 months of data. No such luck!

I have attached a graph of the backup times since I started using  
rdiff-backup in June. The blue, green, yellow and orange lines  
correspond to Server Backup, Server Verify, Rsync and Remote Verify  
respectively, and their times (in hours) are marked on the left scale  
of the graph. The maroon line represents the total time of all backup  
and verify tasks, and its time values (in hours) are on the right scale.

I really need to get the remote verify time under control. At the  
current rate it will soon be overlapping with the start of the next  
backup (a drop-dead and reconfigure scenario). I would like to verify  
as much data as possible to be sure that my historical backups are not  
becoming corrupt.

Recommendations and insite on what may be going wrong are much  
appreciated.

~ Daniel