Re: Speed and regression

[Eric Lavarde]

Hi,

On 27/09/2024 07:09, Duffer wrote:
> On 2024-09-25 22:30, Duffer wrote:
> > Hi.
> >
> > I'm using rdiff-backup 2.2.6 for a large amount of data. All traffic 
> > is encrypted via stunnel.
> >
> > I have two concerns:
> >
> > 1. Speed
> >
> > With SSH, the speed I've seen is 25-50 Mbps, probably averaging 30-40 
> > Mbps. iperf3 gets around 500 Mbps between my local system 
> > (destination) and the remote system (source). The SSH speed seems 
> > excessively slow to me - maybe 6-8% of the measured bandwidth.
> >
> > I switched to mapping the data source via NFS. Now I'm seeing full 
> > speed for the transfers.
> >
> > Why is SSH so incredibly slow? Is that expected? I'm okay using NFS 
> > but I liked the convenience of SSH.

Well, I can only guess that it's the pickling and unpickling necessary 
over the network. I'm working on improving performance but it takes 
time. I first need to refactor and simplify the code. It could also be 
worsened due to a weak CPU on one end or the other. Many small files 
also increase a lot the overhead.

> > 2. Regression
> >
> > As noted, my backup was very slow so I stopped it (CTRL-C). The next 
> > time I ran the backup, it went into "regress" mode, so I lost all the 
> > progress. Is there no way to avoid this and resume somehow? I couldn't 
> > find anything in the FAQ.

No, but there is already an issue for this. Again, takes time...

> The regress didn't take long at all but it's now been running over 10 
> hours and I've seen little progress. Disk usage hasn't increased at all. 
> Is this expected? I thought after the regress it would start 
> downloading, like it did before. I can't tell what it's doing but I seem 
> to see it reading a lot from the disk. I used "-v 3" but maybe I should 
> have opted for more output.

-v3 is the default value, I generally use -v5 to see progress (you can 
configure differently logfile and console output BTW).
Difficult to say what's happening from here, but did you check if files 
are appearing in your backup repo, especially under rdiff-backup-data?
If everything breaks, assuming Linux, you can try to guess what it's 
doing using strace.

KR, Eric