Re: cross-platform backup tool Unexpectedly Slow on Initial Backup

[ewl+rdiffbackup]

Hi,

just because I became curious, the numbers are probably not to be compared:

- iperf3 tells me ~170Mbit/s
- I transferred initially my Downloads repository from laptop to server 
(which has roughly as much disk as you have RAM :-P), 2.7GBytes in 
3.45minutes, a mixture of not too big files and quite some small files 
(see the size distribution below).
- this makes it roughly 90-100Mbit/s

As said, nothing comparable, just for comparaison. Even on a local 
backup, the initial backup takes quite a while and becomes better in 
delta mode.

I have the plan to improve performance, without knowing if there is 
really potential, but at this stage, I'm focussing on re-structuring the 
code and understanding it better.

KR, Eric

Neat one liner I found at 
https://superuser.com/questions/565443/generate-distribution-of-file-sizes-from-the-command-prompt
(added as utility at https://github.com/rdiff-backup/rdiff-backup/pull/668)

$ find Downloads -type f -print0 | xargs -0 ls -l | awk 
'{size[int(log($5)/log(2))]++}END{for (i in size) printf("%10d %3d\n", 
2^i, size[i])}' | sort -n
         0   3
         4   4
         8   3
        16   6
        32   5
       128   4
       256   8
       512  14
      1024  27
      2048  29
      4096  30
      8192  18
     16384  15
     32768   9
     65536  16
    131072  15
    262144  12
    524288   6
   1048576   1
   2097152   3
   4194304   7
   8388608   5
  16777216   2
  33554432   4
  67108864   2
 268435456   1
 536870912   1
1073741824   1

On 07/02/2022 18:39, EricZolf wrote:
> Can you send me your hardware that I do some tests?
>
> Seriously, no clue. It might be that you have many very small files but even 
> then the drop in performance is impressive...
> Perhaps someone with a bigger estate can be more helpful.
>
> Eric
>
>
> On 7 February 2022 03:20:15 UTC, Eric Robinson <eric.robinson@psmnv.com> wrote:
> > I have two RHEL 8.5 servers with 48 cores, 1TB RAM, and NVME disks connected 
> > over a 25 Gbit network. I confirmed with iperf that I'm betting 20+ Gbits/sec 
> > bandwidth. Using rdiff-backup 2.0.5, I am only getting 15-25 MB/sec transfer 
> > speed. Why is rdiff-backup so slow? The target server is empty. This is the 
> > first backup attempt.
> >
> > -Eric
> >
> >
> >
> >
> > Disclaimer : This email and any files transmitted with it are confidential and 
> > intended solely for intended recipients. If you are not the named addressee you 
> > should not disseminate, distribute, copy or alter this email. Any views or 
> > opinions presented in this email are solely those of the author and might not 
> > represent those of Physician Select Management. Warning: Although Physician 
> > Select Management has taken reasonable precautions to ensure no viruses are 
> > present in this email, the company cannot accept responsibility for any loss or 
> > damage arising from the use of this email or attachments.