Re: [Duplicity-talk] Verification of filelists, multiprocessing

[edgar . soldin]

On 28.01.2016 01:35, Cláudio Gil wrote:
> Em 27/01/2016 22:27, "Andreas Doll" <address@hidden> escreveu:
>>
>> On 2016-01-27 at 21:27, Cláudio Gil wrote:
>>> Maybe paralelization could be introduced in a few select points of
> duplicity.
>>> But the part that is mostly CPU-bound is the encryption and encrypting
> a file
>>> will use 1 core.
>>
>> Yes, my naive idea was that it may be possible to de/encrypt files in
>> parallel. However, I'm not familiar with duplicitys codebase, and I'm not
> sure
>> if this is feasible.
> 
> Multiple volumes can be encrypted or decrypted in parallel but, regardless
> of being easy or hard to change duplicity to do it, I'm not even sure it's
> worth it. There is a lot of IO involved, for example, and volumes still
> need to be processed in a sequence, by duplicity.
> 

biggest bottleneck (in terms of speed) currently is probably the sequential 
upload to the backend. if someone would completely asynchronize that from 
backup creation, so duplicity could generate away and files would be uploaded 
independently in another thread, maybe even in parallel if there are enough 
volumes finished, that would make a dent i guess.

ede/duply.net