Re: [Duplicity-talk] Verification of filelists, multiprocessing

[Nate Eldredge]

On Wed, 27 Jan 2016, Cláudio Gil wrote:

> Hi,
>
> About pararelism, from what I could see, duplicity processes your files as
> a stream, a long sequence of files. Encryption is also sequential by design
> (that's why GnuPG, what duplicity uses, does not use threads).
>
> Maybe paralelization could be introduced in a few select points of
> duplicity. But the part that is mostly CPU-bound is the encryption and
> encrypting a file will use 1 core.
>
> Since the entire volume is encrypted and not the files in it, you could
> test the difference between between backing up 250MB (10 volumes, by
> default) with encryption and backing up the same files without encryption
> and then encrypting all the volumes with multiple GnuPG processes (using
> "parallel" or something).
>
> It's an interesting comparison. It's not obvious if it be faster and how
> changing the volume size affects the result.

I guess one could write a wrapper around gpg that slurps up the input data 
and exits, keeping gpg itself running on that data in the background, with 
some sort of simple IPC to coordinate a maximum number of gpg processes. 
This could be a useful thing to have in general.  And then one wouldn't 
need to modify duplicity at all.

Maybe a fun project in someone's "copious free time"...

--
Nate Eldredge
address@hidden