[Top][All Lists]

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [Duplicity-talk] Restoring from PCA Backend - download preparation

From: Erwan Vallée
Subject: Re: [Duplicity-talk] Restoring from PCA Backend - download preparation
Date: Wed, 30 Jun 2021 11:31:03 +0200
User-agent: Mailbird/

Hi Kenneth,

I've created the MR : https://gitlab.com/duplicity/duplicity/-/merge_requests/61
I'm sorry, I'm new to gitlab and I'm not sure whether this is correctly done. If you need me to modify something, I can do it. I don't mind at all if you modify the code, and I can check out the pca side of course.


On 6/29/2021 6:01:00 PM, Kenneth Loafman <kenneth@loafman.com> wrote:

Hi Erwan,

Let's do this:

Please submit an MR to GitLab with your working code.  Title it with WIP: in front of the subject.  I'll need access to modify the code if you don't mind.  I can check out the S3 side of things while you check out the PCA side.

From what I can see, there is at least one broken piece of code in duplicity and there may be more.  The example code we'll use is the pre_process_* code in _boto_single.

That will keep the comments we make in the MR and we won't have to deal with the mail breaking up the conversation with RE: instead of Re:.


On Mon, Jun 28, 2021 at 4:58 PM <erwan.vallee@gmail.com> wrote:

Seems that solution is not far, but we’re not there yet…


I renamed the method after your suggestion but found issues in the way pre_process_download_batch is called compared to pre_process_download (as it is today) :

  • pre_process_download_batch is called for each backup_set restored : if restored backup contain 1 full backup and 5 incremental ones, pre_process_download_batch is called 6 times, which prevents unsealing everything at the same time. That’s not the intended behavior.
  • pre_process_download is called with a list of files in current version (and all the volumes from all backup set)


Not really sure where to go from here. Current solution works well for me implementing pre_process_download, but from what you just said, it’s not the way it’s supposed to work.




De : Kenneth Loafman
Envoyé le :lundi 28 juin 2021 15:03
À : erwan.vallee@gmail.com
Cc : Discussion about duplicity backup
Objet :Re: RE : [Duplicity-talk] Restoring from PCA Backend - download preparation


Looks good, except:

  1. pre_process_download() is for one file at a time
  2. pre_process_download_batch() is for a list of files
  3. CI/CD pipeline needs to pass

I think simple name changes are all that's needed and we're good to go.





On Sun, Jun 27, 2021 at 3:58 PM Kenneth Loafman <kenneth@loafman.com> wrote:

Will take a look Monday and let you know.




On Sun, Jun 27, 2021 at 12:58 PM <erwan.vallee@gmail.com> wrote:

Thanks for your answer Kenneth !

I agree with you, and my first guess was to modify only pcabackend.py, and not mess with « core » files.


My understanding is that the two hooks are not called in my case :

  • Pre_process_download is never called as BackendWrapper doesn’t implement it (and doesn’t « forward » call to its underlying backend)
  • Pre_process_download_batch is called if implemented by actual backend because hasattr is called on config.backend.backend, but in my case, it doesn’t go through MultiBackend to the actual backend.

That’s why I’ve implemented it on the following classes:

BackendWrapper > MultiBackend > PCABackend


Did I miss something ?


De : Kenneth Loafman
Envoyé le :dimanche 27 juin 2021 18:45
À : Discussion about duplicity backup
Cc : erwan.vallee@gmail.com
Objet :Re: [Duplicity-talk] Restoring from PCA Backend - download preparation


Hi Erwan,


I think you are on the right track, except, your changes need to be in the backends, no place else.  The way this works is that if the backend implements pre_proccess_download[_batch], then they will be used by duplicity.  The way to go is to implement those two functions in your backends.  duplicity will not be confused since it checks with hasattr() to see if it's available.


We implement site-specific issues in the backend that handles the site.  It keeps the base code clean that way.






On Sun, Jun 27, 2021 at 5:05 AM Erwan via Duplicity-talk <duplicity-talk@nongnu.org> wrote:



I’m using duplicity with multibackend (public cloud archive and object storage from ovh), as described in this documentation from ovh :


I’ve worked on a modification to make PCA Backend work better (for my usage at least !) : allow to unseal all volumes at once when restoring content from PCA, wait for unseal to complete for all volumes before launching download. I’ve used API documentation as mentioned here :



This is a similar change to what has been implemented some time ago for Glacier cold storage :


I’ve implemented the following things available in this commit : https://gitlab.com/erwanBoka/duplicity/-/commit/822e42ba3c26a5b32377e238a9b6040b9b087052

  • Implement in BackendWrapper and MultiBackend pre_process_download (and redirect to sub-backends)
  • Implement for pcabackend pre_process_download with ovh API as a blocking call, waiting for all volumes to be unsealed.


However, I’m a bit confused by what was implemented then (new hook pre_process_download_batch), compared to existing one pre_process_download.

  • pre_process_download is called from dup_main/restore_get_patched_rop_iter with all filenames as parameters
  • it’s implemented only once in _boto_single, but with a single filename as argument…
  • pre_process_download_batch is essentially doing the same thing (call with all volume names)

I’m afraid my modifications in BackendWrapper and MultiBackend might cause pre_process_download to be called for _boto_single, and break things…


I haven’t modified anything yet in _boto_single, but my proposal would be :

  • change pre_process_download method name in _boto_single to an « internal » method used only in _get
  • keep only pre_process_download hook in dup_main
  • implement it the way pre_process_download_batch is implemented today in _boto_single

I don’t have s3 access, and cannot test it though.


Let me know your view about this, and the path forward please !

Best regards,






Duplicity-talk mailing list



reply via email to

[Prev in Thread] Current Thread [Next in Thread]