[Top][All Lists]

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [Duplicity-talk] Restoring from PCA Backend - download preparation

From: Kenneth Loafman
Subject: Re: [Duplicity-talk] Restoring from PCA Backend - download preparation
Date: Sun, 27 Jun 2021 15:58:42 -0500

Will take a look Monday and let you know.


On Sun, Jun 27, 2021 at 12:58 PM <erwan.vallee@gmail.com> wrote:

Thanks for your answer Kenneth !

I agree with you, and my first guess was to modify only pcabackend.py, and not mess with « core » files.


My understanding is that the two hooks are not called in my case :

  • Pre_process_download is never called as BackendWrapper doesn’t implement it (and doesn’t « forward » call to its underlying backend)
  • Pre_process_download_batch is called if implemented by actual backend because hasattr is called on config.backend.backend, but in my case, it doesn’t go through MultiBackend to the actual backend.

That’s why I’ve implemented it on the following classes:

BackendWrapper > MultiBackend > PCABackend


Did I miss something ?


De : Kenneth Loafman
Envoyé le :dimanche 27 juin 2021 18:45
À : Discussion about duplicity backup
Cc : erwan.vallee@gmail.com
Objet :Re: [Duplicity-talk] Restoring from PCA Backend - download preparation


Hi Erwan,


I think you are on the right track, except, your changes need to be in the backends, no place else.  The way this works is that if the backend implements pre_proccess_download[_batch], then they will be used by duplicity.  The way to go is to implement those two functions in your backends.  duplicity will not be confused since it checks with hasattr() to see if it's available.


We implement site-specific issues in the backend that handles the site.  It keeps the base code clean that way.






On Sun, Jun 27, 2021 at 5:05 AM Erwan via Duplicity-talk <duplicity-talk@nongnu.org> wrote:



I’m using duplicity with multibackend (public cloud archive and object storage from ovh), as described in this documentation from ovh :


I’ve worked on a modification to make PCA Backend work better (for my usage at least !) : allow to unseal all volumes at once when restoring content from PCA, wait for unseal to complete for all volumes before launching download. I’ve used API documentation as mentioned here :



This is a similar change to what has been implemented some time ago for Glacier cold storage :


I’ve implemented the following things available in this commit : https://gitlab.com/erwanBoka/duplicity/-/commit/822e42ba3c26a5b32377e238a9b6040b9b087052

  • Implement in BackendWrapper and MultiBackend pre_process_download (and redirect to sub-backends)
  • Implement for pcabackend pre_process_download with ovh API as a blocking call, waiting for all volumes to be unsealed.


However, I’m a bit confused by what was implemented then (new hook pre_process_download_batch), compared to existing one pre_process_download.

  • pre_process_download is called from dup_main/restore_get_patched_rop_iter with all filenames as parameters
  • it’s implemented only once in _boto_single, but with a single filename as argument…
  • pre_process_download_batch is essentially doing the same thing (call with all volume names)

I’m afraid my modifications in BackendWrapper and MultiBackend might cause pre_process_download to be called for _boto_single, and break things…


I haven’t modified anything yet in _boto_single, but my proposal would be :

  • change pre_process_download method name in _boto_single to an « internal » method used only in _get
  • keep only pre_process_download hook in dup_main
  • implement it the way pre_process_download_batch is implemented today in _boto_single

I don’t have s3 access, and cannot test it though.


Let me know your view about this, and the path forward please !

Best regards,






Duplicity-talk mailing list


reply via email to

[Prev in Thread] Current Thread [Next in Thread]