[Duplicity-talk] RE : RE : Restoring from PCA Backend - download preparation

[erwan.vallee]

Seems that solution is not far, but we’re not there yet…

 

I renamed the method after your suggestion but found issues in the way pre_process_download_batch is called compared to pre_process_download (as it is today) :

• pre_process_download_batch is called for each backup_set restored : if restored backup contain 1 full backup and 5 incremental ones, pre_process_download_batch is called 6 times, which prevents unsealing everything at the same time. That’s not the intended behavior.
• pre_process_download is called with a list of files in current version (and all the volumes from all backup set)

 

Not really sure where to go from here. Current solution works well for me implementing pre_process_download, but from what you just said, it’s not the way it’s supposed to work.

 

Erwan

 

De : Kenneth Loafman
Envoyé le :lundi 28 juin 2021 15:03
À : erwan.vallee@gmail.com
Cc : Discussion about duplicity backup
Objet :Re: RE : [Duplicity-talk] Restoring from PCA Backend - download preparation

 

Looks good, except:

1. pre_process_download() is for one file at a time
2. pre_process_download_batch() is for a list of files
3. CI/CD pipeline needs to pass

I think simple name changes are all that's needed and we're good to go.

 

...Ken

 

 

On Sun, Jun 27, 2021 at 3:58 PM Kenneth Loafman <kenneth@loafman.com> wrote:

Will take a look Monday and let you know.

 

...Ken

 

On Sun, Jun 27, 2021 at 12:58 PM <erwan.vallee@gmail.com> wrote:

Thanks for your answer Kenneth !

I agree with you, and my first guess was to modify only pcabackend.py, and not mess with « core » files.

 

My understanding is that the two hooks are not called in my case :

• Pre_process_download is never called as BackendWrapper doesn’t implement it (and doesn’t « forward » call to its underlying backend)
• Pre_process_download_batch is called if implemented by actual backend because hasattr is called on config.backend.backend, but in my case, it doesn’t go through MultiBackend to the actual backend.

That’s why I’ve implemented it on the following classes:

BackendWrapper > MultiBackend > PCABackend

 

Did I miss something ?

 

De : Kenneth Loafman
Envoyé le :dimanche 27 juin 2021 18:45
À : Discussion about duplicity backup
Cc : erwan.vallee@gmail.com
Objet :Re: [Duplicity-talk] Restoring from PCA Backend - download preparation

 

Hi Erwan,

 

I think you are on the right track, except, your changes need to be in the backends, no place else.  The way this works is that if the backend implements pre_proccess_download[_batch], then they will be used by duplicity.  The way to go is to implement those two functions in your backends.  duplicity will not be confused since it checks with hasattr() to see if it's available.

 

We implement site-specific issues in the backend that handles the site.  It keeps the base code clean that way.

 

...Thanks,

...Ken

 

 

On Sun, Jun 27, 2021 at 5:05 AM Erwan via Duplicity-talk <duplicity-talk@nongnu.org> wrote:

Hi,

 

I’m using duplicity with multibackend (public cloud archive and object storage from ovh), as described in this documentation from ovh :

https://docs.ovh.com/gb/en/storage/pca/duplicity/

I’ve worked on a modification to make PCA Backend work better (for my usage at least !) : allow to unseal all volumes at once when restoring content from PCA, wait for unseal to complete for all volumes before launching download. I’ve used API documentation as mentioned here :

https://docs.ovh.com/gb/en/storage/pca/api/

 

This is a similar change to what has been implemented some time ago for Glacier cold storage :

https://lists.nongnu.org/archive/html/duplicity-talk/2020-07/msg00021.html

I’ve implemented the following things available in this commit : https://gitlab.com/erwanBoka/duplicity/-/commit/822e42ba3c26a5b32377e238a9b6040b9b087052

• Implement in BackendWrapper and MultiBackend pre_process_download (and redirect to sub-backends)
• Implement for pcabackend pre_process_download with ovh API as a blocking call, waiting for all volumes to be unsealed.

 

However, I’m a bit confused by what was implemented then (new hook pre_process_download_batch), compared to existing one pre_process_download.

• pre_process_download is called from dup_main/restore_get_patched_rop_iter with all filenames as parameters
• it’s implemented only once in _boto_single, but with a single filename as argument…
• pre_process_download_batch is essentially doing the same thing (call with all volume names)

I’m afraid my modifications in BackendWrapper and MultiBackend might cause pre_process_download to be called for _boto_single, and break things…

 

I haven’t modified anything yet in _boto_single, but my proposal would be :

• change pre_process_download method name in _boto_single to an « internal » method used only in _get
• keep only pre_process_download hook in dup_main
• implement it the way pre_process_download_batch is implemented today in _boto_single

I don’t have s3 access, and cannot test it though.

 

Let me know your view about this, and the path forward please !

Best regards,

Erwan

 

 

 

 

_______________________________________________
Duplicity-talk mailing list
Duplicity-talk@nongnu.org
https://lists.nongnu.org/mailman/listinfo/duplicity-talk