Re: [Duplicity-talk] Fwd: AssertionError on every attempt

[Bruce Merry]

On 10 June 2015 at 16:16, Tim Fletcher <address@hidden> wrote:
> I suspect that this is due to the Google storage back-end having difference
> constancy guarantees for single objects vs directory listings.
>
> See https://cloud.google.com/storage/docs/concepts-techniques#consistency

Thanks, that's an interesting link. It's describing Cloud rather than
Drive, but it wouldn't surprise me if Drive is similar i.e. an object
store with a filesystem duct-taped on.

That makes me think that maximum robustness would be achieved by
having duplicity reference IDs internally and only use filenames for
presentation to the user. That sounds like it would need major
architectural changes though, since the list of IDs forming a backup
set would need to be recorded as part of the backup, instead of being
discovered from a directory listing.

A halfway point might be to have the client keep its own filename<->ID
cache in the Duplicity cache directory. Operations would need to query
the object by ID to validate the cache entry, but I think this would
allow for strong consistency in cases where the same client is doing
the accesses (as is the case when an upload is immediately followed by
a query - different clients are more likely to be separated in time).

I can probably have a go at implementing that the next week or two.
Are there helper functions I should look at for the backend to
discover where the cache directory for the backup lives? And any
preferences for the format of the cache file? My personal inclination
would be to go for sqlite to get all the nice safety guarantees that
gives over just a pickle/yaml/json/xml/whatever file, but that would
introduce a dependency.

Bruce
-- 
Dr Bruce Merry
bmerry <@> gmail <.> com
http://www.brucemerry.org.za/
http://blog.brucemerry.org.za/