[Top][All Lists]

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Discussion about file format for the future

From: Eric L. Zolf
Subject: Re: Discussion about file format for the future
Date: Thu, 4 Jun 2020 19:55:34 +0200
User-agent: Mozilla/5.0 (X11; Linux x86_64; rv:68.0) Gecko/20100101 Thunderbird/68.8.0


that's an interesting idea, I was more in the optic of keeping a simple
file based structure but if there are requirements, we can think about
alternatives, it could be a SQL or even a noSQL DB, or a keystore.

Regarding the terabytes of data, I take the point, I never foresaw to
convert the data itself, only the metadata, whereas I haven't yet a
clear view on a strategy forward: either a converting tool to "upgrade"
a repo, or convert transparently within rdiff-backup itself, or keep old
data and only use the new format for new metadata (I don't really like
this last one, but if it's the only solution...).

scanning the metadata files is a nightmare -> from which point of view?
Performance and/or complexity of code?

KR, Eric

On 04/06/2020 18:43, Patrik Dufresne wrote:
> Hi Eric,
> My priority in that regard would be to make sure all of this backward 
> compatible with the previous repository. I mean, I have terabytes of data 
> and I don't foresee a way to convert all these metadata files to a new 
> format. And I'm probably not alone with this situation.
> We should probably restructure de code to put the read and write of 
> metadata into a single library. Last time I checked, all those metadata are 
> not that different. Then we could make this metadata read write library 
> evolve and allow us to read or write the old format or the new format.
> If we stick with a file base approach yaml is good for me.
> But two cent on the subject is, should we really keep this filebase ? For 
> rdiffweb, scanning the metadata files is a nightmare. When I just need a 
> subset of the data to be displayed to the user. I always thought a database 
> could be better fit for the job. Something like a key store or similar.
> On Thu, Jun 4, 2020 at 9:52 AM <rhkramer@gmail.com> wrote:
>> Thanks!
>> On Thursday, June 04, 2020 08:37:24 AM EricZolf wrote:
>>> Correct, configuration files and "metadata" files, but also potentially
>>> to exchange information between client and server (e.g. the version
>>> string could be enriched to exchange more information than just
>>> version).

reply via email to

[Prev in Thread] Current Thread [Next in Thread]