cross-platform backup tool Some updates about the current direction of rdiff-backup

[EricZolf]

Hi,

I wanted to give you all some news about what's currently happening, as 
you might wonder why nothing is being released.

Well, we're in a major refactoring exercise to be prepared for future 
endeavors. I want a much cleaner architecture which makes it easy to add 
new features, and avoid issues (or easier to fix them :-P).

There are some majors aims with this refactoring:

1. have a clear API between client and server, with versions and clear 
compatibility matrix between different versions of rdiff-backup
2. encapsulate the repository format so that changes can be done without 
impacting the whole code (e.g. different compression algorithm)
3. avoid the use of global variables, especially modified all over the code
4. overall clarify the structure of rdiff-backup, add wherever possible 
a plug-in concept

Currently all these points are not given, and the code has dependencies 
all over the place, so that maintenance is a game with fire (and my 
initial blind migration from Python 2 to 3 didn't make it better).

The target structure has been described at [1] resp. [2] (once merged).

In a few bullet points, the highlights:

1. there is a plug-in architecture for actions [3] on the left hand-side 
(for the ones who've tried the alpha-version, they'll recognize the new 
CLI's actions).
2. the locations don't have (yet?) a plug-in setup but they encapsulate 
the client-server communication _and_ the format of the backup repository.
3. each action on the client side can communicate with one or more 
locations (e.g. backup reads from ReadDir and writes to Repo).
4. it is then the role of the location to communicate with its "shadow" 
hosted on the server side (server is a special action in this regard).
5. the client-server API is then defined solely by:
a. the communication between the locations and their shadows
b. the object types they exchange
c. the global variables exchanged between client and server (too many to 
be completely avoided, this is definitely a sore point and a risk)

So, where do we stand?

The overall structure has been created, a lot of API calls have been 
moved to the new structure (the actions don't anymore communicate 
directly with the server), but there are still many objects 
communicating directly which need to be rewritten.

And because I need to re-write the client-server API while making sure 
that it remains compatible with v2.0.5, and also need to add new tests, 
it takes quite a while. I'm therefor not releasing yet because each new 
release means keeping the compatibility for also this new release.

Anyway, once the whole communication is nicely going through the 
locations, I'll have to do some clean-up because the API doesn't have 
currently much consistency as it comes from a bunch of different 
classes, just merged into the locations. That'll probably mean a 2nd 
complete reshuffling of the code.

The old API has been versioned 200, the new one is currently 201 (on 
purpose not aligned with the rdiff-backup versions), so the plan is the 
following:

* version 2.0.5 has been deemed talking version 200 of the API
* version 2.2 will talk 200 and 201, by default 200
* version 2.4 will talk 201 and 202, by default 201 (I'll probably keep 
200 as well to avoid changing too many tests in this phase)
* etc...
* latest version 2.8 will speak the old version(s) and have a new 
version, say 210, but speaking 20x by default
* then 3.0 will drop the compatibility and only speak 210 (or 
equivalent), definitely deprecating the old cruft, which will again mean 
a lot of changes, especially on the testing side (which currently relies 
mostly on version 200), but avoiding any feature change.
* then we'll start a similar game with the format of the backup 
repository, somewhere between version 3 and 4 of rdiff-backup

Regarding other foreseeable changes, we will have to drop Python 3.6 
relatively soon, and add 3.10.

As for the timeline, no clue, I'm still hesitating on when to freeze the 
API 201 and release version 2.2, but surely sometimes next year.

Stay tuned,
Eric

[1] 
https://github.com/rdiff-backup/rdiff-backup/blob/ericzolf-target-architecture/docs/arch/rdiffbackup_architecture.svg
[2] 
https://github.com/rdiff-backup/rdiff-backup/blob/master/docs/arch/rdiffbackup_architecture.svg
[3] 
https://github.com/rdiff-backup/rdiff-backup/blob/master/docs/arch/plugins_actions.adoc