Re: [Duplicity-talk] Following up on an extended attributes question

[Kenneth Loafman]

Hi Arjun,

Glad you're willing to help.  The best (only) formal docs we have are at readthedocs, or in the docs directory, but those are incomplete.  I started working on them a long while back and other things just got in the way.  Thankfully, the docs are part of the source code, so can be added to as you work on the module.  We'd appreciate you doing that as you go.  It'll help all of us in the long run.

librsync is the 'diff' tool we use.  It's the core part of the rsync algorithm and how we keep network bandwidth low.  This is worth a good read.

As to code style, we use PEP8 style, and use PyLint to lint check the code.  These are all part of the testing we do before release, so just be aware of them.

...Ken

On Mon, May 29, 2017 at 1:30 PM, edgar.soldin--- via Duplicity-talk <address@hidden> wrote:

hey Arjun,

we are a quite friendly and patient bunch on this list. i'd say go for it, every contribution is welcome, time is not a factor :)

..ede/duply.net

On May 29, 2017 8:07:58 PM GMT+02:00, Arjun Krishnan via Duplicity-talk <address@hidden> wrote:
>Hi Ken
>
>I'd be happy to work on it, with a couple of caveats: I'm a research
>mathematician by day, and have little professional programming
>experience.
>So as long as people are willing to indulge slow progress and trivial
>questions as I get familiar with the codebase, I'd be willing to give
>it a
>shot!
>
>
>> Currently we use the GNU_FORMAT, which is the default format, so
>there is
>> no pax_header in that format.  For sake of backwards compatibility,
>we'd
>> need to be able to read GNU_FORMAT, and the PAX_FORMAT would need to
>be
>> readable by GNU tar for manual recovery of a corrupted backup.
>Perhaps
>> it's time to change formats?  I don't know all the tradeoffs we'll
>need to
>> make and what the complexity would be.
>>
>gnu tar appears to support pax_format with `--xattrs` flag. From what
>I've
>seen so far, tarfile happily reads both pax and gnu format
>files, but it neither appends extended attributes when creating files,
>nor
>does it extract files properly and set extended attributes.
>Maybe they didn't want to add a dependency on the xattr module, or
>maybe
>this is for cross platform compatibility.
>
>>
>> Since pax_headers are key/value pairs, we need to track their size
>and be
>> prepared for a value that may exceed our max block size.  I've read
>of
>> cases where key/value pairs exceed the size of the file itself,
>performing
>> the functional equivalent of a dictionary file.  We would have to
>allow for
>> cases where a large value crosses a volume boundary.
>>
>I can see that the blocksize is a diff process parameter,
>and that volsize affects the difftar size. I'm still figuring out
>what's
>being compared during the diff process,
>and how the diffs are being created, so I don't understand how the
>xattrs
>fit in to this process.
>
>Is there a brief technical overview of what duplicity is doing?
>
>Arjun

_______________________________________________
Duplicity-talk mailing list
address@hidden
https://lists.nongnu.org/mailman/listinfo/duplicity-talk