Re: [Duplicity-talk] Duplicity 0.6.16 "File XXX was corrupted during upload." to s3+http://

[Kris]

Sorry, I'm asleep, my last post was nonsense. I'm just going to write up a 
quick tested patch if it helps, unless you're already tackling this.

Kris



On 2011-11-02, at 9:30 PM, Kris wrote:

> Thanks for your help Michael! It's hugely appreciated. I can confirm that I 
> was looking at the same code and saw that the last chunk of a file not 
> perfectly divisible will not have an incorrect number of bytes passed in to 
> param for the last chunk.
> 
> Just to confirm, I believe the value passed to the 'bytes' key should be:
> 
> bytes - (chunk_size*(chunk_size-1))
> 
> Kris
> 
> 
> 
> On 2011-11-02, at 9:18 PM, Michael Terry wrote:
> 
>> On 2 November 2011 19:34, Kris <address@hidden> wrote:
>>> I'm taking a look at it again right now.  Just a FYI before I dig in, I 
>>> removed the volsize cmd line and it did the same thing. The default volsize 
>>> is 25MB in duplicity and that's what the tmp file is. The remote file 
>>> however is 50 megs. The PoolWorkers split the chunk in to two 25 meg chunks 
>>> and uploaded those, which isn't proper.
>> 
>> Good debugging work, Kris.  This isn't bug 498933 after all, but is
>> instead a bug in 0.6.16 specifically.  In that version, the S3 backend
>> got new 'multichunk' support.
>> 
>> But it seems to have a bug that when the size of the volume isn't
>> perfectly divisible by the multichunk size, it will upload too much
>> data.  I've filed bug https://bugs.launchpad.net/duplicity/+bug/885513
>> about it.
>> 
>> For now, I believe you can workaround it with
>> --s3-multipart-chunk-size=1000 (causing duplicity to basically ignore
>> the multipart code).
>> 
>> I feel like suggesting that maybe the multipart branch should be
>> backed out until it can be stress tested a bit more?  (so far we've
>> noticed this bug and bug 878220 (the one where botobackend.py added a
>> new, uncommon, and unconditional import due to the patch)
>> 
>> -mt
> 
> 
> _______________________________________________
> Duplicity-talk mailing list
> address@hidden
> https://lists.nongnu.org/mailman/listinfo/duplicity-talk