Re: [Duplicity-talk] Duplicity 0.6.16 "File XXX was corrupted during upload." to s3+http://

[Michael Terry]

On 2 November 2011 19:34, Kris <address@hidden> wrote:
> I'm taking a look at it again right now.  Just a FYI before I dig in, I 
> removed the volsize cmd line and it did the same thing. The default volsize 
> is 25MB in duplicity and that's what the tmp file is. The remote file however 
> is 50 megs. The PoolWorkers split the chunk in to two 25 meg chunks and 
> uploaded those, which isn't proper.

Good debugging work, Kris.  This isn't bug 498933 after all, but is
instead a bug in 0.6.16 specifically.  In that version, the S3 backend
got new 'multichunk' support.

But it seems to have a bug that when the size of the volume isn't
perfectly divisible by the multichunk size, it will upload too much
data.  I've filed bug https://bugs.launchpad.net/duplicity/+bug/885513
about it.

For now, I believe you can workaround it with
--s3-multipart-chunk-size=1000 (causing duplicity to basically ignore
the multipart code).

I feel like suggesting that maybe the multipart branch should be
backed out until it can be stress tested a bit more?  (so far we've
noticed this bug and bug 878220 (the one where botobackend.py added a
new, uncommon, and unconditional import due to the patch)

-mt