Re: [Duplicity-talk] [Duplicity-team] verify fails on 0.6.18 and 0.7.01

[edgar . soldin]

well done! providing a test case ;)
let's hope it is still maintained.. their latest release is 1,5 years old.

.. ede/duply.net

On 09.02.2015 20:11, Rupert Levene wrote:
> Thanks. I've raised an issue here:
> 
> https://github.com/google/gdata-python-client/issues/5
> 
> Rupert
> 
> On 9 February 2015 at 16:49,  <address@hidden> wrote:
>> good to hear. if you want you may find out where gdocs python API people 
>> hang out and make them aware of the issue.
>>
>> we have this url in our manpage
>>  https://code.google.com/p/gdata-python-client/
>>
>> ..ede/duply.net
>>
>> On 09.02.2015 17:05, Rupert Levene wrote:
>>> I can confirm that verify with 0.7.01 works using a local copy of the
>>> backup folder, but fails using the gdocs backup.
>>>
>>> The large file is a tiny bit larger than 2^31 bytes:
>>>
>>> 2147540150 Feb  6 10:58 
>>> duplicity-full-signatures.20150205T170035Z.sigtar.gpg
>>>
>>> Could an option be added to split signature files into chunks, rather
>>> like --volsize does for the other files?
>>>
>>> I don't think the initial backup gave errors, but I'm not certain. If
>>> you like I can try it again, but it does take rather a long time to
>>> run.
>>>
>>> Rupert
>>>
>>> On 8 February 2015 at 16:32, Kenneth Loafman <address@hidden> wrote:
>>>> Were there any errors during the initial backup?  This looks like a 32-bit
>>>> Python limit.
>>>>
>>>> ...Ken
>>>>
>>>>
>>>>
>>>>
>>>> On Sun, Feb 8, 2015 at 9:16 AM, <address@hidden> wrote:
>>>>>
>>>>> On 08.02.2015 15:46, Rupert Levene wrote:
>>>>>> On 8 February 2015 at 14:32,  <address@hidden> wrote:
>>>>>>> On 08.02.2015 15:05, Rupert Levene wrote:
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> I have made a large to gdocs using 0.6.18 from Ubuntu 12.04. A 2GB
>>>>>>>> file duplicity-full-signatures.20150205T170035Z.sigtar.gpg was
>>>>>>>> created, and duplicity chokes on this. It pauses for a while (I guess
>>>>>>>> while it downloads the file) and then says
>>>>>>>>
>>>>>>>> Attempt 1 failed: BackendException: Failed to download file
>>>>>>>> 'duplicity-full-signatures.20150205T170035Z.sigtar.gpg' in remote
>>>>>>>> folder 'i7-bigstore': join() result is too long for a Python string
>>>>>>>> Backtrace of previous error: Traceback (innermost last):
>>>>>>>>   File "/usr/lib/python2.7/dist-packages/duplicity/backend.py", line
>>>>>>>> 311, in iterate
>>>>>>>>     return fn(*args, **kwargs)
>>>>>>>>   File
>>>>>>>> "/usr/lib/python2.7/dist-packages/duplicity/backends/gdocsbackend.py",
>>>>>>>> line 144, in get
>>>>>>>>     % (remote_filename, self.folder.title.text, str(e)), raise_errors)
>>>>>>>>   File
>>>>>>>> "/usr/lib/python2.7/dist-packages/duplicity/backends/gdocsbackend.py",
>>>>>>>> line 182, in __handle_error
>>>>>>>>     raise BackendException(message)
>>>>>>>>  BackendException: Failed to download file
>>>>>>>> 'duplicity-full-signatures.20150205T170035Z.sigtar.gpg' in remote
>>>>>>>> folder 'i7-bigstore': join() result is too long for a Python string
>>>>>>>>
>>>>>>>> It'll then try again, and give the same error, at which point I hit
>>>>>>>> control-C.
>>>>>>>>
>>>>>>>> Version 0.7.01 fails more quickly:
>>>>>>>>
>>>>>>>> $ apt-cache policy duplicity
>>>>>>>> duplicity:
>>>>>>>>   Installed: 0.7.01-0ubuntu0ppa1063~ubuntu12.04.1
>>>>>>>>   Candidate: 0.7.01-0ubuntu0ppa1063~ubuntu12.04.1
>>>>>>>>   Version table:
>>>>>>>>  *** 0.7.01-0ubuntu0ppa1063~ubuntu12.04.1 0
>>>>>>>>         500 http://ppa.launchpad.net/duplicity-team/ppa/ubuntu/
>>>>>>>> precise/main i386 Packages
>>>>>>>>         100 /var/lib/dpkg/status
>>>>>>>>      0.6.18-0ubuntu3.5 0
>>>>>>>>         500 http://ie.archive.ubuntu.com/ubuntu/ precise-updates/main
>>>>>>>> i386 Packages
>>>>>>>>      0.6.18-0ubuntu3 0
>>>>>>>>         500 http://ie.archive.ubuntu.com/ubuntu/ precise/main i386
>>>>>>>> Packages
>>>>>>>> $ duplicity -V
>>>>>>>> duplicity 0.7.01
>>>>>>>> $ duplicity verify --verbosity '8' --exclude-globbing-filelist
>>>>>>>> /home/rupert/.duply/i7-bigstore/exclude
>>>>>>>> gdocs://address@hidden/backups/duplicity/i7-bigstore
>>>>>>>> /mnt/bigstore
>>>>>>>> Using archive dir: /home/rupert/.cache/duplicity/xxxxxxxxxxxxxxxxxxxxx
>>>>>>>> Using backup name: xxxxxxxxxxxxxxxxxxxxx
>>>>>>>> Import of duplicity.backends.azurebackend Succeeded
>>>>>>>> Import of duplicity.backends.botobackend Succeeded
>>>>>>>> Import of duplicity.backends.cfbackend Succeeded
>>>>>>>> Import of duplicity.backends.copycombackend Succeeded
>>>>>>>> Import of duplicity.backends.dpbxbackend Succeeded
>>>>>>>> Import of duplicity.backends.gdocsbackend Succeeded
>>>>>>>> Import of duplicity.backends.giobackend Succeeded
>>>>>>>> Import of duplicity.backends.hsibackend Succeeded
>>>>>>>> Import of duplicity.backends.hubicbackend Succeeded
>>>>>>>> Import of duplicity.backends.imapbackend Succeeded
>>>>>>>> Import of duplicity.backends.lftpbackend Succeeded
>>>>>>>> Import of duplicity.backends.localbackend Succeeded
>>>>>>>> Import of duplicity.backends.megabackend Succeeded
>>>>>>>> Import of duplicity.backends.ncftpbackend Succeeded
>>>>>>>> Import of duplicity.backends.onedrivebackend Failed: No module named
>>>>>>>> requests
>>>>>>>> Import of duplicity.backends.par2backend Succeeded
>>>>>>>> Import of duplicity.backends.pydrivebackend Succeeded
>>>>>>>> Import of duplicity.backends.rsyncbackend Succeeded
>>>>>>>> Import of duplicity.backends.ssh_paramiko_backend Succeeded
>>>>>>>> Import of duplicity.backends.ssh_pexpect_backend Succeeded
>>>>>>>> Import of duplicity.backends.swiftbackend Succeeded
>>>>>>>> Import of duplicity.backends.sxbackend Succeeded
>>>>>>>> Import of duplicity.backends.tahoebackend Succeeded
>>>>>>>> Import of duplicity.backends.webdavbackend Succeeded
>>>>>>>> Using temporary directory /tmp/duplicity-lCUBfI-tempdir
>>>>>>>> Traceback (most recent call last):
>>>>>>>>   File "/usr/bin/duplicity", line 1497, in <module>
>>>>>>>>     with_tempdir(main)
>>>>>>>>   File "/usr/bin/duplicity", line 1491, in with_tempdir
>>>>>>>>     fn()
>>>>>>>>   File "/usr/bin/duplicity", line 1324, in main
>>>>>>>>     action = commandline.ProcessCommandLine(sys.argv[1:])
>>>>>>>>   File "/usr/lib/python2.7/dist-packages/duplicity/commandline.py",
>>>>>>>> line 1056, in ProcessCommandLine
>>>>>>>>     backup, local_pathname = set_backend(args[0], args[1])
>>>>>>>>   File "/usr/lib/python2.7/dist-packages/duplicity/commandline.py",
>>>>>>>> line 949, in set_backend
>>>>>>>>     globals.backend = backend.get_backend(bend)
>>>>>>>>   File "/usr/lib/python2.7/dist-packages/duplicity/backend.py", line
>>>>>>>> 221, in get_backend
>>>>>>>>     obj = get_backend_object(url_string)
>>>>>>>>   File "/usr/lib/python2.7/dist-packages/duplicity/backend.py", line
>>>>>>>> 207, in get_backend_object
>>>>>>>>     return factory(pu)
>>>>>>>>   File
>>>>>>>> "/usr/lib/python2.7/dist-packages/duplicity/backends/gdocsbackend.py",
>>>>>>>> line 61, in __init__
>>>>>>>>     entries = self._fetch_entries(parent_folder_id, 'folder',
>>>>>>>> folder_name)
>>>>>>>>   File
>>>>>>>> "/usr/lib/python2.7/dist-packages/duplicity/backends/gdocsbackend.py",
>>>>>>>> line 161, in _fetch_entries
>>>>>>>>     entries = self.client.get_all_resources(uri=uri)
>>>>>>>> AttributeError: 'DocsClient' object has no attribute
>>>>>>>> 'get_all_resources'
>>>>>>>>
>>>>>>>> Are these two independent bugs? Is there a workaround or fix?
>>>>>>>>
>>>>>>>
>>>>>>> yes, it looks like two different bugs with two different error
>>>>>>> messages.
>>>>>>>
>>>>>>> first try latest stable duplicity 0.6.25.. there were changes in the
>>>>>>> gdocs backend since your version.
>>>>>>>
>>>>>>> if that fails - can you try downloading your whole backups to a local
>>>>>>> folder and try to restore from there?
>>>>>>>
>>>>>>> ..ede/duply.net
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> Duplicity-talk mailing list
>>>>>>> address@hidden
>>>>>>> https://lists.nongnu.org/mailman/listinfo/duplicity-talk
>>>>>>
>>>>>> I had tried 0.6.25 too, and it was giving the same get_all_resources
>>>>>> error as 0.7.01. However since then I used pip to upgrade gdata, and
>>>>>> 0.7.01 gives me very similar output to my original error from 0.6.18:
>>>>>>
>>>>>> $ duplicity verify --verbosity '9' --exclude-globbing-filelist
>>>>>> /home/rupert/.duply/i7-bigstore/exclude
>>>>>> gdocs://address@hidden/backups/duplicity/i7-bigstore
>>>>>> /mnt/bigstore
>>>>>> Using archive dir:
>>>>>> /home/rupert/.cache/duplicity/xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
>>>>>> Using backup name: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
>>>>>> Import of duplicity.backends.azurebackend Succeeded
>>>>>> Import of duplicity.backends.botobackend Succeeded
>>>>>> Import of duplicity.backends.cfbackend Succeeded
>>>>>> Import of duplicity.backends.copycombackend Succeeded
>>>>>> Import of duplicity.backends.dpbxbackend Succeeded
>>>>>> Import of duplicity.backends.gdocsbackend Succeeded
>>>>>> Import of duplicity.backends.giobackend Succeeded
>>>>>> Import of duplicity.backends.hsibackend Succeeded
>>>>>> Import of duplicity.backends.hubicbackend Succeeded
>>>>>> Import of duplicity.backends.imapbackend Succeeded
>>>>>> Import of duplicity.backends.lftpbackend Succeeded
>>>>>> Import of duplicity.backends.localbackend Succeeded
>>>>>> Import of duplicity.backends.megabackend Succeeded
>>>>>> Import of duplicity.backends.ncftpbackend Succeeded
>>>>>> Import of duplicity.backends.onedrivebackend Failed: No module named
>>>>>> requests
>>>>>> Import of duplicity.backends.par2backend Succeeded
>>>>>> Import of duplicity.backends.pydrivebackend Succeeded
>>>>>> Import of duplicity.backends.rsyncbackend Succeeded
>>>>>> Import of duplicity.backends.ssh_paramiko_backend Succeeded
>>>>>> Import of duplicity.backends.ssh_pexpect_backend Succeeded
>>>>>> Import of duplicity.backends.swiftbackend Succeeded
>>>>>> Import of duplicity.backends.sxbackend Succeeded
>>>>>> Import of duplicity.backends.tahoebackend Succeeded
>>>>>> Import of duplicity.backends.webdavbackend Succeeded
>>>>>> Reading globbing filelist /home/rupert/.duply/i7-bigstore/exclude
>>>>>> Main action: verify
>>>>>>
>>>>>> ================================================================================
>>>>>> duplicity 0.7.01 ($reldate)
>>>>>> Args: /usr/bin/duplicity verify --verbosity 9
>>>>>> --exclude-globbing-filelist /home/rupert/.duply/i7-bigstore/exclude
>>>>>> gdocs://address@hidden/backups/duplicity/i7-bigstore
>>>>>> /mnt/bigstore
>>>>>> Linux levene-i7desktop 3.13.0-44-generic #73~precise1-Ubuntu SMP Wed
>>>>>> Dec 17 00:38:38 UTC 2014 i686 i686
>>>>>> /usr/bin/python 2.7.3 (default, Dec 18 2014, 19:03:52)
>>>>>> [GCC 4.6.3]
>>>>>>
>>>>>> ================================================================================
>>>>>> Synchronizing remote metadata to local cache...
>>>>>> Copying duplicity-full-signatures.20150205T170035Z.sigtar.gpg to local
>>>>>> cache.
>>>>>> Using temporary directory /tmp/duplicity-L9vwHt-tempdir
>>>>>> Registering (mktemp) temporary file
>>>>>> /tmp/duplicity-L9vwHt-tempdir/mktemp-_vVr3t-1
>>>>>> Backtrace of previous error: Traceback (innermost last):
>>>>>>   File "/usr/lib/python2.7/dist-packages/duplicity/backend.py", line
>>>>>> 361, in inner_retry
>>>>>>     return fn(self, *args)
>>>>>>   File "/usr/lib/python2.7/dist-packages/duplicity/backend.py", line
>>>>>> 534, in get
>>>>>>     self.backend._get(remote_filename, local_path)
>>>>>>   File
>>>>>> "/usr/lib/python2.7/dist-packages/duplicity/backends/gdocsbackend.py",
>>>>>> line 110, in _get
>>>>>>     self.client.DownloadResource(entry, local_path.name)
>>>>>>   File "/usr/local/lib/python2.7/dist-packages/gdata/docs/client.py",
>>>>>> line 369, in download_resource
>>>>>>     self._download_file(uri, file_path, **kwargs)
>>>>>>   File "/usr/local/lib/python2.7/dist-packages/gdata/docs/client.py",
>>>>>> line 468, in _download_file
>>>>>>     f.write(self._get_content(uri, **kwargs))
>>>>>>   File "/usr/local/lib/python2.7/dist-packages/gdata/docs/client.py",
>>>>>> line 451, in _get_content
>>>>>>     return server_response.read()
>>>>>>   File "/usr/lib/python2.7/httplib.py", line 541, in read
>>>>>>     return self._read_chunked(amt)
>>>>>>   File "/usr/lib/python2.7/httplib.py", line 624, in _read_chunked
>>>>>>     return ''.join(value)
>>>>>>  OverflowError: join() result is too long for a Python string
>>>>>>
>>>>>> Attempt 1 failed. OverflowError: join() result is too long for a Python
>>>>>> string
>>>>>> Backtrace of previous error: Traceback (innermost last):
>>>>>>   File "/usr/lib/python2.7/dist-packages/duplicity/backend.py", line
>>>>>> 361, in inner_retry
>>>>>>     return fn(self, *args)
>>>>>>   File "/usr/lib/python2.7/dist-packages/duplicity/backend.py", line
>>>>>> 534, in get
>>>>>>     self.backend._get(remote_filename, local_path)
>>>>>>   File
>>>>>> "/usr/lib/python2.7/dist-packages/duplicity/backends/gdocsbackend.py",
>>>>>> line 110, in _get
>>>>>>     self.client.DownloadResource(entry, local_path.name)
>>>>>>   File "/usr/local/lib/python2.7/dist-packages/gdata/docs/client.py",
>>>>>> line 369, in download_resource
>>>>>>     self._download_file(uri, file_path, **kwargs)
>>>>>>   File "/usr/local/lib/python2.7/dist-packages/gdata/docs/client.py",
>>>>>> line 468, in _download_file
>>>>>>     f.write(self._get_content(uri, **kwargs))
>>>>>>   File "/usr/local/lib/python2.7/dist-packages/gdata/docs/client.py",
>>>>>> line 451, in _get_content
>>>>>>     return server_response.read()
>>>>>>   File "/usr/lib/python2.7/httplib.py", line 541, in read
>>>>>>     return self._read_chunked(amt)
>>>>>>   File "/usr/lib/python2.7/httplib.py", line 590, in _read_chunked
>>>>>>     value.append(self._safe_read(chunk_left))
>>>>>>   File "/usr/lib/python2.7/httplib.py", line 647, in _safe_read
>>>>>>     chunk = self.fp.read(min(amt, MAXAMOUNT))
>>>>>>   File "/usr/lib/python2.7/socket.py", line 396, in read
>>>>>>     buf.write(data)
>>>>>>  MemoryError: out of memory
>>>>>>
>>>>>> Attempt 2 failed. MemoryError: out of memory
>>>>>> Backtrace of previous error: Traceback (innermost last):
>>>>>>   File "/usr/lib/python2.7/dist-packages/duplicity/backend.py", line
>>>>>> 361, in inner_retry
>>>>>>     return fn(self, *args)
>>>>>>   File "/usr/lib/python2.7/dist-packages/duplicity/backend.py", line
>>>>>> 534, in get
>>>>>>     self.backend._get(remote_filename, local_path)
>>>>>>   File
>>>>>> "/usr/lib/python2.7/dist-packages/duplicity/backends/gdocsbackend.py",
>>>>>> line 110, in _get
>>>>>>     self.client.DownloadResource(entry, local_path.name)
>>>>>>   File "/usr/local/lib/python2.7/dist-packages/gdata/docs/client.py",
>>>>>> line 369, in download_resource
>>>>>>     self._download_file(uri, file_path, **kwargs)
>>>>>>   File "/usr/local/lib/python2.7/dist-packages/gdata/docs/client.py",
>>>>>> line 468, in _download_file
>>>>>>     f.write(self._get_content(uri, **kwargs))
>>>>>>   File "/usr/local/lib/python2.7/dist-packages/gdata/docs/client.py",
>>>>>> line 451, in _get_content
>>>>>>     return server_response.read()
>>>>>>   File "/usr/lib/python2.7/httplib.py", line 541, in read
>>>>>>     return self._read_chunked(amt)
>>>>>>   File "/usr/lib/python2.7/httplib.py", line 624, in _read_chunked
>>>>>>     return ''.join(value)
>>>>>>  OverflowError: join() result is too long for a Python string
>>>>>>
>>>>>> Attempt 3 failed. OverflowError: join() result is too long for a Python
>>>>>> string
>>>>>> Backtrace of previous error: Traceback (innermost last):
>>>>>>   File "/usr/lib/python2.7/dist-packages/duplicity/backend.py", line
>>>>>> 361, in inner_retry
>>>>>>     return fn(self, *args)
>>>>>>   File "/usr/lib/python2.7/dist-packages/duplicity/backend.py", line
>>>>>> 534, in get
>>>>>>     self.backend._get(remote_filename, local_path)
>>>>>>   File
>>>>>> "/usr/lib/python2.7/dist-packages/duplicity/backends/gdocsbackend.py",
>>>>>> line 110, in _get
>>>>>>     self.client.DownloadResource(entry, local_path.name)
>>>>>>   File "/usr/local/lib/python2.7/dist-packages/gdata/docs/client.py",
>>>>>> line 369, in download_resource
>>>>>>     self._download_file(uri, file_path, **kwargs)
>>>>>>   File "/usr/local/lib/python2.7/dist-packages/gdata/docs/client.py",
>>>>>> line 468, in _download_file
>>>>>>     f.write(self._get_content(uri, **kwargs))
>>>>>>   File "/usr/local/lib/python2.7/dist-packages/gdata/docs/client.py",
>>>>>> line 451, in _get_content
>>>>>>     return server_response.read()
>>>>>>   File "/usr/lib/python2.7/httplib.py", line 541, in read
>>>>>>     return self._read_chunked(amt)
>>>>>>   File "/usr/lib/python2.7/httplib.py", line 590, in _read_chunked
>>>>>>     value.append(self._safe_read(chunk_left))
>>>>>>   File "/usr/lib/python2.7/httplib.py", line 647, in _safe_read
>>>>>>     chunk = self.fp.read(min(amt, MAXAMOUNT))
>>>>>>   File "/usr/lib/python2.7/socket.py", line 400, in read
>>>>>>     buf.write(data)
>>>>>>  MemoryError: out of memory
>>>>>>
>>>>>> Attempt 4 failed. MemoryError: out of memory
>>>>>> Backtrace of previous error: Traceback (innermost last):
>>>>>>   File "/usr/lib/python2.7/dist-packages/duplicity/backend.py", line
>>>>>> 361, in inner_retry
>>>>>>     return fn(self, *args)
>>>>>>   File "/usr/lib/python2.7/dist-packages/duplicity/backend.py", line
>>>>>> 534, in get
>>>>>>     self.backend._get(remote_filename, local_path)
>>>>>>   File
>>>>>> "/usr/lib/python2.7/dist-packages/duplicity/backends/gdocsbackend.py",
>>>>>> line 110, in _get
>>>>>>     self.client.DownloadResource(entry, local_path.name)
>>>>>>   File "/usr/local/lib/python2.7/dist-packages/gdata/docs/client.py",
>>>>>> line 369, in download_resource
>>>>>>     self._download_file(uri, file_path, **kwargs)
>>>>>>   File "/usr/local/lib/python2.7/dist-packages/gdata/docs/client.py",
>>>>>> line 468, in _download_file
>>>>>>     f.write(self._get_content(uri, **kwargs))
>>>>>>   File "/usr/local/lib/python2.7/dist-packages/gdata/docs/client.py",
>>>>>> line 451, in _get_content
>>>>>>     return server_response.read()
>>>>>>   File "/usr/lib/python2.7/httplib.py", line 541, in read
>>>>>>     return self._read_chunked(amt)
>>>>>>   File "/usr/lib/python2.7/httplib.py", line 624, in _read_chunked
>>>>>>     return ''.join(value)
>>>>>>  OverflowError: join() result is too long for a Python string
>>>>>>
>>>>>> Giving up after 5 attempts. OverflowError: join() result is too long
>>>>>> for a Python string
>>>>>> Releasing lockfile <lockfile.LinkFileLock instance at 0x9d0986c>
>>>>>> Removing still remembered temporary file
>>>>>> /tmp/duplicity-L9vwHt-tempdir/mktemp-_vVr3t-1
>>>>>> $
>>>>>>
>>>>>> I am running 32-bit linux with 4gb RAM and 4gb swap.
>>>>>>
>>>>>> I'll try downloading the backup to local storage as you suggest.
>>>>>>
>>>>>
>>>>> please do.. looks like httplib fails on 2GB files and there isn't much we
>>>>> can do about that.
>>>>>
>>>>> Mike, Ken: any idea?
>>>>>
>>>>> ..ede/duply.net
>>>>>
>>>>> _______________________________________________
>>>>> Mailing list: https://launchpad.net/~duplicity-team
>>>>> Post to     : address@hidden
>>>>> Unsubscribe : https://launchpad.net/~duplicity-team
>>>>> More help   : https://help.launchpad.net/ListHelp
>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> Duplicity-talk mailing list
>>>> address@hidden
>>>> https://lists.nongnu.org/mailman/listinfo/duplicity-talk
>>>>
>>
>> _______________________________________________
>> Duplicity-talk mailing list
>> address@hidden
>> https://lists.nongnu.org/mailman/listinfo/duplicity-talk
> 
> _______________________________________________
> Duplicity-talk mailing list
> address@hidden
> https://lists.nongnu.org/mailman/listinfo/duplicity-talk
>