Re: [Duplicity-talk] [Duplicity-team] verify fails on 0.6.18 and 0.7.01

[Rupert Levene]

Thanks. I've raised an issue here:

https://github.com/google/gdata-python-client/issues/5

Rupert

On 9 February 2015 at 16:49,  <address@hidden> wrote:
> good to hear. if you want you may find out where gdocs python API people hang 
> out and make them aware of the issue.
>
> we have this url in our manpage
>  https://code.google.com/p/gdata-python-client/
>
> ..ede/duply.net
>
> On 09.02.2015 17:05, Rupert Levene wrote:
>> I can confirm that verify with 0.7.01 works using a local copy of the
>> backup folder, but fails using the gdocs backup.
>>
>> The large file is a tiny bit larger than 2^31 bytes:
>>
>> 2147540150 Feb  6 10:58 duplicity-full-signatures.20150205T170035Z.sigtar.gpg
>>
>> Could an option be added to split signature files into chunks, rather
>> like --volsize does for the other files?
>>
>> I don't think the initial backup gave errors, but I'm not certain. If
>> you like I can try it again, but it does take rather a long time to
>> run.
>>
>> Rupert
>>
>> On 8 February 2015 at 16:32, Kenneth Loafman <address@hidden> wrote:
>>> Were there any errors during the initial backup?  This looks like a 32-bit
>>> Python limit.
>>>
>>> ...Ken
>>>
>>>
>>>
>>>
>>> On Sun, Feb 8, 2015 at 9:16 AM, <address@hidden> wrote:
>>>>
>>>> On 08.02.2015 15:46, Rupert Levene wrote:
>>>>> On 8 February 2015 at 14:32,  <address@hidden> wrote:
>>>>>> On 08.02.2015 15:05, Rupert Levene wrote:
>>>>>>> Hi,
>>>>>>>
>>>>>>> I have made a large to gdocs using 0.6.18 from Ubuntu 12.04. A 2GB
>>>>>>> file duplicity-full-signatures.20150205T170035Z.sigtar.gpg was
>>>>>>> created, and duplicity chokes on this. It pauses for a while (I guess
>>>>>>> while it downloads the file) and then says
>>>>>>>
>>>>>>> Attempt 1 failed: BackendException: Failed to download file
>>>>>>> 'duplicity-full-signatures.20150205T170035Z.sigtar.gpg' in remote
>>>>>>> folder 'i7-bigstore': join() result is too long for a Python string
>>>>>>> Backtrace of previous error: Traceback (innermost last):
>>>>>>>   File "/usr/lib/python2.7/dist-packages/duplicity/backend.py", line
>>>>>>> 311, in iterate
>>>>>>>     return fn(*args, **kwargs)
>>>>>>>   File
>>>>>>> "/usr/lib/python2.7/dist-packages/duplicity/backends/gdocsbackend.py",
>>>>>>> line 144, in get
>>>>>>>     % (remote_filename, self.folder.title.text, str(e)), raise_errors)
>>>>>>>   File
>>>>>>> "/usr/lib/python2.7/dist-packages/duplicity/backends/gdocsbackend.py",
>>>>>>> line 182, in __handle_error
>>>>>>>     raise BackendException(message)
>>>>>>>  BackendException: Failed to download file
>>>>>>> 'duplicity-full-signatures.20150205T170035Z.sigtar.gpg' in remote
>>>>>>> folder 'i7-bigstore': join() result is too long for a Python string
>>>>>>>
>>>>>>> It'll then try again, and give the same error, at which point I hit
>>>>>>> control-C.
>>>>>>>
>>>>>>> Version 0.7.01 fails more quickly:
>>>>>>>
>>>>>>> $ apt-cache policy duplicity
>>>>>>> duplicity:
>>>>>>>   Installed: 0.7.01-0ubuntu0ppa1063~ubuntu12.04.1
>>>>>>>   Candidate: 0.7.01-0ubuntu0ppa1063~ubuntu12.04.1
>>>>>>>   Version table:
>>>>>>>  *** 0.7.01-0ubuntu0ppa1063~ubuntu12.04.1 0
>>>>>>>         500 http://ppa.launchpad.net/duplicity-team/ppa/ubuntu/
>>>>>>> precise/main i386 Packages
>>>>>>>         100 /var/lib/dpkg/status
>>>>>>>      0.6.18-0ubuntu3.5 0
>>>>>>>         500 http://ie.archive.ubuntu.com/ubuntu/ precise-updates/main
>>>>>>> i386 Packages
>>>>>>>      0.6.18-0ubuntu3 0
>>>>>>>         500 http://ie.archive.ubuntu.com/ubuntu/ precise/main i386
>>>>>>> Packages
>>>>>>> $ duplicity -V
>>>>>>> duplicity 0.7.01
>>>>>>> $ duplicity verify --verbosity '8' --exclude-globbing-filelist
>>>>>>> /home/rupert/.duply/i7-bigstore/exclude
>>>>>>> gdocs://address@hidden/backups/duplicity/i7-bigstore
>>>>>>> /mnt/bigstore
>>>>>>> Using archive dir: /home/rupert/.cache/duplicity/xxxxxxxxxxxxxxxxxxxxx
>>>>>>> Using backup name: xxxxxxxxxxxxxxxxxxxxx
>>>>>>> Import of duplicity.backends.azurebackend Succeeded
>>>>>>> Import of duplicity.backends.botobackend Succeeded
>>>>>>> Import of duplicity.backends.cfbackend Succeeded
>>>>>>> Import of duplicity.backends.copycombackend Succeeded
>>>>>>> Import of duplicity.backends.dpbxbackend Succeeded
>>>>>>> Import of duplicity.backends.gdocsbackend Succeeded
>>>>>>> Import of duplicity.backends.giobackend Succeeded
>>>>>>> Import of duplicity.backends.hsibackend Succeeded
>>>>>>> Import of duplicity.backends.hubicbackend Succeeded
>>>>>>> Import of duplicity.backends.imapbackend Succeeded
>>>>>>> Import of duplicity.backends.lftpbackend Succeeded
>>>>>>> Import of duplicity.backends.localbackend Succeeded
>>>>>>> Import of duplicity.backends.megabackend Succeeded
>>>>>>> Import of duplicity.backends.ncftpbackend Succeeded
>>>>>>> Import of duplicity.backends.onedrivebackend Failed: No module named
>>>>>>> requests
>>>>>>> Import of duplicity.backends.par2backend Succeeded
>>>>>>> Import of duplicity.backends.pydrivebackend Succeeded
>>>>>>> Import of duplicity.backends.rsyncbackend Succeeded
>>>>>>> Import of duplicity.backends.ssh_paramiko_backend Succeeded
>>>>>>> Import of duplicity.backends.ssh_pexpect_backend Succeeded
>>>>>>> Import of duplicity.backends.swiftbackend Succeeded
>>>>>>> Import of duplicity.backends.sxbackend Succeeded
>>>>>>> Import of duplicity.backends.tahoebackend Succeeded
>>>>>>> Import of duplicity.backends.webdavbackend Succeeded
>>>>>>> Using temporary directory /tmp/duplicity-lCUBfI-tempdir
>>>>>>> Traceback (most recent call last):
>>>>>>>   File "/usr/bin/duplicity", line 1497, in <module>
>>>>>>>     with_tempdir(main)
>>>>>>>   File "/usr/bin/duplicity", line 1491, in with_tempdir
>>>>>>>     fn()
>>>>>>>   File "/usr/bin/duplicity", line 1324, in main
>>>>>>>     action = commandline.ProcessCommandLine(sys.argv[1:])
>>>>>>>   File "/usr/lib/python2.7/dist-packages/duplicity/commandline.py",
>>>>>>> line 1056, in ProcessCommandLine
>>>>>>>     backup, local_pathname = set_backend(args[0], args[1])
>>>>>>>   File "/usr/lib/python2.7/dist-packages/duplicity/commandline.py",
>>>>>>> line 949, in set_backend
>>>>>>>     globals.backend = backend.get_backend(bend)
>>>>>>>   File "/usr/lib/python2.7/dist-packages/duplicity/backend.py", line
>>>>>>> 221, in get_backend
>>>>>>>     obj = get_backend_object(url_string)
>>>>>>>   File "/usr/lib/python2.7/dist-packages/duplicity/backend.py", line
>>>>>>> 207, in get_backend_object
>>>>>>>     return factory(pu)
>>>>>>>   File
>>>>>>> "/usr/lib/python2.7/dist-packages/duplicity/backends/gdocsbackend.py",
>>>>>>> line 61, in __init__
>>>>>>>     entries = self._fetch_entries(parent_folder_id, 'folder',
>>>>>>> folder_name)
>>>>>>>   File
>>>>>>> "/usr/lib/python2.7/dist-packages/duplicity/backends/gdocsbackend.py",
>>>>>>> line 161, in _fetch_entries
>>>>>>>     entries = self.client.get_all_resources(uri=uri)
>>>>>>> AttributeError: 'DocsClient' object has no attribute
>>>>>>> 'get_all_resources'
>>>>>>>
>>>>>>> Are these two independent bugs? Is there a workaround or fix?
>>>>>>>
>>>>>>
>>>>>> yes, it looks like two different bugs with two different error
>>>>>> messages.
>>>>>>
>>>>>> first try latest stable duplicity 0.6.25.. there were changes in the
>>>>>> gdocs backend since your version.
>>>>>>
>>>>>> if that fails - can you try downloading your whole backups to a local
>>>>>> folder and try to restore from there?
>>>>>>
>>>>>> ..ede/duply.net
>>>>>>
>>>>>> _______________________________________________
>>>>>> Duplicity-talk mailing list
>>>>>> address@hidden
>>>>>> https://lists.nongnu.org/mailman/listinfo/duplicity-talk
>>>>>
>>>>> I had tried 0.6.25 too, and it was giving the same get_all_resources
>>>>> error as 0.7.01. However since then I used pip to upgrade gdata, and
>>>>> 0.7.01 gives me very similar output to my original error from 0.6.18:
>>>>>
>>>>> $ duplicity verify --verbosity '9' --exclude-globbing-filelist
>>>>> /home/rupert/.duply/i7-bigstore/exclude
>>>>> gdocs://address@hidden/backups/duplicity/i7-bigstore
>>>>> /mnt/bigstore
>>>>> Using archive dir:
>>>>> /home/rupert/.cache/duplicity/xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
>>>>> Using backup name: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
>>>>> Import of duplicity.backends.azurebackend Succeeded
>>>>> Import of duplicity.backends.botobackend Succeeded
>>>>> Import of duplicity.backends.cfbackend Succeeded
>>>>> Import of duplicity.backends.copycombackend Succeeded
>>>>> Import of duplicity.backends.dpbxbackend Succeeded
>>>>> Import of duplicity.backends.gdocsbackend Succeeded
>>>>> Import of duplicity.backends.giobackend Succeeded
>>>>> Import of duplicity.backends.hsibackend Succeeded
>>>>> Import of duplicity.backends.hubicbackend Succeeded
>>>>> Import of duplicity.backends.imapbackend Succeeded
>>>>> Import of duplicity.backends.lftpbackend Succeeded
>>>>> Import of duplicity.backends.localbackend Succeeded
>>>>> Import of duplicity.backends.megabackend Succeeded
>>>>> Import of duplicity.backends.ncftpbackend Succeeded
>>>>> Import of duplicity.backends.onedrivebackend Failed: No module named
>>>>> requests
>>>>> Import of duplicity.backends.par2backend Succeeded
>>>>> Import of duplicity.backends.pydrivebackend Succeeded
>>>>> Import of duplicity.backends.rsyncbackend Succeeded
>>>>> Import of duplicity.backends.ssh_paramiko_backend Succeeded
>>>>> Import of duplicity.backends.ssh_pexpect_backend Succeeded
>>>>> Import of duplicity.backends.swiftbackend Succeeded
>>>>> Import of duplicity.backends.sxbackend Succeeded
>>>>> Import of duplicity.backends.tahoebackend Succeeded
>>>>> Import of duplicity.backends.webdavbackend Succeeded
>>>>> Reading globbing filelist /home/rupert/.duply/i7-bigstore/exclude
>>>>> Main action: verify
>>>>>
>>>>> ================================================================================
>>>>> duplicity 0.7.01 ($reldate)
>>>>> Args: /usr/bin/duplicity verify --verbosity 9
>>>>> --exclude-globbing-filelist /home/rupert/.duply/i7-bigstore/exclude
>>>>> gdocs://address@hidden/backups/duplicity/i7-bigstore
>>>>> /mnt/bigstore
>>>>> Linux levene-i7desktop 3.13.0-44-generic #73~precise1-Ubuntu SMP Wed
>>>>> Dec 17 00:38:38 UTC 2014 i686 i686
>>>>> /usr/bin/python 2.7.3 (default, Dec 18 2014, 19:03:52)
>>>>> [GCC 4.6.3]
>>>>>
>>>>> ================================================================================
>>>>> Synchronizing remote metadata to local cache...
>>>>> Copying duplicity-full-signatures.20150205T170035Z.sigtar.gpg to local
>>>>> cache.
>>>>> Using temporary directory /tmp/duplicity-L9vwHt-tempdir
>>>>> Registering (mktemp) temporary file
>>>>> /tmp/duplicity-L9vwHt-tempdir/mktemp-_vVr3t-1
>>>>> Backtrace of previous error: Traceback (innermost last):
>>>>>   File "/usr/lib/python2.7/dist-packages/duplicity/backend.py", line
>>>>> 361, in inner_retry
>>>>>     return fn(self, *args)
>>>>>   File "/usr/lib/python2.7/dist-packages/duplicity/backend.py", line
>>>>> 534, in get
>>>>>     self.backend._get(remote_filename, local_path)
>>>>>   File
>>>>> "/usr/lib/python2.7/dist-packages/duplicity/backends/gdocsbackend.py",
>>>>> line 110, in _get
>>>>>     self.client.DownloadResource(entry, local_path.name)
>>>>>   File "/usr/local/lib/python2.7/dist-packages/gdata/docs/client.py",
>>>>> line 369, in download_resource
>>>>>     self._download_file(uri, file_path, **kwargs)
>>>>>   File "/usr/local/lib/python2.7/dist-packages/gdata/docs/client.py",
>>>>> line 468, in _download_file
>>>>>     f.write(self._get_content(uri, **kwargs))
>>>>>   File "/usr/local/lib/python2.7/dist-packages/gdata/docs/client.py",
>>>>> line 451, in _get_content
>>>>>     return server_response.read()
>>>>>   File "/usr/lib/python2.7/httplib.py", line 541, in read
>>>>>     return self._read_chunked(amt)
>>>>>   File "/usr/lib/python2.7/httplib.py", line 624, in _read_chunked
>>>>>     return ''.join(value)
>>>>>  OverflowError: join() result is too long for a Python string
>>>>>
>>>>> Attempt 1 failed. OverflowError: join() result is too long for a Python
>>>>> string
>>>>> Backtrace of previous error: Traceback (innermost last):
>>>>>   File "/usr/lib/python2.7/dist-packages/duplicity/backend.py", line
>>>>> 361, in inner_retry
>>>>>     return fn(self, *args)
>>>>>   File "/usr/lib/python2.7/dist-packages/duplicity/backend.py", line
>>>>> 534, in get
>>>>>     self.backend._get(remote_filename, local_path)
>>>>>   File
>>>>> "/usr/lib/python2.7/dist-packages/duplicity/backends/gdocsbackend.py",
>>>>> line 110, in _get
>>>>>     self.client.DownloadResource(entry, local_path.name)
>>>>>   File "/usr/local/lib/python2.7/dist-packages/gdata/docs/client.py",
>>>>> line 369, in download_resource
>>>>>     self._download_file(uri, file_path, **kwargs)
>>>>>   File "/usr/local/lib/python2.7/dist-packages/gdata/docs/client.py",
>>>>> line 468, in _download_file
>>>>>     f.write(self._get_content(uri, **kwargs))
>>>>>   File "/usr/local/lib/python2.7/dist-packages/gdata/docs/client.py",
>>>>> line 451, in _get_content
>>>>>     return server_response.read()
>>>>>   File "/usr/lib/python2.7/httplib.py", line 541, in read
>>>>>     return self._read_chunked(amt)
>>>>>   File "/usr/lib/python2.7/httplib.py", line 590, in _read_chunked
>>>>>     value.append(self._safe_read(chunk_left))
>>>>>   File "/usr/lib/python2.7/httplib.py", line 647, in _safe_read
>>>>>     chunk = self.fp.read(min(amt, MAXAMOUNT))
>>>>>   File "/usr/lib/python2.7/socket.py", line 396, in read
>>>>>     buf.write(data)
>>>>>  MemoryError: out of memory
>>>>>
>>>>> Attempt 2 failed. MemoryError: out of memory
>>>>> Backtrace of previous error: Traceback (innermost last):
>>>>>   File "/usr/lib/python2.7/dist-packages/duplicity/backend.py", line
>>>>> 361, in inner_retry
>>>>>     return fn(self, *args)
>>>>>   File "/usr/lib/python2.7/dist-packages/duplicity/backend.py", line
>>>>> 534, in get
>>>>>     self.backend._get(remote_filename, local_path)
>>>>>   File
>>>>> "/usr/lib/python2.7/dist-packages/duplicity/backends/gdocsbackend.py",
>>>>> line 110, in _get
>>>>>     self.client.DownloadResource(entry, local_path.name)
>>>>>   File "/usr/local/lib/python2.7/dist-packages/gdata/docs/client.py",
>>>>> line 369, in download_resource
>>>>>     self._download_file(uri, file_path, **kwargs)
>>>>>   File "/usr/local/lib/python2.7/dist-packages/gdata/docs/client.py",
>>>>> line 468, in _download_file
>>>>>     f.write(self._get_content(uri, **kwargs))
>>>>>   File "/usr/local/lib/python2.7/dist-packages/gdata/docs/client.py",
>>>>> line 451, in _get_content
>>>>>     return server_response.read()
>>>>>   File "/usr/lib/python2.7/httplib.py", line 541, in read
>>>>>     return self._read_chunked(amt)
>>>>>   File "/usr/lib/python2.7/httplib.py", line 624, in _read_chunked
>>>>>     return ''.join(value)
>>>>>  OverflowError: join() result is too long for a Python string
>>>>>
>>>>> Attempt 3 failed. OverflowError: join() result is too long for a Python
>>>>> string
>>>>> Backtrace of previous error: Traceback (innermost last):
>>>>>   File "/usr/lib/python2.7/dist-packages/duplicity/backend.py", line
>>>>> 361, in inner_retry
>>>>>     return fn(self, *args)
>>>>>   File "/usr/lib/python2.7/dist-packages/duplicity/backend.py", line
>>>>> 534, in get
>>>>>     self.backend._get(remote_filename, local_path)
>>>>>   File
>>>>> "/usr/lib/python2.7/dist-packages/duplicity/backends/gdocsbackend.py",
>>>>> line 110, in _get
>>>>>     self.client.DownloadResource(entry, local_path.name)
>>>>>   File "/usr/local/lib/python2.7/dist-packages/gdata/docs/client.py",
>>>>> line 369, in download_resource
>>>>>     self._download_file(uri, file_path, **kwargs)
>>>>>   File "/usr/local/lib/python2.7/dist-packages/gdata/docs/client.py",
>>>>> line 468, in _download_file
>>>>>     f.write(self._get_content(uri, **kwargs))
>>>>>   File "/usr/local/lib/python2.7/dist-packages/gdata/docs/client.py",
>>>>> line 451, in _get_content
>>>>>     return server_response.read()
>>>>>   File "/usr/lib/python2.7/httplib.py", line 541, in read
>>>>>     return self._read_chunked(amt)
>>>>>   File "/usr/lib/python2.7/httplib.py", line 590, in _read_chunked
>>>>>     value.append(self._safe_read(chunk_left))
>>>>>   File "/usr/lib/python2.7/httplib.py", line 647, in _safe_read
>>>>>     chunk = self.fp.read(min(amt, MAXAMOUNT))
>>>>>   File "/usr/lib/python2.7/socket.py", line 400, in read
>>>>>     buf.write(data)
>>>>>  MemoryError: out of memory
>>>>>
>>>>> Attempt 4 failed. MemoryError: out of memory
>>>>> Backtrace of previous error: Traceback (innermost last):
>>>>>   File "/usr/lib/python2.7/dist-packages/duplicity/backend.py", line
>>>>> 361, in inner_retry
>>>>>     return fn(self, *args)
>>>>>   File "/usr/lib/python2.7/dist-packages/duplicity/backend.py", line
>>>>> 534, in get
>>>>>     self.backend._get(remote_filename, local_path)
>>>>>   File
>>>>> "/usr/lib/python2.7/dist-packages/duplicity/backends/gdocsbackend.py",
>>>>> line 110, in _get
>>>>>     self.client.DownloadResource(entry, local_path.name)
>>>>>   File "/usr/local/lib/python2.7/dist-packages/gdata/docs/client.py",
>>>>> line 369, in download_resource
>>>>>     self._download_file(uri, file_path, **kwargs)
>>>>>   File "/usr/local/lib/python2.7/dist-packages/gdata/docs/client.py",
>>>>> line 468, in _download_file
>>>>>     f.write(self._get_content(uri, **kwargs))
>>>>>   File "/usr/local/lib/python2.7/dist-packages/gdata/docs/client.py",
>>>>> line 451, in _get_content
>>>>>     return server_response.read()
>>>>>   File "/usr/lib/python2.7/httplib.py", line 541, in read
>>>>>     return self._read_chunked(amt)
>>>>>   File "/usr/lib/python2.7/httplib.py", line 624, in _read_chunked
>>>>>     return ''.join(value)
>>>>>  OverflowError: join() result is too long for a Python string
>>>>>
>>>>> Giving up after 5 attempts. OverflowError: join() result is too long
>>>>> for a Python string
>>>>> Releasing lockfile <lockfile.LinkFileLock instance at 0x9d0986c>
>>>>> Removing still remembered temporary file
>>>>> /tmp/duplicity-L9vwHt-tempdir/mktemp-_vVr3t-1
>>>>> $
>>>>>
>>>>> I am running 32-bit linux with 4gb RAM and 4gb swap.
>>>>>
>>>>> I'll try downloading the backup to local storage as you suggest.
>>>>>
>>>>
>>>> please do.. looks like httplib fails on 2GB files and there isn't much we
>>>> can do about that.
>>>>
>>>> Mike, Ken: any idea?
>>>>
>>>> ..ede/duply.net
>>>>
>>>> _______________________________________________
>>>> Mailing list: https://launchpad.net/~duplicity-team
>>>> Post to     : address@hidden
>>>> Unsubscribe : https://launchpad.net/~duplicity-team
>>>> More help   : https://help.launchpad.net/ListHelp
>>>
>>>
>>>
>>> _______________________________________________
>>> Duplicity-talk mailing list
>>> address@hidden
>>> https://lists.nongnu.org/mailman/listinfo/duplicity-talk
>>>
>
> _______________________________________________
> Duplicity-talk mailing list
> address@hidden
> https://lists.nongnu.org/mailman/listinfo/duplicity-talk