Re: [Duplicity-talk] verify fails on 0.6.18 and 0.7.01

[Rupert Levene]

On 8 February 2015 at 14:32,  <address@hidden> wrote:
> On 08.02.2015 15:05, Rupert Levene wrote:
>> Hi,
>>
>> I have made a large to gdocs using 0.6.18 from Ubuntu 12.04. A 2GB
>> file duplicity-full-signatures.20150205T170035Z.sigtar.gpg was
>> created, and duplicity chokes on this. It pauses for a while (I guess
>> while it downloads the file) and then says
>>
>> Attempt 1 failed: BackendException: Failed to download file
>> 'duplicity-full-signatures.20150205T170035Z.sigtar.gpg' in remote
>> folder 'i7-bigstore': join() result is too long for a Python string
>> Backtrace of previous error: Traceback (innermost last):
>>   File "/usr/lib/python2.7/dist-packages/duplicity/backend.py", line
>> 311, in iterate
>>     return fn(*args, **kwargs)
>>   File "/usr/lib/python2.7/dist-packages/duplicity/backends/gdocsbackend.py",
>> line 144, in get
>>     % (remote_filename, self.folder.title.text, str(e)), raise_errors)
>>   File "/usr/lib/python2.7/dist-packages/duplicity/backends/gdocsbackend.py",
>> line 182, in __handle_error
>>     raise BackendException(message)
>>  BackendException: Failed to download file
>> 'duplicity-full-signatures.20150205T170035Z.sigtar.gpg' in remote
>> folder 'i7-bigstore': join() result is too long for a Python string
>>
>> It'll then try again, and give the same error, at which point I hit 
>> control-C.
>>
>> Version 0.7.01 fails more quickly:
>>
>> $ apt-cache policy duplicity
>> duplicity:
>>   Installed: 0.7.01-0ubuntu0ppa1063~ubuntu12.04.1
>>   Candidate: 0.7.01-0ubuntu0ppa1063~ubuntu12.04.1
>>   Version table:
>>  *** 0.7.01-0ubuntu0ppa1063~ubuntu12.04.1 0
>>         500 http://ppa.launchpad.net/duplicity-team/ppa/ubuntu/
>> precise/main i386 Packages
>>         100 /var/lib/dpkg/status
>>      0.6.18-0ubuntu3.5 0
>>         500 http://ie.archive.ubuntu.com/ubuntu/ precise-updates/main
>> i386 Packages
>>      0.6.18-0ubuntu3 0
>>         500 http://ie.archive.ubuntu.com/ubuntu/ precise/main i386 Packages
>> $ duplicity -V
>> duplicity 0.7.01
>> $ duplicity verify --verbosity '8' --exclude-globbing-filelist
>> /home/rupert/.duply/i7-bigstore/exclude
>> gdocs://address@hidden/backups/duplicity/i7-bigstore
>> /mnt/bigstore
>> Using archive dir: /home/rupert/.cache/duplicity/xxxxxxxxxxxxxxxxxxxxx
>> Using backup name: xxxxxxxxxxxxxxxxxxxxx
>> Import of duplicity.backends.azurebackend Succeeded
>> Import of duplicity.backends.botobackend Succeeded
>> Import of duplicity.backends.cfbackend Succeeded
>> Import of duplicity.backends.copycombackend Succeeded
>> Import of duplicity.backends.dpbxbackend Succeeded
>> Import of duplicity.backends.gdocsbackend Succeeded
>> Import of duplicity.backends.giobackend Succeeded
>> Import of duplicity.backends.hsibackend Succeeded
>> Import of duplicity.backends.hubicbackend Succeeded
>> Import of duplicity.backends.imapbackend Succeeded
>> Import of duplicity.backends.lftpbackend Succeeded
>> Import of duplicity.backends.localbackend Succeeded
>> Import of duplicity.backends.megabackend Succeeded
>> Import of duplicity.backends.ncftpbackend Succeeded
>> Import of duplicity.backends.onedrivebackend Failed: No module named requests
>> Import of duplicity.backends.par2backend Succeeded
>> Import of duplicity.backends.pydrivebackend Succeeded
>> Import of duplicity.backends.rsyncbackend Succeeded
>> Import of duplicity.backends.ssh_paramiko_backend Succeeded
>> Import of duplicity.backends.ssh_pexpect_backend Succeeded
>> Import of duplicity.backends.swiftbackend Succeeded
>> Import of duplicity.backends.sxbackend Succeeded
>> Import of duplicity.backends.tahoebackend Succeeded
>> Import of duplicity.backends.webdavbackend Succeeded
>> Using temporary directory /tmp/duplicity-lCUBfI-tempdir
>> Traceback (most recent call last):
>>   File "/usr/bin/duplicity", line 1497, in <module>
>>     with_tempdir(main)
>>   File "/usr/bin/duplicity", line 1491, in with_tempdir
>>     fn()
>>   File "/usr/bin/duplicity", line 1324, in main
>>     action = commandline.ProcessCommandLine(sys.argv[1:])
>>   File "/usr/lib/python2.7/dist-packages/duplicity/commandline.py",
>> line 1056, in ProcessCommandLine
>>     backup, local_pathname = set_backend(args[0], args[1])
>>   File "/usr/lib/python2.7/dist-packages/duplicity/commandline.py",
>> line 949, in set_backend
>>     globals.backend = backend.get_backend(bend)
>>   File "/usr/lib/python2.7/dist-packages/duplicity/backend.py", line
>> 221, in get_backend
>>     obj = get_backend_object(url_string)
>>   File "/usr/lib/python2.7/dist-packages/duplicity/backend.py", line
>> 207, in get_backend_object
>>     return factory(pu)
>>   File "/usr/lib/python2.7/dist-packages/duplicity/backends/gdocsbackend.py",
>> line 61, in __init__
>>     entries = self._fetch_entries(parent_folder_id, 'folder', folder_name)
>>   File "/usr/lib/python2.7/dist-packages/duplicity/backends/gdocsbackend.py",
>> line 161, in _fetch_entries
>>     entries = self.client.get_all_resources(uri=uri)
>> AttributeError: 'DocsClient' object has no attribute 'get_all_resources'
>>
>> Are these two independent bugs? Is there a workaround or fix?
>>
>
> yes, it looks like two different bugs with two different error messages.
>
> first try latest stable duplicity 0.6.25.. there were changes in the gdocs 
> backend since your version.
>
> if that fails - can you try downloading your whole backups to a local folder 
> and try to restore from there?
>
> ..ede/duply.net
>
> _______________________________________________
> Duplicity-talk mailing list
> address@hidden
> https://lists.nongnu.org/mailman/listinfo/duplicity-talk

I had tried 0.6.25 too, and it was giving the same get_all_resources
error as 0.7.01. However since then I used pip to upgrade gdata, and
0.7.01 gives me very similar output to my original error from 0.6.18:

$ duplicity verify --verbosity '9' --exclude-globbing-filelist
/home/rupert/.duply/i7-bigstore/exclude
gdocs://address@hidden/backups/duplicity/i7-bigstore
/mnt/bigstore
Using archive dir:
/home/rupert/.cache/duplicity/xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
Using backup name: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
Import of duplicity.backends.azurebackend Succeeded
Import of duplicity.backends.botobackend Succeeded
Import of duplicity.backends.cfbackend Succeeded
Import of duplicity.backends.copycombackend Succeeded
Import of duplicity.backends.dpbxbackend Succeeded
Import of duplicity.backends.gdocsbackend Succeeded
Import of duplicity.backends.giobackend Succeeded
Import of duplicity.backends.hsibackend Succeeded
Import of duplicity.backends.hubicbackend Succeeded
Import of duplicity.backends.imapbackend Succeeded
Import of duplicity.backends.lftpbackend Succeeded
Import of duplicity.backends.localbackend Succeeded
Import of duplicity.backends.megabackend Succeeded
Import of duplicity.backends.ncftpbackend Succeeded
Import of duplicity.backends.onedrivebackend Failed: No module named requests
Import of duplicity.backends.par2backend Succeeded
Import of duplicity.backends.pydrivebackend Succeeded
Import of duplicity.backends.rsyncbackend Succeeded
Import of duplicity.backends.ssh_paramiko_backend Succeeded
Import of duplicity.backends.ssh_pexpect_backend Succeeded
Import of duplicity.backends.swiftbackend Succeeded
Import of duplicity.backends.sxbackend Succeeded
Import of duplicity.backends.tahoebackend Succeeded
Import of duplicity.backends.webdavbackend Succeeded
Reading globbing filelist /home/rupert/.duply/i7-bigstore/exclude
Main action: verify
================================================================================
duplicity 0.7.01 ($reldate)
Args: /usr/bin/duplicity verify --verbosity 9
--exclude-globbing-filelist /home/rupert/.duply/i7-bigstore/exclude
gdocs://address@hidden/backups/duplicity/i7-bigstore
/mnt/bigstore
Linux levene-i7desktop 3.13.0-44-generic #73~precise1-Ubuntu SMP Wed
Dec 17 00:38:38 UTC 2014 i686 i686
/usr/bin/python 2.7.3 (default, Dec 18 2014, 19:03:52)
[GCC 4.6.3]
================================================================================
Synchronizing remote metadata to local cache...
Copying duplicity-full-signatures.20150205T170035Z.sigtar.gpg to local cache.
Using temporary directory /tmp/duplicity-L9vwHt-tempdir
Registering (mktemp) temporary file
/tmp/duplicity-L9vwHt-tempdir/mktemp-_vVr3t-1
Backtrace of previous error: Traceback (innermost last):
  File "/usr/lib/python2.7/dist-packages/duplicity/backend.py", line
361, in inner_retry
    return fn(self, *args)
  File "/usr/lib/python2.7/dist-packages/duplicity/backend.py", line 534, in get
    self.backend._get(remote_filename, local_path)
  File "/usr/lib/python2.7/dist-packages/duplicity/backends/gdocsbackend.py",
line 110, in _get
    self.client.DownloadResource(entry, local_path.name)
  File "/usr/local/lib/python2.7/dist-packages/gdata/docs/client.py",
line 369, in download_resource
    self._download_file(uri, file_path, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/gdata/docs/client.py",
line 468, in _download_file
    f.write(self._get_content(uri, **kwargs))
  File "/usr/local/lib/python2.7/dist-packages/gdata/docs/client.py",
line 451, in _get_content
    return server_response.read()
  File "/usr/lib/python2.7/httplib.py", line 541, in read
    return self._read_chunked(amt)
  File "/usr/lib/python2.7/httplib.py", line 624, in _read_chunked
    return ''.join(value)
 OverflowError: join() result is too long for a Python string

Attempt 1 failed. OverflowError: join() result is too long for a Python string
Backtrace of previous error: Traceback (innermost last):
  File "/usr/lib/python2.7/dist-packages/duplicity/backend.py", line
361, in inner_retry
    return fn(self, *args)
  File "/usr/lib/python2.7/dist-packages/duplicity/backend.py", line 534, in get
    self.backend._get(remote_filename, local_path)
  File "/usr/lib/python2.7/dist-packages/duplicity/backends/gdocsbackend.py",
line 110, in _get
    self.client.DownloadResource(entry, local_path.name)
  File "/usr/local/lib/python2.7/dist-packages/gdata/docs/client.py",
line 369, in download_resource
    self._download_file(uri, file_path, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/gdata/docs/client.py",
line 468, in _download_file
    f.write(self._get_content(uri, **kwargs))
  File "/usr/local/lib/python2.7/dist-packages/gdata/docs/client.py",
line 451, in _get_content
    return server_response.read()
  File "/usr/lib/python2.7/httplib.py", line 541, in read
    return self._read_chunked(amt)
  File "/usr/lib/python2.7/httplib.py", line 590, in _read_chunked
    value.append(self._safe_read(chunk_left))
  File "/usr/lib/python2.7/httplib.py", line 647, in _safe_read
    chunk = self.fp.read(min(amt, MAXAMOUNT))
  File "/usr/lib/python2.7/socket.py", line 396, in read
    buf.write(data)
 MemoryError: out of memory

Attempt 2 failed. MemoryError: out of memory
Backtrace of previous error: Traceback (innermost last):
  File "/usr/lib/python2.7/dist-packages/duplicity/backend.py", line
361, in inner_retry
    return fn(self, *args)
  File "/usr/lib/python2.7/dist-packages/duplicity/backend.py", line 534, in get
    self.backend._get(remote_filename, local_path)
  File "/usr/lib/python2.7/dist-packages/duplicity/backends/gdocsbackend.py",
line 110, in _get
    self.client.DownloadResource(entry, local_path.name)
  File "/usr/local/lib/python2.7/dist-packages/gdata/docs/client.py",
line 369, in download_resource
    self._download_file(uri, file_path, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/gdata/docs/client.py",
line 468, in _download_file
    f.write(self._get_content(uri, **kwargs))
  File "/usr/local/lib/python2.7/dist-packages/gdata/docs/client.py",
line 451, in _get_content
    return server_response.read()
  File "/usr/lib/python2.7/httplib.py", line 541, in read
    return self._read_chunked(amt)
  File "/usr/lib/python2.7/httplib.py", line 624, in _read_chunked
    return ''.join(value)
 OverflowError: join() result is too long for a Python string

Attempt 3 failed. OverflowError: join() result is too long for a Python string
Backtrace of previous error: Traceback (innermost last):
  File "/usr/lib/python2.7/dist-packages/duplicity/backend.py", line
361, in inner_retry
    return fn(self, *args)
  File "/usr/lib/python2.7/dist-packages/duplicity/backend.py", line 534, in get
    self.backend._get(remote_filename, local_path)
  File "/usr/lib/python2.7/dist-packages/duplicity/backends/gdocsbackend.py",
line 110, in _get
    self.client.DownloadResource(entry, local_path.name)
  File "/usr/local/lib/python2.7/dist-packages/gdata/docs/client.py",
line 369, in download_resource
    self._download_file(uri, file_path, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/gdata/docs/client.py",
line 468, in _download_file
    f.write(self._get_content(uri, **kwargs))
  File "/usr/local/lib/python2.7/dist-packages/gdata/docs/client.py",
line 451, in _get_content
    return server_response.read()
  File "/usr/lib/python2.7/httplib.py", line 541, in read
    return self._read_chunked(amt)
  File "/usr/lib/python2.7/httplib.py", line 590, in _read_chunked
    value.append(self._safe_read(chunk_left))
  File "/usr/lib/python2.7/httplib.py", line 647, in _safe_read
    chunk = self.fp.read(min(amt, MAXAMOUNT))
  File "/usr/lib/python2.7/socket.py", line 400, in read
    buf.write(data)
 MemoryError: out of memory

Attempt 4 failed. MemoryError: out of memory
Backtrace of previous error: Traceback (innermost last):
  File "/usr/lib/python2.7/dist-packages/duplicity/backend.py", line
361, in inner_retry
    return fn(self, *args)
  File "/usr/lib/python2.7/dist-packages/duplicity/backend.py", line 534, in get
    self.backend._get(remote_filename, local_path)
  File "/usr/lib/python2.7/dist-packages/duplicity/backends/gdocsbackend.py",
line 110, in _get
    self.client.DownloadResource(entry, local_path.name)
  File "/usr/local/lib/python2.7/dist-packages/gdata/docs/client.py",
line 369, in download_resource
    self._download_file(uri, file_path, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/gdata/docs/client.py",
line 468, in _download_file
    f.write(self._get_content(uri, **kwargs))
  File "/usr/local/lib/python2.7/dist-packages/gdata/docs/client.py",
line 451, in _get_content
    return server_response.read()
  File "/usr/lib/python2.7/httplib.py", line 541, in read
    return self._read_chunked(amt)
  File "/usr/lib/python2.7/httplib.py", line 624, in _read_chunked
    return ''.join(value)
 OverflowError: join() result is too long for a Python string

Giving up after 5 attempts. OverflowError: join() result is too long
for a Python string
Releasing lockfile <lockfile.LinkFileLock instance at 0x9d0986c>
Removing still remembered temporary file
/tmp/duplicity-L9vwHt-tempdir/mktemp-_vVr3t-1
$

I am running 32-bit linux with 4gb RAM and 4gb swap.

I'll try downloading the backup to local storage as you suggest.

Rupert