[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: [Duplicity-talk] webdavs and large sync volume failing
From: |
address@hidden |
Subject: |
Re: [Duplicity-talk] webdavs and large sync volume failing |
Date: |
Wed, 20 Apr 2016 11:02:01 +0200 |
Hi Edgar,
no news on this, I have not further tested it with lftp+webdavs backend as I am
now using webdav directly.
Though I need to admit I have not run these huge backups any more as the
increments now are much smaller.
Anything I could still test to support the lftp+webdavs backend trouble
shooting?
Gruss,
Wolle
> Am 17.04.2016 um 16:58 schrieb address@hidden:
>
> hey Wolle,
>
> any news on this issue? i'd assume a timeout on the webserver wrt. the amount
> of files that it has to list. if so there is nothing duplicity can do. you
> could to raise the volume size to minimize the number of volumes, but htat
> wouldn't guarantee anything as the number of files will increase on the
> backend.
>
> ..ede/duply.net
>
> On 06.02.2016 09:09, address@hidden wrote:
>> Hi,
>>
>> I have reproduced the error with verbosity 9 and sent the logfiles to
>> ede/duply.net.
>> As stated in that mail it looks like this is a memory issue
>> (/var/log/messages shows that oom-killer kicks in)
>>
>> In parallel I have now upgraded my system to
>> duplicity 0.7.06
>> duply 1.11.1
>>
>> and re-run the backup with backend lftp+webdavs as proposed below. This
>> works fine without errors
>>
>> but
>>
>> If I check the status of this backup via the same backend lftp+webdavs it
>> shows no backup present.
>> If I change back to webdavs backend for the status command I can see the 1
>> successfull full backup I created with the lftp+webdavs backend.
>> I tested the same with a small backup repository, here the backend
>> lftp+webdavs works fine both for the backup and the status command
>>
>> A quick investigation I did shows that the status command for the big backup
>> (roughly 14GB of photos) has an issue when retrieving the file info from the
>> webdav server. There is no data returned - in the log below there should be
>> a long file list after STDOUT: but this is empty.
>>
>> ——— snip from log ——
>> CMD: lftp -c 'source
>> '/home/backup/tmp/duplicity-uzDqze-tempdir/mkstemp-yXoZe9-1'; cd
>> 'backup/photo/1971-2005/' || exit 0; ls'
>> Reading results of 'lftp -c 'source
>> '/home/backup/tmp/duplicity-uzDqze-tempdir/mkstemp-yXoZe9-1'; cd
>> 'backup/photo/1971-2005/' || exit 0; ls''
>> STDERR:
>> ---- Resolving host address...
>> ---- 1 address found: xxxxxxx
>> ---- Connecting to sd2dav. xxxxxxx port 443
>> ---- Sending request...
>> ---> PROPFIND / HTTP/1.1
>> ---> Host: sd2dav. xxxxxxx
>> ---> User-Agent: lftp/4.6.0
>> ---> Accept: */*
>> ---> Depth: 0
>> ---> Authorization: Basic xxxxxxx
>> ---> Connection: keep-alive
>> --->
>> Certificate: xxxxxxx
>> Trusted
>> <--- HTTP/1.1 207 Multi-Status
>> <--- Date: Sat, 06 Feb 2016 06:55:09 GMT
>> <--- Server: Apache
>> <--- ETag: "xxxxxxx ="
>> <--- Content-Length: 1857
>> <--- Content-Type: text/xml; charset="utf-8"
>> <--- Vary: Accept-Encoding
>> <--- Keep-Alive: timeout=3, max=100
>> <--- Connection: Keep-Alive
>> <---
>> ---- Receiving body...
>> ---- Hit EOF
>> ---- Closing HTTP connection
>> ---- Connecting to sd2dav. xxxxxxx port 443
>> ---- Sending request...
>> ---> PROPFIND /backup/photo/1971-2005/ HTTP/1.1
>> ---> Host: sd2dav. xxxxxxx
>> ---> User-Agent: lftp/4.6.0
>> ---> Accept: */*
>> ---> Depth: 0
>> ---> Authorization: Basic xxxxxxx
>> ---> Connection: keep-alive
>> --->
>> Certificate: xxxxxxx
>> Trusted
>> <--- HTTP/1.1 207 Multi-Status
>> <--- Date: Sat, 06 Feb 2016 06:55:19 GMT
>> <--- Server: Apache
>> <--- ETag: "xxxxxxx ="
>> <--- Content-Length: 1368
>> <--- Content-Type: text/xml; charset="utf-8"
>> <--- Vary: Accept-Encoding
>> <--- Keep-Alive: timeout=3, max=100
>> <--- Connection: Keep-Alive
>> <---
>> ---- Receiving body...
>> ---- Hit EOF
>> ---- Closing HTTP connection
>> ---- Connecting to sd2dav. xxxxxxx port 443
>> ---- Sending request...
>> ---> GET /backup/photo/1971-2005/ HTTP/1.1
>> ---> Host: sd2dav. xxxxxxx
>> ---> User-Agent: lftp/4.6.0
>> ---> Accept: */*
>> ---> Authorization: Basic xxxxxxx
>> ---> Connection: keep-alive
>> --->
>> Certificate: xxxxxxx
>> Trusted
>> <--- HTTP/1.1 200 OK
>> <--- Date: Sat, 06 Feb 2016 06:55:20 GMT
>> <--- Server: Apache
>> <--- Last-Modified: Mon, 01 Feb 2016 19:44:00 GMT
>> <--- ETag: "xxxxxxx ="
>> <--- Accept-Ranges: bytes
>> <--- Content-Length: 0
>> <--- Content-Type: text/html; charset="UTF-8"
>> <--- Vary: Accept-Encoding
>> <--- Keep-Alive: timeout=3, max=100
>> <--- Connection: Keep-Alive
>> <---
>> ---- Receiving body...
>> ---- Received all
>> ---- Closing HTTP connection
>>
>> STDOUT:
>>
>> Local and Remote metadata are synchronized, no sync needed.
>>
>> ——— snip end from log ——
>>
>>> Am 02.02.2016 um 21:27 schrieb address@hidden:
>>>
>>> On 02.02.2016 21:18, address@hidden wrote:
>>>> Hi,
>>>>
>>>> I am trying to use Duplicity and duply to backup appr 60GB data to webdavs
>>>> storage.
>>>>
>>>> The script is running on a raspberrypi with raspbian jessie
>>>>
>>>> Small backups are working.
>>>> I have tried to run this backup in one go, it failed; I then split into
>>>> 15GB chunks it failed again with exit code 137
>>>>
>>>> Any ideas what to do / where to look for?
>>>
>>> can you run duplicity in max. verbosity '-v9' and send me the _complete_
>>> output privately?
>>>
>>>> Is the webdavs access not stable enough for such backup?
>>>
>>> should make no difference. the backup is split into volumes of the same
>>> size anyway.
>>>
>>>> Is it maybe a performance issue with the raspberry
>>>>
>>>> btw I started the backup script via a cron job, it was then running
>>>> several hours before it failed.
>>>>
>>>
>>> what's your duplicity version? with the latest duplicity you can install
>>> lftp and use that as alernative webdav backend via lftp+webdav://
>>>
>>> ..ede/duply.net
>>>
>>>
>>
>>
>> _______________________________________________
>> Duplicity-talk mailing list
>> address@hidden
>> https://lists.nongnu.org/mailman/listinfo/duplicity-talk
>>
>
> _______________________________________________
> Duplicity-talk mailing list
> address@hidden
> https://lists.nongnu.org/mailman/listinfo/duplicity-talk