[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: [Duplicity-talk] Show volume Duplicity is working on
From: |
Remy van Elst |
Subject: |
Re: [Duplicity-talk] Show volume Duplicity is working on |
Date: |
Sat, 18 Oct 2014 15:13:24 +0200 |
User-agent: |
Mozilla/5.0 (X11; Linux x86_64; rv:31.0) Gecko/20100101 Thunderbird/31.2.0 |
On 10/18/14 14:49, address@hidden wrote:
> 1. exactly what are you trying to achieve?
I have a server with 1.6 TB of data that takes almost a week to do a
full backup over an umetered gbit connection. It backs up to Rackspace
Openstack Swift, with a default 25MB volume size.
The backup becomes inconsistent because stuff changes a lot on that
server. It has multiple +100 GB databases running. I sent an email
earlier to this list regarding --max-blocksize, but did not receive a
response yet.
(http://lists.nongnu.org/archive/html/duplicity-talk/2014-10/msg00002.html)
With a volume size of 250 MB, or with 2048 MB the upload still takes
more than 3 days sadly.
What I want to achieve is solving the long running backup issue.
> 2. why do you need to know which volume is uploaded?
I was troubleshooting with strace and saw the volume number. I know the
volume size, so with the current volume number I know which one is being
uploaded (async upload enabled). With that data I should be able to
estimate how far the backup is.
> 3. duplicity is written in python script language. you can easily patch your
> local duplicity to print out the current volume name.
>
Is also an option, however I would use that as a last resort. I'm using
Duplicity 0.6.24, compiled from source on Debian 7.
> ..ede/duply.net
>
> On 18.10.2014 14:34, Remy van Elst wrote:
>> I've not tried it with more verbosity yet. I did write a simple script
>> which shows some output, but it requires strace. Example output:
>>
>> Diskspace used: 1591229 MiB / 1553 GiB
>> Duplicity volume: 10127
>> Amount uploaded: 253175 MiB / 247 GiB.
>> 15.91% done, 1306 GiB left to upload of 1553 GiB.
>>
>>
>> Script:
>>
>> #!/bin/bash
>> if [[ -z "$(pgrep strace)" ]]; then
>>
>> # Attach strace to Duplicity and log writes to /tmp/duplicity.strace
>> # look for lines like write(6, "Volume 8113:\n StartingPath
>> "..., 246) = 246
>> strace -e trace=write -o /tmp/duplicity.strace -p $(pgrep duplicity) &
>> sleep 5
>> fi
>>
>> if [[ -f "/tmp/duplicity.strace" ]]; then
>> sleep 5
>>
>> TOTAL_DISK_USED="$(df -BM --total 2>/dev/null| awk '/total/ {print
>> $3}' | sed -e 's/M//g')"
>>
>> DUPLICITY_VOL_DONE="$(grep -a -oE 'Volume [0-9]{1,6}'
>> /tmp/duplicity.strace | grep -a -oE '[0-9]{1,6}' | tail -n 1)"
>> if [[ -z "${DUPLICITY_VOL_DONE}" ]]; then
>> echo "Error reading current volume"
>> exit 1
>> fi
>>
>> DUPLICITY_MiB_DONE="$(( ${DUPLICITY_VOL_DONE} * 25 ))"
>>
>> DUPLICITY_GiB_DONE="$(( ( ${DUPLICITY_VOL_DONE} * 25 ) / 1024))"
>>
>> MiB_LEFT="$(( ${TOTAL_DISK_USED} - ${DUPLICITY_MiB_DONE} ))"
>>
>> PERC_DONE="$(awk "BEGIN {printf
>> \"%.2f\",(${DUPLICITY_MiB_DONE}/${TOTAL_DISK_USED})*100}")"
>>
>> if [[ -n "${TOTAL_DISK_USED}" ]]; then
>> echo "Diskspace used: ${TOTAL_DISK_USED} MiB / $((
>> ${TOTAL_DISK_USED} / 1024 )) GiB"
>> fi
>>
>> if [[ -n "${DUPLICITY_VOL_DONE}" ]]; then
>> echo "Duplicity volume: ${DUPLICITY_VOL_DONE}"
>> fi
>>
>> if [[ -n "${DUPLICITY_MiB_DONE}" ]]; then
>> echo -n "Amount uploaded: ${DUPLICITY_MiB_DONE} MiB / "
>> echo "${DUPLICITY_GiB_DONE} GiB."
>> fi
>>
>> echo "${PERC_DONE}% done, $(( ${MiB_LEFT} / 1024 )) GiB left to
>> upload of $(( ${TOTAL_DISK_USED} / 1024 )) GiB."
>>
>> echo >/tmp/duplicity.strace
>>
>> fi
>>
>>
>>
>>
>> On 10/18/14 14:17, address@hidden wrote:
>>> On 18.10.2014 10:11, Remy van Elst wrote:
>>>> When I strace attach to duplicity I can see it writing out lines like
>>>> the following:
>>>>
>>>> [pid 22316] write(6, "Volume 7949:\n StartingPath "..., 246) = 246
>>>>
>>>> (strace -e trace=write -p <pid>)
>>>>
>>>> That volume number combined with the size of the disk and the volume
>>>> size gives me a better idea how far Duplicity is with it's backup.
>>>>
>>>> How do I access this information without using strace? Is this logged
>>>> somewhere?
>>>>
>>>
>>> did you have a look at the manpage?
>>> http://duplicity.nongnu.org/duplicity.1.html
>>> raising the log level '--verbosity' might help.
>>>
>>> ..ede/duply.net
>>>
>>> _______________________________________________
>>> Duplicity-talk mailing list
>>> address@hidden
>>> https://lists.nongnu.org/mailman/listinfo/duplicity-talk
>>>
>>>
>>>
>>> _______________________________________________
>>> Duplicity-talk mailing list
>>> address@hidden
>>> https://lists.nongnu.org/mailman/listinfo/duplicity-talk
>
> _______________________________________________
> Duplicity-talk mailing list
> address@hidden
> https://lists.nongnu.org/mailman/listinfo/duplicity-talk
>
0x1B7F88DC.asc
Description: application/pgp-keys
signature.asc
Description: OpenPGP digital signature