[Top][All Lists]
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: [Duplicity-talk] [patch] Use --num-retries option for Amazon S3
From: |
Kenneth Loafman |
Subject: |
Re: [Duplicity-talk] [patch] Use --num-retries option for Amazon S3 |
Date: |
Sun, 18 Nov 2007 15:11:38 -0600 |
User-agent: |
Thunderbird 1.5.0.14pre (X11/20071023) |
Thanks. Will get it out in the next release.
...Ken
Greg Hewgill wrote:
> The following patch implements the --num-retries option for the Boto
> Amazon S3 backend.
>
> I successfully backed up 30 GB of data using this patch. Previously, S3
> was failing often enough to prevent me from completing a full backup.
>
> Greg Hewgill
> http://hewgill.com
>
>
> ------------------------------------------------------------------------
>
> Only in duplicity-0.4.4.RC3.orig: build
> diff -ur duplicity-0.4.4.RC3.orig/src/backends.py
> duplicity-0.4.4.RC3/src/backends.py
> --- duplicity-0.4.4.RC3.orig/src/backends.py Tue Oct 2 12:06:51 2007
> +++ duplicity-0.4.4.RC3/src/backends.py Sun Nov 18 03:55:09 2007
> @@ -701,18 +701,36 @@
> def put(self, source_path, remote_filename=None):
> if not remote_filename:
> remote_filename = source_path.get_filename()
> - log.Log("Uploading %s to Amazon S3" % remote_filename, 5)
> key = self.key_class(self.bucket)
> key.key = remote_filename
> - key.set_contents_from_filename(source_path.name,
> - {'Content-Type': 'application/octet-stream'})
> + for n in range(1, globals.num_retries+1):
> + log.Log("Uploading %s to Amazon S3 (attempt #%d)" %
> (remote_filename, n), 5)
> + try:
> +
> key.set_contents_from_filename(source_path.name,
> + {'Content-Type':
> 'application/octet-stream'})
> + return
> + except:
> + pass
> + log.Log("Uploading %s failed (attempt #%d)" %
> (remote_filename, n), 1)
> + time.sleep(30)
> + log.Log("Giving up trying to upload %s after %d attempts" %
> (remote_filename, globals.num_retries), 1)
> + raise BackendException("Error uploading %s" % remote_filename)
>
> def get(self, remote_filename, local_path):
> - log.Log("Downloading %s from Amazon S3" % remote_filename, 5)
> key = self.key_class(self.bucket)
> key.key = remote_filename
> - key.get_contents_to_filename(local_path.name)
> - local_path.setdata()
> + for n in range(1, globals.num_retries+1):
> + log.Log("Downloading %s from Amazon S3 (attempt #%d)" %
> (remote_filename, n), 5)
> + try:
> + key.get_contents_to_filename(local_path.name)
> + local_path.setdata()
> + return
> + except:
> + pass
> + log.Log("Downloading %s failed (attempt #%d)" %
> (remote_filename, n), 1)
> + time.sleep(30)
> + log.Log("Giving up trying to download %s after %d attempts" %
> (remote_filename, globals.num_retries), 1)
> + raise BackendException("Error downloading %s" % remote_filename)
>
> def list(self):
> filename_list = [k.key for k in self.bucket]
>
>
> ------------------------------------------------------------------------
>
> _______________________________________________
> Duplicity-talk mailing list
> address@hidden
> http://lists.nongnu.org/mailman/listinfo/duplicity-talk
signature.asc
Description: OpenPGP digital signature