[Top][All Lists]
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
[Duplicity-talk] Proposal for backgrounded concurrent I/O with new retry
From: |
Peter Schuller |
Subject: |
[Duplicity-talk] Proposal for backgrounded concurrent I/O with new retry support |
Date: |
Fri, 30 Nov 2007 12:13:56 +0100 |
User-agent: |
KMail/1.9.7 |
Hello,
I have been looking through the code some, with an eye towards two features:
(1) backgrounding all backend I/O, and allowing for concurrency
(2) the ability to pause retries until an operator indicates the backend
should resume retrying, as has been previously discussed on the list
(2) has the benefit discussed previously.
The benefit of (1) is two-fold: Firstly, even with concurrency = 1,
backgrounding the I/O means that duplicity can keep working on producting
backup data for a chunk, while the previous chunk is transferred. Depending
on the relative availability of CPU vs. bandwidth throughput, this can
potentially double the speed of backups.
In addition, implementing that would also mean that it is trivial to add
support for backgrounding multiple I/O operations concurrently, which would
cater to high latency links. This should help people backing up a lot of data
to remote machines, where neither en is limited by bandwidth but the
throughput of the TCP connections end up being limited by bandwidth delay
product and other issues resulting from high latency or perhaps packet loss.
I wanted to describe what I want to do before I start, in case people have
objections or better ideas.
=== RETRY ===
Firstly, I have been looking through the backends and it seems some of them do
not implement any retry at all, some of them implement their own retry
(sometimes on only put()/get(), sometimes on all methods), and yet some use
Backend.run_command_persist().
I wanted to generalize retry support in a way that minimized the chance of
breakage; allowing for incrementally adding/improving retry for individual
backens or even individual methods of backends. In orde to do this my plan is
to implement a fairly simple function:
def call_persistently(txn):
"""
Calls the given function txn ("transaction") until it either completes
or fails permanently. A permanent failure is defined as:
- A user specified number of retries, with a user specified backoff
behavior, has failed with a retryable error. A retryable error being
signalled by raising a RetryableError exception.
- A call to the function causes any other exception to be raised.
On failure, the original exception is propagated back to the caller.
In the case of a RetryableError, the root cause is propagated rather
than the RetryableError itself.
Implicit in the above description is that the function can be called any
number of times, and should therefor be idempotent except insofar
as the implementation is aware of the potential for repeated
execution.
"""
...
This function would handle everything, including potentially pausing and
waiting for operator attention. Individual backend methods can be
converted/upgraded to use this mechanism, and individually tested. It also
does not change the public interface of backends, nor the relationship
between Backend and its subclasses.
=== Background concurrent I/O ===
Firstly, a new method is added to Backend's public interface:
def supports_concurrency():
"""
Returns whether this backend supports concurrency (multiple
invocations of backend I/O methods concurrently in multiple
threads without ill effects).
By default False is returned. Backend implementations that
support concurrency must override this method.
"""
return False
This is to allow concurrency while not requireing that all backends are
converted to support it.
Note that some backends use instance data during I/O (such as a per-instance
connection objects), and will thus require some changes to support concurrent
I/O. Others will already support it automatically.
On the front-end of things in duplicity-bin, there are various iterators used
to read and write chunks to/from the backend. These iterators will be
modified such that on creation, they schedule all expected I/O operations
with a scheduler (backed by n worker threads). The iterators then wait on the
next chunk to have completed I/O, instead of directly instigating the I/O.
This latter change should be the only part that will have widespread effects
on all backends at once. But on the upside the code is common to all and thus
testable without having access to backends of all types.
Sample future duplicity invocation:
duplicity incremental /path/to/dir scp://address@hidden:/path \
--concurrency=5 \
--retry-failure-mode=intervention \
--retry-failure-flag=/path/to/intervention-flag
Opinions/thoughts?
--
/ Peter Schuller
PGP userID: 0xE9758B7D or 'Peter Schuller <address@hidden>'
Key retrieval: Send an E-Mail to address@hidden
E-Mail: address@hidden Web: http://www.scode.org
signature.asc
Description: This is a digitally signed message part.
- [Duplicity-talk] Proposal for backgrounded concurrent I/O with new retry support,
Peter Schuller <=