[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: [Qemu-devel] [PATCH v11 00/14] block: incremental backup transaction
From: |
Stefan Hajnoczi |
Subject: |
Re: [Qemu-devel] [PATCH v11 00/14] block: incremental backup transactions using BlockJobTxn |
Date: |
Tue, 10 Nov 2015 13:34:48 +0000 |
User-agent: |
Mutt/1.5.23 (2015-06-09) |
On Thu, Nov 05, 2015 at 06:13:06PM -0500, John Snow wrote:
> Welcome to the Incremental Backup Transactions Newsletter!
>
> What's new?
>
> I replaced the per-action "transactional-cancel" parameter with
> a per-transaction paremeter named "completion-mode" which is implemented
> as an enum in case we want to add new behaviors in the future, such
> as a "jobs only" cancel mode.
>
> For now, it's "grouped" or "individual", and if you use it with actions
> that do not support the latent transactional cancel, you will receive
> an error for your troubles.
>
> Version 10 primarily changed V7's patches 10-11 and replaced them
> with patches 10-12 that are cut a little differently.
>
> This is based on top of the work by Stefan Hajnoczi and Fam Zheng.
>
> Recap: motivation for block job transactions
> --------------------------------------------
> If an incremental backup block job fails then we reclaim the bitmap so
> the job can be retried. The problem comes when multiple jobs are started as
> part of a qmp 'transaction' command. We need to group these jobs in a
> transaction so that either all jobs complete successfully or all bitmaps are
> reclaimed.
>
> Without transactions, there is a case where some jobs complete successfully
> and
> throw away their bitmaps, making it impossible to retry the backup by
> rerunning
> the command if one of the jobs fails.
>
> How does this implementation work?
> ----------------------------------
> These patches add a BlockJobTxn object with the following API:
>
> txn = block_job_txn_new();
> block_job_txn_add_job(txn, job1);
> block_job_txn_add_job(txn, job2);
>
> The jobs either both complete successfully or they both fail/cancel. If the
> user cancels job1 then job2 will also be cancelled and vice versa.
>
> Jobs objects stay alive waiting for other jobs to complete, even if the
> coroutines have returned. They can be cancelled by the user during this time.
> Job blockers are still in effect and no other block job can run on this device
> in the meantime (since QEMU currently only allows 1 job per device). This is
> the main drawback to this approach but reasonable since you probably don't
> want
> to run other jobs/operations until you're sure the backup was successful (you
> won't be able to retry a failed backup if there's a new job running).
>
> [History]
>
> v11: Renamed "err-cancel" to "completion-mode"
> "none" becomes "individual"
> "all" becomes "grouped"
>
> v10: Took series back from Fam. (jsnow)
> Replaced per-action parameter with per-transaction properties.
> Patches 10,11 were split into 10-12.
>
> v9: this version fixes a reference count problem with job->bs,
> in patch 05.
>
> v8: Rebase on to master.
> Minor fixes addressing John Snow's comments.
>
> v7: Add Eric's rev-by in 1, 11.
> Add Max's rev-by in 4, 5, 9, 10, 11.
> Add John's rev-by in 5, 6, 8.
> Fix wording for 6. [John]
> Fix comment of block_job_txn_add_job() in 9. [Max]
> Remove superfluous hunks, and document default value in 11. [Eric]
> Update Makefile dep in 14. [Max]
>
> ________________________________________________________________________________
>
> For convenience, this branch is available at:
> https://github.com/jnsnow/qemu.git branch block-transpop
> https://github.com/jnsnow/qemu/tree/block-transpop
>
> This version is tagged block-transpop-v11:
> https://github.com/jnsnow/qemu/releases/tag/block-transpop-v11
>
> Fam Zheng (6):
> backup: Extract dirty bitmap handling as a separate function
> blockjob: Introduce reference count and fix reference to job->bs
> blockjob: Add .commit and .abort block job actions
> blockjob: Add "completed" and "ret" in BlockJob
> blockjob: Simplify block_job_finish_sync
> block: Add block job transactions
>
> John Snow (7):
> qapi: Add transaction support to block-dirty-bitmap operations
> iotests: add transactional incremental backup test
> block: rename BlkTransactionState and BdrvActionOps
> block/backup: Rely on commit/abort for cleanup
> block: Add BlockJobTxn support to backup_run
> block: add transactional properties
> iotests: 124 - transactional failure test
>
> Stefan Hajnoczi (1):
> tests: add BlockJobTxn unit test
>
> block.c | 19 +-
> block/backup.c | 50 ++++--
> block/mirror.c | 2 +-
> blockdev.c | 432
> ++++++++++++++++++++++++++++++++++-----------
> blockjob.c | 189 ++++++++++++++++----
> docs/bitmaps.md | 6 +-
> include/block/block.h | 2 +-
> include/block/block_int.h | 6 +-
> include/block/blockjob.h | 85 ++++++++-
> qapi-schema.json | 56 +++++-
> qemu-img.c | 3 -
> qmp-commands.hx | 2 +-
> tests/Makefile | 3 +
> tests/qemu-iotests/124 | 182 ++++++++++++++++++-
> tests/qemu-iotests/124.out | 4 +-
> tests/test-blockjob-txn.c | 250 ++++++++++++++++++++++++++
> 16 files changed, 1118 insertions(+), 173 deletions(-)
> create mode 100644 tests/test-blockjob-txn.c
>
> --
> 2.4.3
>
Thanks, applied to my block tree:
https://github.com/stefanha/qemu/commits/block
Stefan
signature.asc
Description: PGP signature
- [Qemu-devel] [PATCH v11 03/14] block: rename BlkTransactionState and BdrvActionOps, (continued)
- [Qemu-devel] [PATCH v11 03/14] block: rename BlkTransactionState and BdrvActionOps, John Snow, 2015/11/05
- [Qemu-devel] [PATCH v11 05/14] blockjob: Introduce reference count and fix reference to job->bs, John Snow, 2015/11/05
- [Qemu-devel] [PATCH v11 04/14] backup: Extract dirty bitmap handling as a separate function, John Snow, 2015/11/05
- [Qemu-devel] [PATCH v11 09/14] block: Add block job transactions, John Snow, 2015/11/05
- [Qemu-devel] [PATCH v11 10/14] block/backup: Rely on commit/abort for cleanup, John Snow, 2015/11/05
- [Qemu-devel] [PATCH v11 13/14] iotests: 124 - transactional failure test, John Snow, 2015/11/05
- [Qemu-devel] [PATCH v11 14/14] tests: add BlockJobTxn unit test, John Snow, 2015/11/05
- [Qemu-devel] [PATCH v11 08/14] blockjob: Simplify block_job_finish_sync, John Snow, 2015/11/05
- [Qemu-devel] [PATCH v11 12/14] block: add transactional properties, John Snow, 2015/11/05
- [Qemu-devel] [PATCH v11 11/14] block: Add BlockJobTxn support to backup_run, John Snow, 2015/11/05
- Re: [Qemu-devel] [PATCH v11 00/14] block: incremental backup transactions using BlockJobTxn,
Stefan Hajnoczi <=