[Qemu-block] [PATCH v12 00/19] Support streaming to an intermediate layer

[Alberto Garcia]

Hi all,

here's version 12 of the series.

The problem with version 11 and the bdrv_drain_all() split is that it
won't work if new block jobs are created or new BDSs are added between
the _begin() and _end() calls.

We were discussing this on the mailing list and on IRC. Adding a
global quiesce counter for the bdrv_drain_all() function, and making
sure that new BDSs are disabled when they are added in the middle of a
bdrv_drain_all() call should be enough in principle.

There's however other potential cases of failure, like what happens if
the AioContext is changed while a BDS is disabled.

In the end Paolo said that he wants to look into that in the future,
and suggested that since at the moment the only use case for the
bdrv_drain_all() split is bdrv_reopen_multiple() then it's easier not
to worry about it in this series and document the fact that no jobs or
BDSs can be created during bdrv_drain_all_begin/end().

This is what this series does. I also rebased it on top of the current
master in order to fix a merge conflict, but it's otherwise identical
to v11.

Berto

v12:
- Rebase on top of the current master.
- Patch 1: Add a note explaining that no new block jobs or BDSs can be
  added between bdrv_drain_all_begin/end().

v11: https://lists.gnu.org/archive/html/qemu-block/2016-10/msg00518.html
- Patches 1-2: Add bdrv_drain_all_{begin,end}() and use these
  functions in bdrv_reopen_multiple() instead of pausing all block
  jobs.
- Patch 3: Don't unblock BLOCK_OP_TYPE_DATAPLANE in
  block_job_add_bdrv()
- Patch 7: In commit_start(), don't call block_job_add_bdrv() twice on
  the active node.
- Patch 10: In qmp_block_stream(), don't check twice for op blockers
  in 'bs'.
- Patch 11: Clarify that live-block-ops.txt is incomplete and it only
  covers the block-stream operation.
- Patches 18-19: Add a 'base-node' parameter to 'block-stream', and
  iotests for it.

v10: https://lists.gnu.org/archive/html/qemu-devel/2016-10/msg01047.html
- Rebase the code.
- Pause block jobs during bdrv_reopen().
- Allow jobs in parallel again, and modify jobs to block all involved
  nodes, not just the root one.
- Add more tests.

v9: https://lists.gnu.org/archive/html/qemu-devel/2016-04/msg00411.html
- Rebase the code
- Block the active layer in order to forbid other block jobs in the
  same chain.
- Split the patch that adds block_job_next() into 4 (new patches 1-4).
- Replace the test that performs several block-stream operations in
  parallel with one that check that they're forbidden.
- Remove patches that have already been merged.

v8: https://lists.gnu.org/archive/html/qemu-devel/2015-06/msg05754.html
- Rebased on top of Stefan's block branch (0a35bce416)
- The loop that pauses the block jobs in bdrv_drain_all() is now split
  in two: one that iterates the list of block jobs to stop them, and
  one that iterates the root bds in order to get the aio contexts.

v7: https://lists.gnu.org/archive/html/qemu-devel/2015-05/msg02580.html
- Rebased against the current master
- Updated bdrv_drain_all() to use the new block_job_next() API.

v6: https://lists.gnu.org/archive/html/qemu-devel/2015-04/msg03046.html
- fix the no-op test following Max's suggestions

v5: https://lists.gnu.org/archive/html/qemu-devel/2015-04/msg03006.html
- Fix a few typos
- Minor documentation updates
- Update test_stream_partial() to test no-ops
- New test case: test_stream_parallel()
- New test case: test_stream_overlapping()

v4: https://lists.gnu.org/archive/html/qemu-devel/2015-04/msg01878.html
- Refactor find_block_job to use the error from bdrv_lookup_bs()
- Don't use QERR_DEVICE_IN_USE in block_job_create() since we can be
  dealing with nodes now.
- Fix @device comment in the BlockJobInfo documentation
- stream_start(): simplify the bdrv_reopen() call and use
  bdrv_get_device_or_node_name() for error messages.
- Use a different variable name for BlockDriverState *i
- Documentation fixes in docs/live-block-ops.txt
- Update iotest 30 since now test_device_not_found() returns
  GenericError
- Fix test case test_stream_partial()
- Add new test case test_stream_intermediate()
- Fix typos

v3: https://lists.gnu.org/archive/html/qemu-devel/2015-04/msg00806.html
- Keep a list of block jobs and make qmp_query_block_jobs() iterate
  over it.

v2: https://lists.gnu.org/archive/html/qemu-devel/2015-03/msg04798.html
- The 'block-stream' command does not have a 'node-name' parameter
  anymore and reuses 'device' for that purpose.
- Block jobs can now be owned by any intermediate node, and not just
  by the ones at the root. query-block-jobs is updated to reflect that
  change.
- The 'device' parameter of all 'block-job-*' commands can now take a
  node name.
- The BlockJobInfo type and all BLOCK_JOB_* events report the node
  name in the 'device' field if the node does not have a device name.
- All intermediate nodes are blocked (and checked for blockers) during
  the streaming operation.

v1: https://lists.gnu.org/archive/html/qemu-devel/2015-02/msg04116.html

backport-diff against v11:
Key:
[----] : patches are identical
[####] : number of functional differences between upstream/downstream patch
[down] : patch is downstream-only
The flags [FC] indicate (F)unctional and (C)ontextual differences, respectively

001/19:[0003] [FC] 'block: Add bdrv_drain_all_{begin,end}()'
002/19:[----] [--] 'block: Pause all jobs during bdrv_reopen_multiple()'
003/19:[----] [--] 'block: Add block_job_add_bdrv()'
004/19:[----] [--] 'block: Use block_job_add_bdrv() in mirror_start_job()'
005/19:[----] [--] 'block: Use block_job_add_bdrv() in backup_start()'
006/19:[----] [--] 'block: Check blockers in all nodes involved in a 
block-commit job'
007/19:[----] [--] 'block: Block all nodes involved in the block-commit 
operation'
008/19:[----] [--] 'block: Block all intermediate nodes in 
commit_active_start()'
009/19:[----] [--] 'block: Support streaming to an intermediate layer'
010/19:[----] [--] 'block: Add QMP support for streaming to an intermediate 
layer'
011/19:[----] [--] 'docs: Document how to stream to an intermediate layer'
012/19:[----] [--] 'qemu-iotests: Test streaming to an intermediate layer'
013/19:[----] [--] 'qemu-iotests: Test block-stream operations in parallel'
014/19:[----] [--] 'qemu-iotests: Test overlapping stream and commit operations'
015/19:[----] [--] 'qemu-iotests: Test block-stream and block-commit in 
parallel'
016/19:[----] [-C] 'qemu-iotests: Add iotests.supports_quorum()'
017/19:[----] [--] 'qemu-iotests: Test streaming to a Quorum child'
018/19:[----] [--] 'block: Add 'base-node' parameter to the 'block-stream' 
command'
019/19:[----] [--] 'qemu-iotests: Test the 'base-node' parameter of 
'block-stream''

Alberto Garcia (19):
  block: Add bdrv_drain_all_{begin,end}()
  block: Pause all jobs during bdrv_reopen_multiple()
  block: Add block_job_add_bdrv()
  block: Use block_job_add_bdrv() in mirror_start_job()
  block: Use block_job_add_bdrv() in backup_start()
  block: Check blockers in all nodes involved in a block-commit job
  block: Block all nodes involved in the block-commit operation
  block: Block all intermediate nodes in commit_active_start()
  block: Support streaming to an intermediate layer
  block: Add QMP support for streaming to an intermediate layer
  docs: Document how to stream to an intermediate layer
  qemu-iotests: Test streaming to an intermediate layer
  qemu-iotests: Test block-stream operations in parallel
  qemu-iotests: Test overlapping stream and commit operations
  qemu-iotests: Test block-stream and block-commit in parallel
  qemu-iotests: Add iotests.supports_quorum()
  qemu-iotests: Test streaming to a Quorum child
  block: Add 'base-node' parameter to the 'block-stream' command
  qemu-iotests: Test the 'base-node' parameter of 'block-stream'

 block.c                       |   9 +-
 block/backup.c                |   5 +-
 block/commit.c                |  14 ++
 block/io.c                    |  27 +++-
 block/mirror.c                |  11 +-
 block/stream.c                |  24 ++++
 blockdev.c                    |  41 +++++-
 blockjob.c                    |  17 ++-
 docs/live-block-ops.txt       |  36 +++--
 docs/qmp-commands.txt         |   7 +-
 hmp.c                         |   2 +-
 include/block/block.h         |   2 +
 include/block/blockjob.h      |  14 ++
 qapi/block-core.json          |  18 ++-
 tests/qemu-iotests/030        | 309 +++++++++++++++++++++++++++++++++++++++++-
 tests/qemu-iotests/030.out    |   4 +-
 tests/qemu-iotests/041        |  27 ++--
 tests/qemu-iotests/139        |   3 +-
 tests/qemu-iotests/iotests.py |   5 +-
 19 files changed, 518 insertions(+), 57 deletions(-)

-- 
2.9.3