[Qemu-devel] [PATCH v4 00/26] qcow2: Support refcount orders != 4

[Max Reitz]

As of version 3, the qcow2 file format supports different widths for
refcount entries, ranging from 1 to 64 bit (only powers of two).
Currently, qemu only supports 16 bit, which is the only width supported
by version 2 (compat=0.10) images.

This series adds support to qemu for all other valid refcount orders.
This is mainly done by adding two function pointers into the
BDRVQcowState structure for reading and writing refcount values
independently of the current refcount entry width; all in-memory
refcount arrays (mostly cached refcount blocks) now are void pointers
and are accessed through these functions alone.

Thanks to previous work of making the qemu code agnostic of e.g. the
number of refcount entries per refcount block, the rest is fairly
trivial. The most complex patch in this series is patch 18 which
implements changing the refcount order through qemu-img amend.

To test different refcount widths, simply invoke the qemu-iotests check
program with -o refcount_width=${your_desired_width}. The final test in
this series adds some tests for operations which do not work with
certain refcount orders and for refcount order amendment.


v4:
- Patch 1: No longer limit to INT64_MAX but allow full range up to
  UINT64_MAX; also, use a non-overflowing shift to avoid the special
  case for s->refcount_order == 6 [Stefan]
- Patch 2: %s/refcount_width/refcount_bits/g (etc.) [Stefan]
- Patch 3: Added
- Patch 4: Added [Stefan]
- Patch 5: Added
- Patch 6 (prev. 3):
  - Rebase conflicts due to patches 3, 4 and 5
  - Always use uint64_t for refcounts instead of int64_t [Stefan]
- Patch 7 (prev. 4):
  - Rebase conflicts due to patch 5
  - qcow2_update_cluster_refcount() only returns an int (not an
    int64_t), thus "int ret" will suffice (due to patch 3)
- Patch 8 (prev. 5):
  - Refcounts are always uint64_t [Stefan]
  - Rebase conflict due to patch 4
- Patch 9 (prev. 6): Allow shrinking of the array in
  realloc_refcount_array() [Stefan]
- Patch 10 (prev. 7):
  - Rebase conflicts due to patch 4
  - Range checks are no longer necessary [Stefan]
  - Refcounts are always uint64_t [Stefan]
- Patch 11 (prev 8): It is now allowed to set the MSb in 64 bit
  refcounts, therefore drop the assert() in set_refcount_ro6() [Stefan]
- Patch 13 (prev. 10): %s/refcount_width/refcount_bits/g [Stefan]
- Patch 14: Added
- Patch 15 (prev. 11): %s/refcount_width/refcount_bits/g [Stefan]
- Patch 16 (prev. 12):
  - s/BLOCK_OPT_REFCOUNT_WIDTH/BLOCK_OPT_REFCOUNT_BITS/g [Stefan]
  - s/"refcount_width"/BLOCK_OPT_REFCOUNT_BITS/
  - %s/refcount_width/refcount_bits/g [Stefan]
- Patch 17 (prev. 13): As a very good example on why I am opposed to
  leaving the opening brace of an "if" block on the last line of
  multi-line conditions, I followed Eric's proposal and removed the
  superfluous parentheses in the condition touched by this patch [Eric]
- Patch 23 (prev. 19): Same change as in patch 1; and fixed an overly
  long line
- Patch 24 (prev. 20):
  - s/"refcount_width"/BLOCK_OPT_REFCOUNT_BITS/
  - %s/refcount_width/refcount_bits/g [Stefan]
- Patch 26 (prev. 22):
  - %s/refcount_width/refcount_bits/g [Stefan]
  - Setting the MSb in a 64 bit refcount value should work now [Stefan]
  - Added a test for snapshotting a cluster with a refcount of 2^64 - 1


git-backport-diff against v3:

Key:
[----] : patches are identical
[####] : number of functional differences between upstream/downstream patch
[down] : patch is downstream-only
The flags [FC] indicate (F)unctional and (C)ontextual differences, respectively

001/26:[0010] [FC] 'qcow2: Add two new fields to BDRVQcowState'
002/26:[down] 'qcow2: Add refcount_bits to format-specific info'
        ^^^^ wrong, should be:
       [0054] [FC]
003/26:[down] 'qcow2: Do not return new value after refcount update'
004/26:[down] 'qcow2: Only return status from qcow2_get_refcount'
005/26:[down] 'qcow2: Use unsigned addend for update_refcount()'
006/26:[0093] [FC] 'qcow2: Use 64 bits for refcount values'
007/26:[0011] [FC] 'qcow2: Respect error in qcow2_alloc_bytes()'
008/26:[0015] [FC] 'qcow2: Refcount overflow and qcow2_alloc_bytes()'
009/26:[0008] [FC] 'qcow2: Helper for refcount array reallocation'
010/26:[0017] [FC] 'qcow2: Helper function for refcount modification'
011/26:[0003] [FC] 'qcow2: More helpers for refcount modification'
012/26:[----] [--] 'qcow2: Open images with refcount order != 4'
013/26:[0006] [FC] 'qcow2: refcount_order parameter for qcow2_create2'
014/26:[down] 'qcow2: Use symbolic macros in qcow2_amend_options'
015/26:[down] 'iotests: Prepare for refcount_bits option'
        ^^^^ wrong, should be:
       [0038] [FC]
016/26:[0228] [FC] 'qcow2: Allow creation with refcount order != 4'
017/26:[0002] [FC] 'progress: Allow regressing progress'
018/26:[----] [--] 'block: Add opaque value to the amend CB'
019/26:[----] [-C] 'qcow2: Use error_report() in qcow2_amend_options()'
020/26:[----] [--] 'qcow2: Use abort() instead of assert(false)'
021/26:[----] [--] 'qcow2: Split upgrade/downgrade paths for amend'
022/26:[----] [--] 'qcow2: Use intermediate helper CB for amend'
023/26:[0010] [FC] 'qcow2: Add function for refcount order amendment'
024/26:[0020] [FC] 'qcow2: Invoke refcount order amendment function'
025/26:[----] [--] 'qcow2: Point to amend function in check'
026/26:[0212] [FC] 'iotests: Add test for different refcount widths'


Max Reitz (26):
  qcow2: Add two new fields to BDRVQcowState
  qcow2: Add refcount_bits to format-specific info
  qcow2: Do not return new value after refcount update
  qcow2: Only return status from qcow2_get_refcount
  qcow2: Use unsigned addend for update_refcount()
  qcow2: Use 64 bits for refcount values
  qcow2: Respect error in qcow2_alloc_bytes()
  qcow2: Refcount overflow and qcow2_alloc_bytes()
  qcow2: Helper for refcount array reallocation
  qcow2: Helper function for refcount modification
  qcow2: More helpers for refcount modification
  qcow2: Open images with refcount order != 4
  qcow2: refcount_order parameter for qcow2_create2
  qcow2: Use symbolic macros in qcow2_amend_options
  iotests: Prepare for refcount_bits option
  qcow2: Allow creation with refcount order != 4
  progress: Allow regressing progress
  block: Add opaque value to the amend CB
  qcow2: Use error_report() in qcow2_amend_options()
  qcow2: Use abort() instead of assert(false)
  qcow2: Split upgrade/downgrade paths for amend
  qcow2: Use intermediate helper CB for amend
  qcow2: Add function for refcount order amendment
  qcow2: Invoke refcount order amendment function
  qcow2: Point to amend function in check
  iotests: Add test for different refcount widths

 block.c                          |    4 +-
 block/qcow2-cluster.c            |   24 +-
 block/qcow2-refcount.c           | 1052 +++++++++++++++++++++++++++++++-------
 block/qcow2.c                    |  286 ++++++++---
 block/qcow2.h                    |   23 +-
 include/block/block.h            |    4 +-
 include/block/block_int.h        |    4 +-
 qapi/block-core.json             |    5 +-
 qemu-img.c                       |    5 +-
 tests/qemu-iotests/007           |    3 +
 tests/qemu-iotests/015           |    2 +
 tests/qemu-iotests/026           |    7 +
 tests/qemu-iotests/029           |    2 +
 tests/qemu-iotests/049.out       |  112 ++--
 tests/qemu-iotests/051           |    3 +
 tests/qemu-iotests/058           |    2 +
 tests/qemu-iotests/060.out       |    1 +
 tests/qemu-iotests/061.out       |   14 +-
 tests/qemu-iotests/065           |   23 +-
 tests/qemu-iotests/067           |    2 +
 tests/qemu-iotests/067.out       |    5 +
 tests/qemu-iotests/079           |   10 +-
 tests/qemu-iotests/079.out       |   38 +-
 tests/qemu-iotests/080           |    2 +
 tests/qemu-iotests/082.out       |   48 +-
 tests/qemu-iotests/085.out       |   38 +-
 tests/qemu-iotests/089           |    2 +
 tests/qemu-iotests/089.out       |    2 +
 tests/qemu-iotests/108           |    2 +
 tests/qemu-iotests/112           |  296 +++++++++++
 tests/qemu-iotests/112.out       |  155 ++++++
 tests/qemu-iotests/common.filter |    3 +-
 tests/qemu-iotests/group         |    1 +
 util/qemu-progress.c             |    3 +-
 34 files changed, 1783 insertions(+), 400 deletions(-)
 create mode 100755 tests/qemu-iotests/112
 create mode 100644 tests/qemu-iotests/112.out

-- 
1.9.3