[Qemu-devel] [PATCH v4 00/17] qcow2: Add new overlap check functions

[Max Reitz]

This is a continuation of previous versions of this series. v2's cover
letter was the most elaborate, which you can find here (includes
benchmarks):
http://lists.nongnu.org/archive/html/qemu-devel/2014-11/msg03430.html

See patch 1 for an explanation of why this series exists and what it
does. Patch 1 is basically the core of this series, the rest just
employs the functions introduced there.

The new thing about v4 is that there now is a way to limit the total
combined size of all data structures used for the new overlap prevention
algorithm. Bad news: The limit is disabled by default. This is mainly
because for image creation, it is pretty hard to find a sane default
(it depends on the desired image size; but the image is not created with
that size, but rather with size 0 and then truncated; inferring the
default from the image size seems to make sense, but we cannot do this
for images that are going to be truncated).

RFC: An alternative would be to infer the limit like so:
MAX(1M, guest_image_size / 1M). It seems sane to me and I guess it
should work even for images with very, very large guest sizes. But I
think it will break if you're trying to preallocate like a 1 PB image
(which is not really a concern right now, but let's see about it in
ten years)...
On the other hand, the worst that happens is that the event is generated
and the overlap checks won't be performed for some parts of the image
during its creation. Doesn't sound too bad to me. What do you think,
dears reviewers?

In a later patch, we may want to change the meaning of the "constant"
overlap checking option to mean the same as "cached", which is
everything except for inactive L2 tables. This series does make
checking for overlaps with inactive L2 tables at runtime just as cheap
as everything else (constant time plus caching), but using these checks
means qemu has to read all the snapshot L1 tables when opening a qcow2
file. This does not take long, of course, but it does result in a bit of
overhead so I did not want to enable it by default.


v4:
- Patch 1:
  - Use the length of bs->file instead of bs itself as a basis on how
    many windows should be allocated initially
  - If the fragment list would take up more memory than the bitmap
    (which is conceivable, considering that every cluster in the
    bitmap/bytemap uses exactly one byte, whereas every cluster may use
    up to one whole entry in the fragment list (which takes up four
    bytes)), do not drop the bitmap on cache eviction, but rather just
    abort the generation of the fragment list
  - The cache_size parameter should be divided by WINDOW_SIZE only
    instead of WINDOW_SIZE + sizeof(*s->metadata_list->cached_windows);
    the user is more likely to specify multiples of WINDOW_SIZE, so we
    should take that into account (the sizeof(...) part is really
    negligible, and also, if the user really wants to limit the memory
    usage, there are now four new patches in this series to do that)
  - Just drop the bitmap when the metadata list is destroyed; do not use
    destroy_window_bitmap() to do so, because this will (unneccessarily)
    generate a fragment list (which is going to be dropped, too, anyway)
- Patch 13: Added; adds a new QMP event to be used by patch 14
- Patch 14: Added; adds a control mechanism to limit the amount of
  memory used for the overlap structures (and issues the event from
  patch 13 if that limit is hit)
- Patch 15: Added; adds runtime options to qcow2 to control the cache
  size and the total memory limit for the overlap checking
- Patch 16: Added; exposes these options via QMP
- Patch 17: Added; some tests for these new options


git-backport-diff against v3:

Key:
[----] : patches are identical
[####] : number of functional differences between upstream/downstream patch
[down] : patch is downstream-only
The flags [FC] indicate (F)unctional and (C)ontextual differences, respectively

001/17:[0071] [FC] 'qcow2: Add new overlap check functions'
002/17:[----] [--] 'qcow2: Pull up overlap check option evaluation'
003/17:[----] [-C] 'qcow2: Create metadata list'
004/17:[----] [-C] 'qcow2/overlaps: Protect image header'
005/17:[----] [--] 'qcow2/overlaps: Protect refcount table'
006/17:[----] [-C] 'qcow2/overlaps: Protect refcount blocks'
007/17:[----] [--] 'qcow2/overlaps: Protect active L1 table'
008/17:[----] [-C] 'qcow2/overlaps: Protect active L2 tables'
009/17:[----] [-C] 'qcow2/overlaps: Protect snapshot table'
010/17:[----] [--] 'qcow2/overlaps: Protect inactive L1 tables'
011/17:[----] [-C] 'qcow2/overlaps: Protect inactive L2 tables'
012/17:[----] [--] 'qcow2: Use new metadata overlap check function'
013/17:[down] 'qcow2/overlaps: Add "memory limit reached" event'
014/17:[down] 'qcow2/overlaps: Add memory usage limit'
015/17:[down] 'qcow2: Add overlap structure memory size options'
016/17:[down] 'qapi: Expose new qcow2 overlap check options'
017/17:[down] 'iotests: Test qcow2's overlap check memory limit'


Max Reitz (17):
  qcow2: Add new overlap check functions
  qcow2: Pull up overlap check option evaluation
  qcow2: Create metadata list
  qcow2/overlaps: Protect image header
  qcow2/overlaps: Protect refcount table
  qcow2/overlaps: Protect refcount blocks
  qcow2/overlaps: Protect active L1 table
  qcow2/overlaps: Protect active L2 tables
  qcow2/overlaps: Protect snapshot table
  qcow2/overlaps: Protect inactive L1 tables
  qcow2/overlaps: Protect inactive L2 tables
  qcow2: Use new metadata overlap check function
  qcow2/overlaps: Add "memory limit reached" event
  qcow2/overlaps: Add memory usage limit
  qcow2: Add overlap structure memory size options
  qapi: Expose new qcow2 overlap check options
  iotests: Test qcow2's overlap check memory limit

 block/Makefile.objs        |   3 +-
 block/qcow2-cluster.c      |  13 ++
 block/qcow2-overlap.c      | 572 +++++++++++++++++++++++++++++++++++++++++++++
 block/qcow2-refcount.c     | 202 +++++++---------
 block/qcow2-snapshot.c     | 105 ++++++++-
 block/qcow2.c              | 162 +++++++++----
 block/qcow2.h              |  15 ++
 docs/qmp/qmp-events.txt    |  28 +++
 qapi/block-core.json       |  37 +++
 qapi/event.json            |  27 +++
 tests/qemu-iotests/060     | 222 ++++++++++++++++++
 tests/qemu-iotests/060.out |  47 ++++
 tests/qemu-iotests/group   |   2 +-
 13 files changed, 1262 insertions(+), 173 deletions(-)
 create mode 100644 block/qcow2-overlap.c

-- 
2.3.7