[Qemu-devel] [RFC PATCH 00/16] Qemu Bit Map (QBM) - an overlay format for persistent dirty bitmap

[Fam Zheng]

Hi all,

This series introduces a simple format to enable support of persistence of
block dirty bitmaps. Block dirty bitmap is the tool to achieve incremental
backup, and persistence of block dirty bitmap makes incrememtal backup possible
across VM shutdowns, where existing in-memory dirty bitmaps cannot survive.

When user creates a "persisted" dirty bitmap, the QBM driver will create a
binary file and synchronize it with the existing in-memory block dirty bitmap
(BdrvDirtyBitmap). When the VM is powered down, the binary file has all the
bits saved on disk, which will be loaded and used to initialize the in-memory
block dirty bitmap next time the guest is started.

The idea of the format is to reuse as much existing infrastructure as possible
and avoid introducing complex data structures - it works with any image format,
by gluing it together plain bitmap files with a json descriptor file. The
advantage of this approach over extending existing formats, such as qcow2, is
that the new feature is implemented by an orthogonal driver, in a format
agnostic way. This way, even raw images can have their persistent dirty
bitmaps.  (And you will notice in this series, with a little forging to the
spec, raw images can also have backing files through a QBM overlay!)

Rather than superseding it, this intends to be coexistent in parallel with the
qcow2 bitmap extension that Vladimir is working on.  The block driver interface
changes in this series also try to be generic and compatible for both drivers.

The format's specification is added to docs/specs/, see patch 1.

Patches 2-7 are necessary block layer changes in order be friendly to
persistent dirty bitmap drivers.

Patches 8, 9 and 11 extends the QMP interface to expose the added feature.

Patch 10 implements the driver. (todo: checksum extension for image/bitmap
integrity check)

Patch 12 - 16 are the tests I have for QBM so far. I'm sure more can be added
as questions emerge. :)

The series applies on top of my "qemu-img map" series and "meta bitmap" series:

https://lists.gnu.org/archive/html/qemu-devel/2016-01/msg04866.html
https://lists.gnu.org/archive/html/qemu-devel/2016-01/msg03656.html

If you feel like to play with it, git branch is also available at:

    https://github.com/famz/qemu qbm

Comments are welcome!

Fam


Fam Zheng (16):
  doc: Add QBM format specification
  block: Set dirty before doing write
  block: Allow .bdrv_close callback to release dirty bitmaps
  block: Move filename_decompose to block.c
  block: Make bdrv_get_cluster_size public
  block: Introduce bdrv_dirty_bitmap_set_persistent
  block: Only swap non-persistent dirty bitmaps
  qmp: Add optional parameter "persistent" in block-dirty-bitmap-add
  qmp: Add block-dirty-bitmap-set-persistent
  qbm: Implement format driver
  qapi: Add "qbm" as a generic cow format driver
  iotests: Add qbm format to 041
  iotests: Add qbm to case 097
  iotests: Add qbm to applicable test cases
  iotests: Add qbm specific test case 140
  iotests: Add persistent bitmap test case 141

 block.c                      |   54 +-
 block/Makefile.objs          |    1 +
 block/dirty-bitmap.c         |   63 ++
 block/io.c                   |    6 +-
 block/qbm.c                  | 1315 ++++++++++++++++++++++++++++++++++++++++++
 block/vmdk.c                 |   40 --
 blockdev.c                   |   28 +-
 docs/specs/qbm.md            |  118 ++++
 include/block/block.h        |    4 +-
 include/block/block_int.h    |    8 +
 include/block/dirty-bitmap.h |    5 +
 qapi/block-core.json         |   31 +-
 qmp-commands.hx              |   34 +-
 tests/qemu-iotests/004       |    2 +-
 tests/qemu-iotests/017       |    2 +-
 tests/qemu-iotests/018       |    2 +-
 tests/qemu-iotests/019       |    2 +-
 tests/qemu-iotests/020       |    2 +-
 tests/qemu-iotests/024       |    2 +-
 tests/qemu-iotests/025       |    2 +-
 tests/qemu-iotests/027       |    2 +-
 tests/qemu-iotests/028       |    2 +-
 tests/qemu-iotests/030       |    2 +-
 tests/qemu-iotests/034       |    2 +-
 tests/qemu-iotests/037       |    2 +-
 tests/qemu-iotests/038       |    2 +-
 tests/qemu-iotests/040       |    2 +-
 tests/qemu-iotests/041       |   18 +-
 tests/qemu-iotests/050       |    2 +-
 tests/qemu-iotests/055       |    2 +-
 tests/qemu-iotests/056       |    2 +-
 tests/qemu-iotests/069       |    2 +-
 tests/qemu-iotests/072       |    2 +-
 tests/qemu-iotests/086       |    2 +-
 tests/qemu-iotests/095       |    2 +-
 tests/qemu-iotests/096       |    2 +-
 tests/qemu-iotests/097       |    4 +-
 tests/qemu-iotests/099       |    2 +-
 tests/qemu-iotests/110       |    2 +-
 tests/qemu-iotests/129       |    2 +-
 tests/qemu-iotests/132       |    2 +-
 tests/qemu-iotests/139       |    2 +-
 tests/qemu-iotests/140       |   80 +++
 tests/qemu-iotests/140.out   |  145 +++++
 tests/qemu-iotests/141       |   62 ++
 tests/qemu-iotests/141.out   |    5 +
 tests/qemu-iotests/common    |    6 +
 tests/qemu-iotests/group     |    2 +
 48 files changed, 1998 insertions(+), 85 deletions(-)
 create mode 100644 block/qbm.c
 create mode 100644 docs/specs/qbm.md
 create mode 100755 tests/qemu-iotests/140
 create mode 100644 tests/qemu-iotests/140.out
 create mode 100644 tests/qemu-iotests/141
 create mode 100644 tests/qemu-iotests/141.out

-- 
2.4.3