[Qemu-devel] [PATCH v6 00/12] Dirty bitmaps migration

[Vladimir Sementsov-Ogievskiy]

From: Vladimir Sementsov-Ogievskiy <address@hidden>

These patches provide dirty bitmap migration feature. Only named dirty
bitmaps are to be migrated. Migration may be enabled using migration
capabilities.

v6:
    - rebase on master
       - function qemu_get_string() deleted, used existed function
         qemu_get_counted_string()
       - function qemu_put_string() renamed appropriately to
         qemu_put_counted_string()
         (and now it is new separate patch)
    =====================================================
    0004: Reviewed-by: John Snow <address@hidden>
    0006: (since 2.3) -> (since 2.4)
    0007: new patch, mentioned above
    0008: printf format fixed, DEBUG definition set back to 0, in
          dirty_bitmap_live_iterate_is_active conditions swapped.
          qemu_{get,put}_string removed, used ..counted.. verisions
    0011: (since 2.3) -> (since 2.4)
    0012: assertIsNotNone -> assertNotEqual for compitibility with
          old Python
          Reviewed-by: John Snow <address@hidden>


v5:
    - rebase on master
    - drop [PATCH RFC v4 10/13] iotests: add event_wait to VM class
    - remove rfc, as incremental backup series by John Snow are in
      upstream
    

v4 significant changes:
 0001: tiny bugfix: out[i] -> out[i-start], same for 'in'
 0007: set chunk size to 1kb, disable live iteration for
       migrating data < 1mb size.

 tests: only one with md5 sum is here. used function event_wait
        by John Snow. (I hope, you don't mind me just adding this
        function with your 'Signed-off-by')

 rfc: This patch set is based on v13 of
      "block: incremental backup series" by John Snow, which are
      not pushed yet.

v3:
 based on v13 of "block: incremental backup series" by John Snow.

 changes from v2:
 removed patch for adding dirty parameter (migration capablities used
 instead).
 
 0001: printf's dropped, qapi used
 0002: part0 -> zeroes
 0003: part0 -> zeroes
 0005: dirty_dirty -> meta
       add comments about meta bitmap
       
 0006: the format is changed, nodes used instead of devices.

 other patches are new.

 rfc: there are two tests. They are the same but using different
 interfaces: md5 checksum of the bitmap last layer in query-block or
 separate query-block-dirty-bitmap with dirty bitmap regions.
 The second form is more appropriate for debugging, the first is more
 appropriate for simple regression control. Which should go to
 upstream?

v2:
 1. bug-fixes, that are already in upstream, and renaming of function
 bdrv_reset_dirty_bitmap (which is already in Snow's series) are
 dropped
 2. bitmap store/restore: the concept renamed to serialization, added
 function hbitmap_deserialize_part0, to not transfer zero blocks
 3. migration dirty parameter: added description comment
 4. Other patches are new.

v2.rfc:
Actually, in this version of the series I'm trying not use
migration/block.c at all. Instead a separate migration unit is added
in the new file migration/dirty-bitmap.c. Now bitmaps are migrated
like blocks in block migration, they have their "dirty-dirty" bitmaps,
for tracking set/unset changes during migration.

The advantages are:
  - no complications of migration/block.c
  - separate dirty-dirty bitmaps provide handling of "unset's"
  - more effective meta-data/data ratio - no tiny bitmap-blocks.



v1:
These patches provide dirty bitmap migration feature. Only named dirty
bitmaps are to be migrated. Migration is made as a part of block
migration in block-migration.c.

Dirty bitmap migration may be enabled by "dirty" parameter for qmp migrate
command. If "blk" and "inc" parameters are false when "dirty" is true
block migration is actually skipped: no allocatoions, no bdrv_read's,
no bdrv_write's, only bitmaps are migrated.

The patch set includes two my previous bug fixes, which are necessary
for it. The patch set is based on Incremental backup series by John
Snow.
Vladimir Sementsov-Ogievskiy (12):
  hbitmap: serialization
  block: BdrvDirtyBitmap serialization interface
  block: tiny refactoring: minimize hbitmap_(set/reset) usage
  block: add meta bitmaps
  block: add bdrv_next_dirty_bitmap()
  qapi: add dirty-bitmaps migration capability
  migration: add migration/block-dirty-bitmap.c
  iotests: maintain several vms in test
  iotests: add add_incoming_migration to VM class
  qapi: add md5 checksum of last dirty bitmap level to query-block
  iotests: add dirty bitmap migration test
  migration/qemu-file: make functions qemu_(get/put)_string public

 block.c                        |  98 +++++-
 include/block/block.h          |  22 ++
 include/migration/block.h      |   1 +
 include/migration/migration.h  |   1 +
 include/migration/qemu-file.h  |  17 +
 include/qemu/hbitmap.h         |  67 ++++
 migration/Makefile.objs        |   2 +-
 migration/block-dirty-bitmap.c | 693 +++++++++++++++++++++++++++++++++++++++++
 migration/migration.c          |   9 +
 migration/qemu-file.c          |  18 ++
 qapi-schema.json               |   4 +-
 qapi/block-core.json           |   4 +-
 tests/qemu-iotests/117         |  84 +++++
 tests/qemu-iotests/117.out     |   5 +
 tests/qemu-iotests/group       |   1 +
 tests/qemu-iotests/iotests.py  |  14 +-
 util/hbitmap.c                 | 106 +++++++
 vl.c                           |   1 +
 18 files changed, 1137 insertions(+), 10 deletions(-)
 create mode 100644 migration/block-dirty-bitmap.c
 create mode 100755 tests/qemu-iotests/117
 create mode 100644 tests/qemu-iotests/117.out

-- 
1.9.1