[Qemu-devel] [PATCH v7 00/13] Dirty bitmaps migration

[Vladimir Sementsov-Ogievskiy]

These patches provide dirty bitmap migration feature. Only named dirty
bitmaps are to be migrated. Migration may be enabled using migration
capabilities.

v7:

0001: main functions are mostly rewritten to be safer. Granularity of
      serialization is now explicit thing. Only the last chunk of
      bitmap data may be not aligned.
      Reviewed-by deleted.

0002: changed accordingly to 0001 chnages
      Reviewed-by deleted.

0004: use ctz64() instead of ffsll(), accordingly to checkpatch
      Reviewed-by left. Hope, you don't mind. 

0006: since 2.4 -> since 2.6
      Reviewed-by's left (Eric Blake's one added).

0007: s/int/size_t for strlen result, add comment
      Reviewed-by left. Hope, you don't mind. 

0008:
    Rename some static functions for better style:
    qemu_get_flags -> qemu_get_bitmap_flags
    qemu_put_flags -> qemu_put_bitmap_flags
    blk_mig_reset_dirty_cursor -> dirty_bitmap_mig_reset_dirty_cursor
    
    Logic, related to 0001 changes:
    - bdrv_dirty_bitmap_data_size calls with one count parameter are
    changed by bdrv_dirty_bitmap_serialization_size calls with 
    two parameters, specifying the range
    - change dirty_bitmap_save_pending for not using
    bdrv_dirty_bitmap_data_size

    Rebase on master:
    - rename _cancel to _cleanup and .cancel to .cleanup
    - change dirty_bitmap_save_pending (postcopy parameters)
    - rename .save_live_complete to .save_live_complete_precopy

    Reviewed-by deleted.

    also: new function hbitmap_serialization_granularity made public,
    and may be used for selecting chunk size (it gives the minimum) for
    migration, storing, loding.  But it is not used, because, I think,
    checking that CHUNK_SIZE from migration/block-dirty-bitmap.c is
    greater than sizeof(long).

0011:
    # @md5: md5 checksum of the last bitmap level (since 2.4)
to
    # @md5: md5 checksum (as a hexadecimal string) of the last bitmap level
    #       (since 2.6)
    Reviewed-by left. Hope, my new comment is ok and you don't mind.

0012: new patch


v6:
    - rebase on master
       - function qemu_get_string() deleted, used existed function
         qemu_get_counted_string()
       - function qemu_put_string() renamed appropriately to
         qemu_put_counted_string()
         (and now it is new separate patch)
    =====================================================
    0004: Reviewed-by: John Snow <address@hidden>
    0006: (since 2.3) -> (since 2.4)
    0007: new patch, mentioned above
    0008: printf format fixed, DEBUG definition set back to 0, in
          dirty_bitmap_live_iterate_is_active conditions swapped.
          qemu_{get,put}_string removed, used ..counted.. verisions
    0011: (since 2.3) -> (since 2.4)
    0012: assertIsNotNone -> assertNotEqual for compitibility with
          old Python
          Reviewed-by: John Snow <address@hidden>


v5:
    - rebase on master
    - drop [PATCH RFC v4 10/13] iotests: add event_wait to VM class
    - remove rfc, as incremental backup series by John Snow are in
      upstream
    

v4 significant changes:
 0001: tiny bugfix: out[i] -> out[i-start], same for 'in'
 0007: set chunk size to 1kb, disable live iteration for
       migrating data < 1mb size.

 tests: only one with md5 sum is here. used function event_wait
        by John Snow. (I hope, you don't mind me just adding this
        function with your 'Signed-off-by')

 rfc: This patch set is based on v13 of
      "block: incremental backup series" by John Snow, which are
      not pushed yet.

v3:
 based on v13 of "block: incremental backup series" by John Snow.

 changes from v2:
 removed patch for adding dirty parameter (migration capablities used
 instead).
 
 0001: printf's dropped, qapi used
 0002: part0 -> zeroes
 0003: part0 -> zeroes
 0005: dirty_dirty -> meta
       add comments about meta bitmap
       
 0006: the format is changed, nodes used instead of devices.

 other patches are new.

 rfc: there are two tests. They are the same but using different
 interfaces: md5 checksum of the bitmap last layer in query-block or
 separate query-block-dirty-bitmap with dirty bitmap regions.
 The second form is more appropriate for debugging, the first is more
 appropriate for simple regression control. Which should go to
 upstream?

v2:
 1. bug-fixes, that are already in upstream, and renaming of function
 bdrv_reset_dirty_bitmap (which is already in Snow's series) are
 dropped
 2. bitmap store/restore: the concept renamed to serialization, added
 function hbitmap_deserialize_part0, to not transfer zero blocks
 3. migration dirty parameter: added description comment
 4. Other patches are new.

v2.rfc:
Actually, in this version of the series I'm trying not use
migration/block.c at all. Instead a separate migration unit is added
in the new file migration/dirty-bitmap.c. Now bitmaps are migrated
like blocks in block migration, they have their "dirty-dirty" bitmaps,
for tracking set/unset changes during migration.

The advantages are:
  - no complications of migration/block.c
  - separate dirty-dirty bitmaps provide handling of "unset's"
  - more effective meta-data/data ratio - no tiny bitmap-blocks.



v1:
These patches provide dirty bitmap migration feature. Only named dirty
bitmaps are to be migrated. Migration is made as a part of block
migration in block-migration.c.

Dirty bitmap migration may be enabled by "dirty" parameter for qmp migrate
command. If "blk" and "inc" parameters are false when "dirty" is true
block migration is actually skipped: no allocatoions, no bdrv_read's,
no bdrv_write's, only bitmaps are migrated.

Vladimir Sementsov-Ogievskiy (13):
  hbitmap: serialization
  block: BdrvDirtyBitmap serialization interface
  block: tiny refactoring: minimize hbitmap_(set/reset) usage
  block: add meta bitmaps
  block: add bdrv_next_dirty_bitmap()
  qapi: add dirty-bitmaps migration capability
  migration/qemu-file: add qemu_put_counted_string()
  migration: add migration/block-dirty-bitmap.c
  iotests: maintain several vms in test
  iotests: add add_incoming_migration to VM class
  qapi: add md5 checksum of last dirty bitmap level to query-block
  iotests: add default node-name
  iotests: add dirty bitmap migration test

 block.c                        |  93 +++++-
 include/block/block.h          |  22 ++
 include/migration/block.h      |   1 +
 include/migration/migration.h  |   1 +
 include/migration/qemu-file.h  |   2 +
 include/qemu/hbitmap.h         |  81 +++++
 migration/Makefile.objs        |   2 +-
 migration/block-dirty-bitmap.c | 708 +++++++++++++++++++++++++++++++++++++++++
 migration/migration.c          |   9 +
 migration/qemu-file.c          |  13 +
 qapi-schema.json               |   4 +-
 qapi/block-core.json           |   5 +-
 tests/qemu-iotests/117         |  84 +++++
 tests/qemu-iotests/117.out     |   5 +
 tests/qemu-iotests/group       |   1 +
 tests/qemu-iotests/iotests.py  |  17 +-
 util/hbitmap.c                 | 122 +++++++
 vl.c                           |   1 +
 18 files changed, 1163 insertions(+), 8 deletions(-)
 create mode 100644 migration/block-dirty-bitmap.c
 create mode 100755 tests/qemu-iotests/117
 create mode 100644 tests/qemu-iotests/117.out

-- 
2.1.4