[Qemu-devel] [PATCH v3 RFC 0/17] block: persistent dirty bitmaps

[Vladimir Sementsov-Ogievskiy]

v3:

Hi all! After long delay I am starting a new thread about persistent
dirty bitmaps. Not all ideas/requests are done from v2 thread, but I
need to start somewhere. Also, there are a lot of code changes to be
considered before everything is finished. Also, v2 thread is too large
and it is hard to handle it.

Here I'll summarize, what was changed. All ideas/comments, that was not
done from v2 thread I'll add as replays to this cover letter. If I miss
something, please add.

01-02: new patches, necessary preparation

03: docs
- add constraints
- document auto clear bit
- renames
    L1 table -> Dirty Bitmap Table
    Dirty Bitmaps Table -> Dirty Bitmap Directory
- remove Standard cluster with its zero bit,

struct changes:

extension header:
  add 32bit dirty_bitmap_directory_size
  it is needed to read the whole Dirty Bitmap Directory.
  nb_dirty_bitmaps becomes excessive (it may be calculated by looking
  through Dirty Bitmap Directory).
  RFC: is it bad? should I drop nb_dirty_bitmaps?

directory entry:
- bitmap size is renamed to 'number of virtual bits'
- flags added. (self/aut/read_only are not reliazed in code for now)
- granularity is now represented by granularity_bits
 

04: structs and consts
Structures changed, accordingly to spec.
QCowDirtyBitmap now only contains name and offset. Offset is offset in
bytes in Dirty Bitmap Directory, so, bitmap header can be accessed:
cached version: s->dirty_bitmap_directory + offset
in image file: s->dirty_bitmap_directory_offset + offset

== dirty-bitmaps feature ==

initial patch [PATCH 2/8] qcow2: add dirty-bitmaps feature
is splitted out to 5 parts. The first with license notice and
constants is in 04. Then, there are:
05: read Dirty Bitmap Directory
06: load dirty bitmap
07: store dirty bitmap
08: add dirty bitmap extension

The code is significantly changed and reorganized. For your comments:
- fixes about flushes
- g_try_... functions are used. But I left the old g_malloc/g_new for
things with constant or strictly limited size (dirty bitmap header,
with its name, name size < 1024, one cluster, bitmap name (< 1024))
- input validation added. 
- dirty bitmap directory is read/written wholly, not by parts.
- load/store through bdrv_dirty_bitmap_[de]serialize_part
- 'delete dirty bitmap' function is removed. This function is not used
in v3. May be this functionality should appear in qemu_img, or with qmp
series.
- qcow2_write_dirty_bitamps function is removed. This functionality is
rewritten, now Bitmap Directory is rewritten from
s->dirty_bitmap_directory, (directory_write()), and not generated from
s->dirty_bitmaps

09: load check
- new functionality, just check existance of dirty bitmap by name

10: store block dirty bitmap
- bdrv_store_dirty_bitmap simplified, as bitmap is simply passed to
qcow2_dirty_bitmap_store

11: load block dirty bitmap
- bdrv_load_dirty_bitmap simplified, as bitmap is created by
qcow2_dirty_bitmap_load. Also, granularity parameter is removed

patch [[PATCH 5/8] qcow2: add qcow2_dirty_bitmap_delete_all] is
removed. This function is not used in v3. May be this functionality
should appear in qemu_img, or with qmp series.

12: autoclear bit
- if it is not set, but bitmap extension is found, it just skipped and
warning printed. Dirty bitmaps are not cleread from the image as in v2
- handling in qcow2-dirty-bitmap.c is changed, the bit is just set when
adding entry to Dirty Bitmap Directory in the image.

13: cmd line
create=on|off flag added. Now, the bitmap is not auto-created.
By default flag is off.
If on, then new bitmap will be created in the image, if the bitmap with
same name is already exists an error will be generated.
If off, then the bitmap will be loaded from the image, if there is no
one an error will be generated.
If create=off and granularity is specified then granularity will be
checked for loaded bitmap and if not match an error will be generated.

also, s/drive/node

also, dirty_bitmap_func():
    - add errp parameter and handle this in main()
    - open image with BDRV_O_RDWR for loading bitmaps

14-16: new patches

17: test
- add three test cases.


v2:
 - rebase on my 'Dirty bitmaps migration' series
 - remove 'print dirty bitmap', 'query-dirty-bitmap' and use md5 for
   testing like with dirty bitmaps migration
 - autoclean features

v1:

The bitmaps are saved into qcow2 file format. It provides both
'internal' and 'external' dirty bitmaps feature:
 - for qcow2 drives we can store bitmaps in the same file
 - for other formats we can store bitmaps in the separate qcow2 file

QCow2 header is extended by fields 'nb_dirty_bitmaps' and
'dirty_bitmaps_offset' like with snapshots.

Proposed command line syntax is the following:

-dirty-bitmap [option1=val1][,option2=val2]...
    Available options are:
    name         The name for the bitmap (necessary).

    file         The file to load the bitmap from.

    file_id      When specified with 'file' option, then this file will
                 be available through this id for other -dirty-bitmap
                 options when specified without 'file' option, then it
                 is a reference to 'file', specified with another
                 -dirty-bitmap option, and it will be used to load the
                 bitmap from.

    drive        The drive to bind the bitmap to. It should be specified
                 as 'id' suboption of one of -drive options. If nor
                 'file' neither 'file_id' are specified, then the bitmap
                 will be loaded from that drive (internal dirty bitmap).

    granularity  The granularity for the bitmap. Not necessary, the
                 default value may be used.

    enabled      on|off. Default is 'on'. Disabled bitmaps are not
                 changing regardless of writes to corresponding drive.

Examples:

qemu -drive file=a.qcow2,id=disk -dirty-bitmap name=b,drive=disk
qemu -drive file=a.raw,id=disk \
     -dirty-bitmap name=b,drive=disk,file=b.qcow2,enabled=off

Vladimir Sementsov-Ogievskiy (8):
  spec: add qcow2-dirty-bitmaps specification
  qcow2: add dirty-bitmaps feature
  block: store persistent dirty bitmaps
  block: add bdrv_load_dirty_bitmap
  qcow2: add qcow2_dirty_bitmap_delete_all
  qcow2: add autoclear bit for dirty bitmaps
  qemu: command line option for dirty bitmaps
  iotests: test internal persistent dirty bitmap

 block.c                       |  82 +++++++
 block/Makefile.objs           |   2 +-
 block/qcow2-dirty-bitmap.c    | 537 ++++++++++++++++++++++++++++++++++++++++++
 block/qcow2.c                 |  69 +++++-
 block/qcow2.h                 |  61 +++++
 blockdev.c                    |  38 +++
 docs/specs/qcow2.txt          |  66 ++++++
 include/block/block.h         |   9 +
 include/block/block_int.h     |  10 +
 include/sysemu/blockdev.h     |   1 +
 include/sysemu/sysemu.h       |   1 +
 qemu-options.hx               |  37 +++
 tests/qemu-iotests/118        |  83 +++++++
 tests/qemu-iotests/118.out    |   5 +
 tests/qemu-iotests/group      |   1 +
 tests/qemu-iotests/iotests.py |   6 +
 vl.c                          | 100 ++++++++
 17 files changed, 1105 insertions(+), 3 deletions(-)
 create mode 100644 block/qcow2-dirty-bitmap.c
 create mode 100755 tests/qemu-iotests/118
 create mode 100644 tests/qemu-iotests/118.out

-- 
1.9.1