Re: [Qemu-devel] [PATCH 03/17] spec: add qcow2-dirty-bitmaps specification

[John Snow]

On 09/05/2015 01:33 PM, Vladimir Sementsov-Ogievskiy wrote:
> On 05.09.2015 19:43, Vladimir Sementsov-Ogievskiy wrote:
>> Persistent dirty bitmaps will be saved into qcow2 files. It may be used
>> as 'internal' bitmaps (for qcow2 drives) or as 'external' bitmaps for
>> other drives (there may be qcow2 file with zero disk size but with
>> several dirty bitmaps for other drives).
>>
>> Signed-off-by: Vladimir Sementsov-Ogievskiy <address@hidden>
>> ---
>>   docs/specs/qcow2.txt | 127
>> ++++++++++++++++++++++++++++++++++++++++++++++++++-
>>   1 file changed, 126 insertions(+), 1 deletion(-)
>>
>> diff --git a/docs/specs/qcow2.txt b/docs/specs/qcow2.txt
>> index 121dfc8..5fc0365 100644
>> --- a/docs/specs/qcow2.txt
>> +++ b/docs/specs/qcow2.txt
>> @@ -103,7 +103,13 @@ in the description of a field.
>>                       write to an image with unknown auto-clear
>> features if it
>>                       clears the respective bits from this field first.
>>   -                    Bits 0-63:  Reserved (set to 0)
>> +                    Bit 0:      Dirty bitmaps bit. If this bit is set
>> then
>> +                                there is a _consistent_ Dirty bitmaps
>> extension
>> +                                in the image. If it is not set, but
>> there is a
>> +                                Dirty bitmaps extension, its data
>> should be
>> +                                considered as inconsistent.
>> +
>> +                    Bits 1-63:  Reserved (set to 0)
>>              96 -  99:  refcount_order
>>                       Describes the width of a reference count block
>> entry (width
>> @@ -123,6 +129,7 @@ be stored. Each extension has a structure like the
>> following:
>>                           0x00000000 - End of the header extension area
>>                           0xE2792ACA - Backing file format name
>>                           0x6803f857 - Feature name table
>> +                        0x23852875 - Dirty bitmaps
>>                           other      - Unknown header extension, can
>> be safely
>>                                        ignored
>>   @@ -166,6 +173,24 @@ the header extension data. Each entry look like
>> this:
>>                       terminated if it has full length)
>>     +== Dirty bitmaps ==
>> +
>> +Dirty bitmaps is an optional header extension. It provides an ability
>> to store
>> +dirty bitmaps in a qcow2 image. The fields are:
>> +
>> +          0 -  3:  nb_dirty_bitmaps
>> +                   The number of dirty bitmaps contained in the
>> image. Valid
>> +                   values: 0 - 65535.
>> +
>> +          4 -  7:  dirty_bitmap_directory_size
>> +                   Size of the Dirty Bitmap Directory in bytes. Valid
>> values:
>> +                   0 - 67108864 (= 1024 * nb_dirty_bitmaps).
>> +
>> +          8 - 15:  dirty_bitmap_directory_offset
>> +                   Offset into the image file at which the Dirty Bitmap
>> +                   Directory starts. Must be aligned to a cluster
>> boundary.
>> +
>> +
>>   == Host cluster management ==
>>     qcow2 manages the allocation of host clusters by maintaining a
>> reference count
>> @@ -360,3 +385,103 @@ Snapshot table entry:
>>             variable:   Padding to round up the snapshot table entry
>> size to the
>>                       next multiple of 8.
>> +
>> +
>> +== Dirty bitmaps ==
>> +
>> +The feature supports storing dirty bitmaps in a qcow2 image.
>> +
>> +=== Cluster mapping ===
>> +
>> +Dirty bitmaps are stored using a ONE-level structure for the mapping of
>> +bitmaps to host clusters. It is called Dirty Bitmap Table.
>> +
>> +The Dirty Bitmap Table has a variable size (stored in the Dirty Bitmap
>> +Directory Entry) and may use multiple clusters, however it must be
>> contiguous
>> +in the image file.
>> +
>> +Given an offset (in bytes) into the bitmap, the offset into the image
>> file can
>> +be obtained as follows:
>> +
>> +    byte_offset =
>> +        dirty_bitmap_table[offset / cluster_size] + (offset %
>> cluster_size)
>> +
>> +Taking into accout the granularity of the bitmap, an offset in bits
>> into the
>> +image file can be obtained like this:
>> +
>> +    bit_offset =
>> +        byte_offset(bit_nr / granularity / 8) * 8 + (bit_nr /
>> granularity) % 8
>> +
>> +Here bit_nr is a number of "virtual" bit of the bitmap, which is
>> covered by
>> +"physical" bit with number (bit_nr / granularity).
>> +
>> +Dirty Bitmap Table entry:
>> +
>> +    Bit  0 -  8:    Reserved
>> +
>> +         9 - 55:    Bits 9-55 of host cluster offset. Must be aligned
>> to a
>> +                    cluster boundary. If the offset is 0, the cluster is
>> +                    unallocated, and should be read as all zeros.
>> +
>> +        56 - 63:    Reserved
>> +
>> +=== Dirty Bitmap Directory ===
>> +
>> +Each dirty bitmap, saved in the image is described in the Dirty Bitmap
>> +Directory entry. Dirty Bitmap Directory is a contiguous area in the
>> image file,
>> +whose starting offset and length are given by the header extension
>> fields
>> +dirty_bitmap_directory_offset and dirty_bitmap_directory_size. The
>> entries of
>> +the bitmap directory have variable length, depending on the length of
>> the
>> +bitmap name.
>> +
>> +Dirty Bitmap Directory Entry:
>> +
>> +    Byte 0 -  7:    dirty_bitmap_table_offset
>> +                    Offset into the image file at which the Dirty
>> Bitmap Table
>> +                    for the bitmap starts. Must be aligned to a cluster
>> +                    boundary.
>> +
>> +         8 - 15:    nb_virtual_bits
>> +                    Number of "virtual" bits in the bitmap. Number of
>> +                    "physical" bits would be:
>> +                    (nb_virtual_bits + granularity - 1) / granularity
>> +
>> +        16 - 19:    dirty_bitmap_table_size
>> +                    Number of entries in the Dirty Bitmap Table of
>> the bitmap.
>> +                    Valid values: 0 - 0x8000000.
>> +                    Also, (dirty_bitmap_table_size * cluster_size)
>> should not
>> +                    be greater than 0x20000000 (512 MB)
>> +
>> +        20 - 23:    granularity_bits
>> +                    Granularity bits. Valid values are: 0 - 63.
>> +
>> +                    Granularity is calculated as
>> +                        granularity = 1 << granularity_bits
>> +
>> +                    Granularity of the bitmap is how many "virtual" bits
>> +                    accounts for one "physical" bit.
>> +
>> +        24 - 27:    flags
>> +                    Bit
>> +                      0: in_use
>> +                         The bitmap is in use and may be inconsistent.
>> +
>> +                      1: self
>> +                         The bitmap is a dirty bitmap for the
>> containing image.
>> +
>> +                      2: auto
>> +                         The bitmap should be autoloaded as block
>> dirty bitmap.
>> +                         Only available if bit 1 (self) is set.
>> +
>> +                      3: read_only
>> +                         The bitmap should not be rewritten.
>> +
>> +                    Bits 4 - 31 are reserved.
> 
> Is this appropriate as field, reserved for future extensiion? Or we need
> an additional one? Do we need scheme like with snapshots? (somthing like
> field 'additional_area_size', and additional offset of this size after
> the name)
> 

I think it would remain appropriate as long as we have a version header
for the bitmap extension as a whole.

e.g. "Bits 4 - 31 are reserved in qcow2.bitmap.v1 ..."

If a program that only knows about v1 opens a v2 file and find it
conforms to spec (does not use the new reserved bits), then it can
continue along happily.

If a reserved bit is set, it's an error and the v1 program must not
alter the image in case it ruins the consistency of the file.

The usual suspects (Kevin, Markus, Stefan and Eric) may have better
suggestions for how to handle future compatibility by drawing upon their
experience with qcow2.


>> +
>> +        28 - 29:    name_size
>> +                    Size of the bitmap name. Valid values: 0 - 1023.
>> +
>> +        variable:   The name of the bitmap (not null terminated).
>> +
>> +        variable:   Padding to round up the Dirty Bitmap Directory
>> Entry size to
>> +                    the next multiple of 8.
> 
> 

-- 
—js