[Qemu-commits] [qemu/qemu] 9c4269: qcow2: Make Qcow2AioTask store the full host offset

[Peter Maydell]

  Branch: refs/heads/master
  Home:   https://github.com/qemu/qemu
  Commit: 9c4269d54b8bcc1939641009a2ea9b24d887fd58
      
https://github.com/qemu/qemu/commit/9c4269d54b8bcc1939641009a2ea9b24d887fd58
  Author: Alberto Garcia <berto@igalia.com>
  Date:   2020-08-25 (Tue, 25 Aug 2020)

  Changed paths:
    M block/qcow2.c
    M block/trace-events

  Log Message:
  -----------
  qcow2: Make Qcow2AioTask store the full host offset

The file_cluster_offset field of Qcow2AioTask stores a cluster-aligned
host offset. In practice this is not very useful because all users(*)
of this structure need the final host offset into the cluster, which
they calculate using

   host_offset = file_cluster_offset + offset_into_cluster(s, offset)

There is no reason why Qcow2AioTask cannot store host_offset directly
and that is what this patch does.

(*) compressed clusters are the exception: in this case what
    file_cluster_offset was storing was the full compressed cluster
    descriptor (offset + size). This does not change with this patch
    but it is documented now.

Signed-off-by: Alberto Garcia <berto@igalia.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Reviewed-by: Max Reitz <mreitz@redhat.com>
Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Message-Id: 
<07c4b15c644dcf06c9459f98846ac1c4ea96e26f.1594396418.git.berto@igalia.com>
Signed-off-by: Max Reitz <mreitz@redhat.com>


  Commit: 388e581615d11126bc6d37223d50cbf8e127bde1
      
https://github.com/qemu/qemu/commit/388e581615d11126bc6d37223d50cbf8e127bde1
  Author: Alberto Garcia <berto@igalia.com>
  Date:   2020-08-25 (Tue, 25 Aug 2020)

  Changed paths:
    M block/qcow2-cluster.c
    M block/qcow2.c
    M block/qcow2.h

  Log Message:
  -----------
  qcow2: Convert qcow2_get_cluster_offset() into qcow2_get_host_offset()

qcow2_get_cluster_offset() takes an (unaligned) guest offset and
returns the (aligned) offset of the corresponding cluster in the qcow2
image.

In practice none of the callers need to know where the cluster starts
so this patch makes the function calculate and return the final host
offset directly. The function is also renamed accordingly.

There is a pre-existing exception with compressed clusters: in this
case the function returns the complete cluster descriptor (containing
the offset and size of the compressed data). This does not change with
this patch but it is now documented.

Signed-off-by: Alberto Garcia <berto@igalia.com>
Reviewed-by: Max Reitz <mreitz@redhat.com>
Message-Id: 
<ffae6cdc5ca8950e8280ac0f696dcc376cb07095.1594396418.git.berto@igalia.com>
Signed-off-by: Max Reitz <mreitz@redhat.com>


  Commit: 8f91d6906c2217b968b6b01cdcb9e0edbb4d18ab
      
https://github.com/qemu/qemu/commit/8f91d6906c2217b968b6b01cdcb9e0edbb4d18ab
  Author: Alberto Garcia <berto@igalia.com>
  Date:   2020-08-25 (Tue, 25 Aug 2020)

  Changed paths:
    M block/qcow2-cluster.c

  Log Message:
  -----------
  qcow2: Add calculate_l2_meta()

handle_alloc() creates a QCowL2Meta structure in order to update the
image metadata and perform the necessary copy-on-write operations.

This patch moves that code to a separate function so it can be used
from other places.

Signed-off-by: Alberto Garcia <berto@igalia.com>
Reviewed-by: Max Reitz <mreitz@redhat.com>
Message-Id: 
<e5bc4a648dac31972bfa7a0e554be8064be78799.1594396418.git.berto@igalia.com>
Signed-off-by: Max Reitz <mreitz@redhat.com>


  Commit: c1587d877e223c4c3a8db6fa620b71384550c492
      
https://github.com/qemu/qemu/commit/c1587d877e223c4c3a8db6fa620b71384550c492
  Author: Alberto Garcia <berto@igalia.com>
  Date:   2020-08-25 (Tue, 25 Aug 2020)

  Changed paths:
    M block/qcow2-cluster.c

  Log Message:
  -----------
  qcow2: Split cluster_needs_cow() out of count_cow_clusters()

We are going to need it in other places.

Signed-off-by: Alberto Garcia <berto@igalia.com>
Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Reviewed-by: Max Reitz <mreitz@redhat.com>
Message-Id: 
<65e5d9627ca2ebe7e62deaeddf60949c33067d9d.1594396418.git.berto@igalia.com>
Signed-off-by: Max Reitz <mreitz@redhat.com>


  Commit: 57538c864f3ad67d515892fd54b11ff59981b84d
      
https://github.com/qemu/qemu/commit/57538c864f3ad67d515892fd54b11ff59981b84d
  Author: Alberto Garcia <berto@igalia.com>
  Date:   2020-08-25 (Tue, 25 Aug 2020)

  Changed paths:
    M block/qcow2-cluster.c

  Log Message:
  -----------
  qcow2: Process QCOW2_CLUSTER_ZERO_ALLOC clusters in handle_copied()

When writing to a qcow2 file there are two functions that take a
virtual offset and return a host offset, possibly allocating new
clusters if necessary:

   - handle_copied() looks for normal data clusters that are already
     allocated and have a reference count of 1. In those clusters we
     can simply write the data and there is no need to perform any
     copy-on-write.

   - handle_alloc() looks for clusters that do need copy-on-write,
     either because they haven't been allocated yet, because their
     reference count is != 1 or because they are ZERO_ALLOC clusters.

The ZERO_ALLOC case is a bit special because those are clusters that
are already allocated and they could perfectly be dealt with in
handle_copied() (as long as copy-on-write is performed when required).

In fact, there is extra code specifically for them in handle_alloc()
that tries to reuse the existing allocation if possible and frees them
otherwise.

This patch changes the handling of ZERO_ALLOC clusters so the
semantics of these two functions are now like this:

   - handle_copied() looks for clusters that are already allocated and
     which we can overwrite (NORMAL and ZERO_ALLOC clusters with a
     reference count of 1).

   - handle_alloc() looks for clusters for which we need a new
     allocation (all other cases).

One important difference after this change is that clusters found
in handle_copied() may now require copy-on-write, but this will be
necessary anyway once we add support for subclusters.

Signed-off-by: Alberto Garcia <berto@igalia.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Reviewed-by: Max Reitz <mreitz@redhat.com>
Message-Id: 
<eb17fc938f6be7be2e8d8ff42763d2c19241f866.1594396418.git.berto@igalia.com>
Signed-off-by: Max Reitz <mreitz@redhat.com>


  Commit: 12c6aebedfb6b90d5f019ab536dd3f2752e6e361
      
https://github.com/qemu/qemu/commit/12c6aebedfb6b90d5f019ab536dd3f2752e6e361
  Author: Alberto Garcia <berto@igalia.com>
  Date:   2020-08-25 (Tue, 25 Aug 2020)

  Changed paths:
    M block/qcow2-cluster.c
    M block/qcow2-refcount.c
    M block/qcow2.h

  Log Message:
  -----------
  qcow2: Add get_l2_entry() and set_l2_entry()

The size of an L2 entry is 64 bits, but if we want to have subclusters
we need extended L2 entries. This means that we have to access L2
tables and slices differently depending on whether an image has
extended L2 entries or not.

This patch replaces all l2_slice[] accesses with calls to
get_l2_entry() and set_l2_entry().

Signed-off-by: Alberto Garcia <berto@igalia.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Reviewed-by: Max Reitz <mreitz@redhat.com>
Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Message-Id: 
<9586363531fec125ba1386e561762d3e4224e9fc.1594396418.git.berto@igalia.com>
Signed-off-by: Max Reitz <mreitz@redhat.com>


  Commit: 30afc12085c8e7759abc77651f65916afdca7bbd
      
https://github.com/qemu/qemu/commit/30afc12085c8e7759abc77651f65916afdca7bbd
  Author: Alberto Garcia <berto@igalia.com>
  Date:   2020-08-25 (Tue, 25 Aug 2020)

  Changed paths:
    M docs/interop/qcow2.txt
    M docs/qcow2-cache.txt

  Log Message:
  -----------
  qcow2: Document the Extended L2 Entries feature

Subcluster allocation in qcow2 is implemented by extending the
existing L2 table entries and adding additional information to
indicate the allocation status of each subcluster.

This patch documents the changes to the qcow2 format and how they
affect the calculation of the L2 cache size.

Signed-off-by: Alberto Garcia <berto@igalia.com>
Reviewed-by: Max Reitz <mreitz@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Message-Id: 
<5199f2e1c717bcaa58b48142c9062b803145ff7f.1594396418.git.berto@igalia.com>
Signed-off-by: Max Reitz <mreitz@redhat.com>


  Commit: a3c7d9162513c2bea742f4278a58e401c6f37d0c
      
https://github.com/qemu/qemu/commit/a3c7d9162513c2bea742f4278a58e401c6f37d0c
  Author: Alberto Garcia <berto@igalia.com>
  Date:   2020-08-25 (Tue, 25 Aug 2020)

  Changed paths:
    M block/qcow2.h

  Log Message:
  -----------
  qcow2: Add dummy has_subclusters() function

This function will be used by the qcow2 code to check if an image has
subclusters or not.

At the moment this simply returns false. Once all patches needed for
subcluster support are ready then QEMU will be able to create and
read images with subclusters and this function will return the actual
value.

Signed-off-by: Alberto Garcia <berto@igalia.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Reviewed-by: Max Reitz <mreitz@redhat.com>
Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Message-Id: 
<905526221083581a1b7057bca1585487661c5c13.1594396418.git.berto@igalia.com>
Signed-off-by: Max Reitz <mreitz@redhat.com>


  Commit: d0346b5591825bab4d4b35aefb0ab0d7650e3641
      
https://github.com/qemu/qemu/commit/d0346b5591825bab4d4b35aefb0ab0d7650e3641
  Author: Alberto Garcia <berto@igalia.com>
  Date:   2020-08-25 (Tue, 25 Aug 2020)

  Changed paths:
    M block/qcow2.c
    M block/qcow2.h

  Log Message:
  -----------
  qcow2: Add subcluster-related fields to BDRVQcow2State

This patch adds the following new fields to BDRVQcow2State:

- subclusters_per_cluster: Number of subclusters in a cluster
- subcluster_size: The size of each subcluster, in bytes
- subcluster_bits: No. of bits so 1 << subcluster_bits = subcluster_size

Images without subclusters are treated as if they had exactly one
subcluster per cluster (i.e. subcluster_size = cluster_size).

Signed-off-by: Alberto Garcia <berto@igalia.com>
Reviewed-by: Max Reitz <mreitz@redhat.com>
Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Message-Id: 
<55bfeac86b092fa2c9d182a95cbeb479ff7eca4f.1594396418.git.berto@igalia.com>
Signed-off-by: Max Reitz <mreitz@redhat.com>


  Commit: a53e8b7202dd941c619a57a640389560214d1f70
      
https://github.com/qemu/qemu/commit/a53e8b7202dd941c619a57a640389560214d1f70
  Author: Alberto Garcia <berto@igalia.com>
  Date:   2020-08-25 (Tue, 25 Aug 2020)

  Changed paths:
    M block/qcow2.h

  Log Message:
  -----------
  qcow2: Add offset_to_sc_index()

For a given offset, return the subcluster number within its cluster
(i.e. with 32 subclusters per cluster it returns a number between 0
and 31).

Signed-off-by: Alberto Garcia <berto@igalia.com>
Reviewed-by: Max Reitz <mreitz@redhat.com>
Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Message-Id: 
<56e3e4ac0d827c6a2f5f259106c5ddb7c4ca2653.1594396418.git.berto@igalia.com>
Signed-off-by: Max Reitz <mreitz@redhat.com>


  Commit: 3e71981592a9f9c52833887f5675b3154d23eec8
      
https://github.com/qemu/qemu/commit/3e71981592a9f9c52833887f5675b3154d23eec8
  Author: Alberto Garcia <berto@igalia.com>
  Date:   2020-08-25 (Tue, 25 Aug 2020)

  Changed paths:
    M block/qcow2.h

  Log Message:
  -----------
  qcow2: Add offset_into_subcluster() and size_to_subclusters()

Like offset_into_cluster() and size_to_clusters(), but for
subclusters.

Signed-off-by: Alberto Garcia <berto@igalia.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Reviewed-by: Max Reitz <mreitz@redhat.com>
Message-Id: 
<3cc2390dcdef3d234d47c741b708bd8734490862.1594396418.git.berto@igalia.com>
Signed-off-by: Max Reitz <mreitz@redhat.com>


  Commit: c8fd8554d972904bd0b04673298fea1af7a37780
      
https://github.com/qemu/qemu/commit/c8fd8554d972904bd0b04673298fea1af7a37780
  Author: Alberto Garcia <berto@igalia.com>
  Date:   2020-08-25 (Tue, 25 Aug 2020)

  Changed paths:
    M block/qcow2-cluster.c
    M block/qcow2-refcount.c
    M block/qcow2.c
    M block/qcow2.h

  Log Message:
  -----------
  qcow2: Add l2_entry_size()

qcow2 images with subclusters have 128-bit L2 entries. The first 64
bits contain the same information as traditional images and the last
64 bits form a bitmap with the status of each individual subcluster.

Because of that we cannot assume that L2 entries are sizeof(uint64_t)
anymore. This function returns the proper value for the image.

Signed-off-by: Alberto Garcia <berto@igalia.com>
Reviewed-by: Max Reitz <mreitz@redhat.com>
Message-Id: 
<d34d578bd0380e739e2dde3e8dd6187d3d249fa9.1594396418.git.berto@igalia.com>
Signed-off-by: Max Reitz <mreitz@redhat.com>


  Commit: 39a9f0a50ea0e85f1be4db1c17ecd25387c3851b
      
https://github.com/qemu/qemu/commit/39a9f0a50ea0e85f1be4db1c17ecd25387c3851b
  Author: Alberto Garcia <berto@igalia.com>
  Date:   2020-08-25 (Tue, 25 Aug 2020)

  Changed paths:
    M block/qcow2.h

  Log Message:
  -----------
  qcow2: Update get/set_l2_entry() and add get/set_l2_bitmap()

Extended L2 entries are 128-bit wide: 64 bits for the entry itself and
64 bits for the subcluster allocation bitmap.

In order to support them correctly get/set_l2_entry() need to be
updated so they take the entry width into account in order to
calculate the correct offset.

This patch also adds the get/set_l2_bitmap() functions that are
used to access the bitmaps. For convenience we allow calling
get_l2_bitmap() on images without subclusters. In this case the
returned value is always 0 and has no meaning.

Signed-off-by: Alberto Garcia <berto@igalia.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Reviewed-by: Max Reitz <mreitz@redhat.com>
Message-Id: 
<6ee0f81ae3329c991de125618b3675e1e46acdbb.1594396418.git.berto@igalia.com>
Signed-off-by: Max Reitz <mreitz@redhat.com>


  Commit: 34905d8eb1ef788a41a490cb1602582ff78cb6f4
      
https://github.com/qemu/qemu/commit/34905d8eb1ef788a41a490cb1602582ff78cb6f4
  Author: Alberto Garcia <berto@igalia.com>
  Date:   2020-08-25 (Tue, 25 Aug 2020)

  Changed paths:
    M block/qcow2.h

  Log Message:
  -----------
  qcow2: Add QCow2SubclusterType and qcow2_get_subcluster_type()

This patch adds QCow2SubclusterType, which is the subcluster-level
version of QCow2ClusterType. All QCOW2_SUBCLUSTER_* values have the
the same meaning as their QCOW2_CLUSTER_* equivalents (when they
exist). See below for details and caveats.

In images without extended L2 entries clusters are treated as having
exactly one subcluster so it is possible to replace one data type with
the other while keeping the exact same semantics.

With extended L2 entries there are new possible values, and every
subcluster in the same cluster can obviously have a different
QCow2SubclusterType so functions need to be adapted to work on the
subcluster level.

There are several things that have to be taken into account:

  a) QCOW2_SUBCLUSTER_COMPRESSED means that the whole cluster is
     compressed. We do not support compression at the subcluster
     level.

  b) There are two different values for unallocated subclusters:
     QCOW2_SUBCLUSTER_UNALLOCATED_PLAIN which means that the whole
     cluster is unallocated, and QCOW2_SUBCLUSTER_UNALLOCATED_ALLOC
     which means that the cluster is allocated but the subcluster is
     not. The latter can only happen in images with extended L2
     entries.

  c) QCOW2_SUBCLUSTER_INVALID is used to detect the cases where an L2
     entry has a value that violates the specification. The caller is
     responsible for handling these situations.

     To prevent compatibility problems with images that have invalid
     values but are currently being read by QEMU without causing side
     effects, QCOW2_SUBCLUSTER_INVALID is only returned for images
     with extended L2 entries.

qcow2_cluster_to_subcluster_type() is added as a separate function
from qcow2_get_subcluster_type(), but this is only temporary and both
will be merged in a subsequent patch.

Signed-off-by: Alberto Garcia <berto@igalia.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Reviewed-by: Max Reitz <mreitz@redhat.com>
Message-Id: 
<26ef38e270f25851c98b51278852b4c4a7f97e69.1594396418.git.berto@igalia.com>
Signed-off-by: Max Reitz <mreitz@redhat.com>


  Commit: 70d1cbae03b54995fdc99f9934c847a87125e4f5
      
https://github.com/qemu/qemu/commit/70d1cbae03b54995fdc99f9934c847a87125e4f5
  Author: Alberto Garcia <berto@igalia.com>
  Date:   2020-08-25 (Tue, 25 Aug 2020)

  Changed paths:
    M block/qcow2-cluster.c

  Log Message:
  -----------
  qcow2: Add qcow2_get_subcluster_range_type()

There are situations in which we want to know how many contiguous
subclusters of the same type there are in a given cluster. This can be
done by simply iterating over the subclusters and repeatedly calling
qcow2_get_subcluster_type() for each one of them.

However once we determined the type of a subcluster we can check the
rest efficiently by counting the number of adjacent ones (or zeroes)
in the bitmap. This is what this function does.

Signed-off-by: Alberto Garcia <berto@igalia.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Reviewed-by: Max Reitz <mreitz@redhat.com>
Message-Id: 
<db917263d568ec6ffb4a41cac3c9100f96bf6c18.1594396418.git.berto@igalia.com>
Signed-off-by: Max Reitz <mreitz@redhat.com>


  Commit: c94d037825ccecf4819ca63f8838561eba49bc80
      
https://github.com/qemu/qemu/commit/c94d037825ccecf4819ca63f8838561eba49bc80
  Author: Alberto Garcia <berto@igalia.com>
  Date:   2020-08-25 (Tue, 25 Aug 2020)

  Changed paths:
    M block/qcow2.h

  Log Message:
  -----------
  qcow2: Add qcow2_cluster_is_allocated()

This helper function tells us if a cluster is allocated (that is,
there is an associated host offset for it).

Signed-off-by: Alberto Garcia <berto@igalia.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Reviewed-by: Max Reitz <mreitz@redhat.com>
Message-Id: 
<6d8771c5c79cbdc6c519875a5078e1cc85856d63.1594396418.git.berto@igalia.com>
Signed-off-by: Max Reitz <mreitz@redhat.com>


  Commit: ca4a0bb81b5259e326a30b758959911459d2a266
      
https://github.com/qemu/qemu/commit/ca4a0bb81b5259e326a30b758959911459d2a266
  Author: Alberto Garcia <berto@igalia.com>
  Date:   2020-08-25 (Tue, 25 Aug 2020)

  Changed paths:
    M block/qcow2-cluster.c
    M block/qcow2.c
    M block/qcow2.h

  Log Message:
  -----------
  qcow2: Add cluster type parameter to qcow2_get_host_offset()

This function returns an integer that can be either an error code or a
cluster type (a value from the QCow2ClusterType enum).

We are going to start using subcluster types instead of cluster types
in some functions so it's better to use the exact data types instead
of integers for clarity and in order to detect errors more easily.

This patch makes qcow2_get_host_offset() return 0 on success and
puts the returned cluster type in a separate parameter. There are no
semantic changes.

Signed-off-by: Alberto Garcia <berto@igalia.com>
Reviewed-by: Max Reitz <mreitz@redhat.com>
Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Message-Id: 
<396b6eab1859a271551dcd7dcba77f8934aa3c3f.1594396418.git.berto@igalia.com>
Signed-off-by: Max Reitz <mreitz@redhat.com>


  Commit: 10dabdc5966759f8c6d9bba5ca80f4ec621f4636
      
https://github.com/qemu/qemu/commit/10dabdc5966759f8c6d9bba5ca80f4ec621f4636
  Author: Alberto Garcia <berto@igalia.com>
  Date:   2020-08-25 (Tue, 25 Aug 2020)

  Changed paths:
    M block/qcow2-cluster.c
    M block/qcow2.c
    M block/qcow2.h

  Log Message:
  -----------
  qcow2: Replace QCOW2_CLUSTER_* with QCOW2_SUBCLUSTER_*

In order to support extended L2 entries some functions of the qcow2
driver need to start dealing with subclusters instead of clusters.

qcow2_get_host_offset() is modified to return the subcluster type
instead of the cluster type, and all callers are updated to replace
all values of QCow2ClusterType with their QCow2SubclusterType
equivalents.

This patch only changes the data types, there are no semantic changes.

Signed-off-by: Alberto Garcia <berto@igalia.com>
Reviewed-by: Max Reitz <mreitz@redhat.com>
Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Message-Id: 
<f6c29737c295f32cbee74c903c30b01820363b34.1594396418.git.berto@igalia.com>
Signed-off-by: Max Reitz <mreitz@redhat.com>


  Commit: 97490a143ee7db264ae91623e3c1a4762835911b
      
https://github.com/qemu/qemu/commit/97490a143ee7db264ae91623e3c1a4762835911b
  Author: Alberto Garcia <berto@igalia.com>
  Date:   2020-08-25 (Tue, 25 Aug 2020)

  Changed paths:
    M block/qcow2.c

  Log Message:
  -----------
  qcow2: Handle QCOW2_SUBCLUSTER_UNALLOCATED_ALLOC

When dealing with subcluster types there is a new value called
QCOW2_SUBCLUSTER_UNALLOCATED_ALLOC that has no equivalent in
QCow2ClusterType.

This patch handles that value in all places where subcluster types
are processed.

Signed-off-by: Alberto Garcia <berto@igalia.com>
Reviewed-by: Max Reitz <mreitz@redhat.com>
Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Message-Id: 
<bf09e2e2439a468a901bb96ace411eed9ee50295.1594396418.git.berto@igalia.com>
Signed-off-by: Max Reitz <mreitz@redhat.com>


  Commit: d53ec3d8d8e12d5a7c7f48347be450af56040b1a
      
https://github.com/qemu/qemu/commit/d53ec3d8d8e12d5a7c7f48347be450af56040b1a
  Author: Alberto Garcia <berto@igalia.com>
  Date:   2020-08-25 (Tue, 25 Aug 2020)

  Changed paths:
    M block/qcow2-cluster.c

  Log Message:
  -----------
  qcow2: Add subcluster support to calculate_l2_meta()

If an image has subclusters then there are more copy-on-write
scenarios that we need to consider. Let's say we have a write request
from the middle of subcluster #3 until the end of the cluster:

1) If we are writing to a newly allocated cluster then we need
   copy-on-write. The previous contents of subclusters #0 to #3 must
   be copied to the new cluster. We can optimize this process by
   skipping all leading unallocated or zero subclusters (the status of
   those skipped subclusters will be reflected in the new L2 bitmap).

2) If we are overwriting an existing cluster:

   2.1) If subcluster #3 is unallocated or has the all-zeroes bit set
        then we need copy-on-write (on subcluster #3 only).

   2.2) If subcluster #3 was already allocated then there is no need
        for any copy-on-write. However we still need to update the L2
        bitmap to reflect possible changes in the allocation status of
        subclusters #4 to #31. Because of this, this function checks
        if all the overwritten subclusters are already allocated and
        in this case it returns without creating a new QCowL2Meta
        structure.

After all these changes l2meta_cow_start() and l2meta_cow_end()
are not necessarily cluster-aligned anymore. We need to update the
calculation of old_start and old_end in handle_dependencies() to
guarantee that no two requests try to write on the same cluster.

Signed-off-by: Alberto Garcia <berto@igalia.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Reviewed-by: Max Reitz <mreitz@redhat.com>
Message-Id: 
<4292dd56e4446d386a2fe307311737a711c00708.1594396418.git.berto@igalia.com>
Signed-off-by: Max Reitz <mreitz@redhat.com>


  Commit: 3f9c6b3b1f48d6c59d27df658f03792b857d7b55
      
https://github.com/qemu/qemu/commit/3f9c6b3b1f48d6c59d27df658f03792b857d7b55
  Author: Alberto Garcia <berto@igalia.com>
  Date:   2020-08-25 (Tue, 25 Aug 2020)

  Changed paths:
    M block/qcow2-cluster.c
    M block/qcow2.h

  Log Message:
  -----------
  qcow2: Add subcluster support to qcow2_get_host_offset()

The logic of this function remains pretty much the same, except that
it uses count_contiguous_subclusters(), which combines the logic of
count_contiguous_clusters() / count_contiguous_clusters_unallocated()
and checks individual subclusters.

qcow2_cluster_to_subcluster_type() is not necessary as a separate
function anymore so it's inlined into its caller.

Signed-off-by: Alberto Garcia <berto@igalia.com>
Reviewed-by: Max Reitz <mreitz@redhat.com>
Message-Id: 
<d2193fd48653a350d80f0eca1c67b1d9053fb2f3.1594396418.git.berto@igalia.com>
[mreitz: Initialize expected_type to anything]
Signed-off-by: Max Reitz <mreitz@redhat.com>


  Commit: 205fa507508e1f8caa79325bafb3cd2ae8814125
      
https://github.com/qemu/qemu/commit/205fa507508e1f8caa79325bafb3cd2ae8814125
  Author: Alberto Garcia <berto@igalia.com>
  Date:   2020-08-25 (Tue, 25 Aug 2020)

  Changed paths:
    M block/qcow2-cluster.c

  Log Message:
  -----------
  qcow2: Add subcluster support to zero_in_l2_slice()

The QCOW_OFLAG_ZERO bit that indicates that a cluster reads as
zeroes is only used in standard L2 entries. Extended L2 entries use
individual 'all zeroes' bits for each subcluster.

This must be taken into account when updating the L2 entry and also
when deciding that an existing entry does not need to be updated.

Signed-off-by: Alberto Garcia <berto@igalia.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Reviewed-by: Max Reitz <mreitz@redhat.com>
Message-Id: 
<b61d61606d8c9b367bd641ab37351ddb9172799a.1594396418.git.berto@igalia.com>
Signed-off-by: Max Reitz <mreitz@redhat.com>


  Commit: a68cd70326e8a1ee710d420c64693c741403897c
      
https://github.com/qemu/qemu/commit/a68cd70326e8a1ee710d420c64693c741403897c
  Author: Alberto Garcia <berto@igalia.com>
  Date:   2020-08-25 (Tue, 25 Aug 2020)

  Changed paths:
    M block/qcow2-cluster.c

  Log Message:
  -----------
  qcow2: Add subcluster support to discard_in_l2_slice()

Two things need to be taken into account here:

1) With full_discard == true the L2 entry must be cleared completely.
   This also includes the L2 bitmap if the image has extended L2
   entries.

2) With full_discard == false we have to make the discarded cluster
   read back as zeroes. With normal L2 entries this is done with the
   QCOW_OFLAG_ZERO bit, whereas with extended L2 entries this is done
   with the individual 'all zeroes' bits for each subcluster.

   Note however that QCOW_OFLAG_ZERO is not supported in v2 qcow2
   images so, if there is a backing file, discard cannot guarantee
   that the image will read back as zeroes. If this is important for
   the caller it should forbid it as qcow2_co_pdiscard() does (see
   80f5c01183 for more details).

Signed-off-by: Alberto Garcia <berto@igalia.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Reviewed-by: Max Reitz <mreitz@redhat.com>
Message-Id: 
<5ef8274e628aa3ab559bfac467abf488534f2b76.1594396418.git.berto@igalia.com>
Signed-off-by: Max Reitz <mreitz@redhat.com>


  Commit: fc2e6528d5c14a15cac586406cb72d0127449178
      
https://github.com/qemu/qemu/commit/fc2e6528d5c14a15cac586406cb72d0127449178
  Author: Alberto Garcia <berto@igalia.com>
  Date:   2020-08-25 (Tue, 25 Aug 2020)

  Changed paths:
    M block/qcow2-refcount.c
    M tests/qemu-iotests/060.out

  Log Message:
  -----------
  qcow2: Add subcluster support to check_refcounts_l2()

The offset field of an uncompressed cluster's L2 entry must be aligned
to the cluster size, otherwise it is invalid. If the cluster has no
data then it means that the offset points to a preallocation, so we
can clear the offset field without affecting the guest-visible data.
This is what 'qemu-img check' does when run in repair mode.

On traditional qcow2 images this can only happen when QCOW_OFLAG_ZERO
is set, and repairing such entries turns the clusters from ZERO_ALLOC
into ZERO_PLAIN.

Extended L2 entries have no ZERO_ALLOC clusters and no QCOW_OFLAG_ZERO
but the idea is the same: if none of the subclusters are allocated
then we can clear the offset field and leave the bitmap untouched.

Signed-off-by: Alberto Garcia <berto@igalia.com>
Reviewed-by: Max Reitz <mreitz@redhat.com>
Message-Id: 
<9f4ed1d0a34b0a545b032c31ecd8c14734065342.1594396418.git.berto@igalia.com>
Signed-off-by: Max Reitz <mreitz@redhat.com>


  Commit: aca00cd971c830ad2a71cc881db33275d0a4107c
      
https://github.com/qemu/qemu/commit/aca00cd971c830ad2a71cc881db33275d0a4107c
  Author: Alberto Garcia <berto@igalia.com>
  Date:   2020-08-25 (Tue, 25 Aug 2020)

  Changed paths:
    M block/qcow2-cluster.c

  Log Message:
  -----------
  qcow2: Update L2 bitmap in qcow2_alloc_cluster_link_l2()

The L2 bitmap needs to be updated after each write to indicate what
new subclusters are now allocated. This needs to happen even if the
cluster was already allocated and the L2 entry was otherwise valid.

In some cases however a write operation doesn't need change the L2
bitmap (because all affected subclusters were already allocated). This
is detected in calculate_l2_meta(), and qcow2_alloc_cluster_link_l2()
is never called in those cases.

Signed-off-by: Alberto Garcia <berto@igalia.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Reviewed-by: Max Reitz <mreitz@redhat.com>
Message-Id: 
<0875620d49f44320334b6a91c73b3f301f975f38.1594396418.git.berto@igalia.com>
Signed-off-by: Max Reitz <mreitz@redhat.com>


  Commit: ff4cdec7f6e7302d2a82fdb627373c32aecb3dfe
      
https://github.com/qemu/qemu/commit/ff4cdec7f6e7302d2a82fdb627373c32aecb3dfe
  Author: Alberto Garcia <berto@igalia.com>
  Date:   2020-08-25 (Tue, 25 Aug 2020)

  Changed paths:
    M block/qcow2-cluster.c

  Log Message:
  -----------
  qcow2: Clear the L2 bitmap when allocating a compressed cluster

Compressed clusters always have the bitmap part of the extended L2
entry set to 0.

Signed-off-by: Alberto Garcia <berto@igalia.com>
Reviewed-by: Max Reitz <mreitz@redhat.com>
Message-Id: 
<04455b3de5dfeb9d1cfe1fc7b02d7060a6e09710.1594396418.git.berto@igalia.com>
Signed-off-by: Max Reitz <mreitz@redhat.com>


  Commit: bf4a66eed423694cfa0b3b5692211264022e2b98
      
https://github.com/qemu/qemu/commit/bf4a66eed423694cfa0b3b5692211264022e2b98
  Author: Alberto Garcia <berto@igalia.com>
  Date:   2020-08-25 (Tue, 25 Aug 2020)

  Changed paths:
    M block/qcow2.c

  Log Message:
  -----------
  qcow2: Add subcluster support to handle_alloc_space()

The bdrv_co_pwrite_zeroes() call here fills complete clusters with
zeroes, but it can happen that some subclusters are not part of the
write request or the copy-on-write. This patch makes sure that only
the affected subclusters are overwritten.

A potential improvement would be to also fill with zeroes the other
subclusters if we can guarantee that we are not overwriting existing
data. However this would waste more disk space, so we should first
evaluate if it's really worth doing.

Signed-off-by: Alberto Garcia <berto@igalia.com>
Reviewed-by: Max Reitz <mreitz@redhat.com>
Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Message-Id: 
<b3dc97e8e2240ddb5191a4f930e8fc9653f94621.1594396418.git.berto@igalia.com>
Signed-off-by: Max Reitz <mreitz@redhat.com>


  Commit: a6841a2de66fa44fe52ed996b70f9fb9f7bd6ca7
      
https://github.com/qemu/qemu/commit/a6841a2de66fa44fe52ed996b70f9fb9f7bd6ca7
  Author: Alberto Garcia <berto@igalia.com>
  Date:   2020-08-25 (Tue, 25 Aug 2020)

  Changed paths:
    M block/qcow2-cluster.c
    M block/qcow2.c
    M block/qcow2.h

  Log Message:
  -----------
  qcow2: Add subcluster support to qcow2_co_pwrite_zeroes()

This works now at the subcluster level and pwrite_zeroes_alignment is
updated accordingly.

qcow2_cluster_zeroize() is turned into qcow2_subcluster_zeroize() with
the following changes:

   - The request can now be subcluster-aligned.

   - The cluster-aligned body of the request is still zeroized using
     zero_in_l2_slice() as before.

   - The subcluster-aligned head and tail of the request are zeroized
     with the new zero_l2_subclusters() function.

There is just one thing to take into account for a possible future
improvement: compressed clusters cannot be partially zeroized so
zero_l2_subclusters() on the head or the tail can return -ENOTSUP.
This makes the caller repeat the *complete* request and write actual
zeroes to disk. This is sub-optimal because

   1) if the head area was compressed we would still be able to use
      the fast path for the body and possibly the tail.

   2) if the tail area was compressed we are writing zeroes to the
      head and the body areas, which are already zeroized.

Signed-off-by: Alberto Garcia <berto@igalia.com>
Reviewed-by: Max Reitz <mreitz@redhat.com>
Message-Id: 
<17e05e2ee7e12f10dcf012da81e83ebe27eb3bef.1594396418.git.berto@igalia.com>
Signed-off-by: Max Reitz <mreitz@redhat.com>


  Commit: 0dd07b298fdecf13e89c1379b903eb9bb0dac3a3
      
https://github.com/qemu/qemu/commit/0dd07b298fdecf13e89c1379b903eb9bb0dac3a3
  Author: Alberto Garcia <berto@igalia.com>
  Date:   2020-08-25 (Tue, 25 Aug 2020)

  Changed paths:
    M block/qcow2.c

  Log Message:
  -----------
  qcow2: Add subcluster support to qcow2_measure()

Extended L2 entries are bigger than normal L2 entries so this has an
impact on the amount of metadata needed for a qcow2 file.

Signed-off-by: Alberto Garcia <berto@igalia.com>
Reviewed-by: Max Reitz <mreitz@redhat.com>
Message-Id: 
<7efae2efd5e36b42d2570743a12576d68ce53685.1594396418.git.berto@igalia.com>
Signed-off-by: Max Reitz <mreitz@redhat.com>


  Commit: 40dee94320c9d76a656bd799870f9a81cf520b60
      
https://github.com/qemu/qemu/commit/40dee94320c9d76a656bd799870f9a81cf520b60
  Author: Alberto Garcia <berto@igalia.com>
  Date:   2020-08-25 (Tue, 25 Aug 2020)

  Changed paths:
    M block/qcow2-cluster.c
    M block/qcow2.c
    M block/qcow2.h

  Log Message:
  -----------
  qcow2: Add prealloc field to QCowL2Meta

This field allows us to indicate that the L2 metadata update does not
come from a write request with actual data but from a preallocation
request.

For traditional images this does not make any difference, but for
images with extended L2 entries this means that the clusters are
allocated normally in the L2 table but individual subclusters are
marked as unallocated.

This will allow preallocating images that have a backing file.

There is one special case: when we resize an existing image we can
also request that the new clusters are preallocated. If the image
already had a backing file then we have to hide any possible stale
data and zero out the new clusters (see commit 955c7d6687 for more
details).

In this case the subclusters cannot be left as unallocated so the L2
bitmap must be updated.

Signed-off-by: Alberto Garcia <berto@igalia.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Reviewed-by: Max Reitz <mreitz@redhat.com>
Message-Id: 
<960d4c444a4f5a870e2b47e5da322a73cd9a2f5a.1594396418.git.berto@igalia.com>
Signed-off-by: Max Reitz <mreitz@redhat.com>


  Commit: 7be20252588422ab92464250d60063997ad29d59
      
https://github.com/qemu/qemu/commit/7be20252588422ab92464250d60063997ad29d59
  Author: Alberto Garcia <berto@igalia.com>
  Date:   2020-08-25 (Tue, 25 Aug 2020)

  Changed paths:
    M block/qcow2.c
    M block/qcow2.h
    M include/block/block_int.h
    M qapi/block-core.json
    M tests/qemu-iotests/031.out
    M tests/qemu-iotests/036.out
    M tests/qemu-iotests/049.out
    M tests/qemu-iotests/060.out
    M tests/qemu-iotests/061.out
    M tests/qemu-iotests/065
    M tests/qemu-iotests/082.out
    M tests/qemu-iotests/085.out
    M tests/qemu-iotests/144.out
    M tests/qemu-iotests/182.out
    M tests/qemu-iotests/185.out
    M tests/qemu-iotests/198
    M tests/qemu-iotests/206.out
    M tests/qemu-iotests/242.out
    M tests/qemu-iotests/255.out
    M tests/qemu-iotests/274.out
    M tests/qemu-iotests/280.out
    M tests/qemu-iotests/291.out
    M tests/qemu-iotests/302.out
    M tests/qemu-iotests/303.out
    M tests/qemu-iotests/common.filter

  Log Message:
  -----------
  qcow2: Add the 'extended_l2' option and the QCOW2_INCOMPAT_EXTL2 bit

Now that the implementation of subclusters is complete we can finally
add the necessary options to create and read images with this feature,
which we call "extended L2 entries".

Signed-off-by: Alberto Garcia <berto@igalia.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Reviewed-by: Max Reitz <mreitz@redhat.com>
Message-Id: 
<6476caaa73216bd05b7bb2d504a20415e1665176.1594396418.git.berto@igalia.com>
[mreitz: %s/5\.1/5.2/; fixed 302's and 303's reference output]
Signed-off-by: Max Reitz <mreitz@redhat.com>


  Commit: 2118771ddf27dbfd6fb0b1e611423aa6cd72e160
      
https://github.com/qemu/qemu/commit/2118771ddf27dbfd6fb0b1e611423aa6cd72e160
  Author: Alberto Garcia <berto@igalia.com>
  Date:   2020-08-25 (Tue, 25 Aug 2020)

  Changed paths:
    M block/qcow2.c
    M tests/qemu-iotests/206.out

  Log Message:
  -----------
  qcow2: Allow preallocation and backing files if extended_l2 is set

Traditional qcow2 images don't allow preallocation if a backing file
is set. This is because once a cluster is allocated there is no way to
tell that its data should be read from the backing file.

Extended L2 entries have individual allocation bits for each
subcluster, and therefore it is perfectly possible to have an
allocated cluster with all its subclusters unallocated.

Signed-off-by: Alberto Garcia <berto@igalia.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Reviewed-by: Max Reitz <mreitz@redhat.com>
Message-Id: 
<6d5b0f38e7dc5f2f31d8cab1cb92044e9909aece.1594396418.git.berto@igalia.com>
Signed-off-by: Max Reitz <mreitz@redhat.com>


  Commit: 7bbb59202a538b12df5840f250899ac3ec39a069
      
https://github.com/qemu/qemu/commit/7bbb59202a538b12df5840f250899ac3ec39a069
  Author: Alberto Garcia <berto@igalia.com>
  Date:   2020-08-25 (Tue, 25 Aug 2020)

  Changed paths:
    M block/qcow2-cluster.c
    M tests/qemu-iotests/061
    M tests/qemu-iotests/061.out

  Log Message:
  -----------
  qcow2: Assert that expand_zero_clusters_in_l1() does not support subclusters

This function is only used by qcow2_expand_zero_clusters() to
downgrade a qcow2 image to a previous version. This would require
transforming all extended L2 entries into normal L2 entries but this
is not a simple task and there are no plans to implement this at the
moment.

Signed-off-by: Alberto Garcia <berto@igalia.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Reviewed-by: Max Reitz <mreitz@redhat.com>
Message-Id: 
<15e65112b4144381b4d8c0bdf8fb76b0d813e3d1.1594396418.git.berto@igalia.com>
[mreitz: Fixed comment style]
Signed-off-by: Max Reitz <mreitz@redhat.com>


  Commit: a5d3cfa2dc775e5d99f013703b8508f1d989d588
      
https://github.com/qemu/qemu/commit/a5d3cfa2dc775e5d99f013703b8508f1d989d588
  Author: Alberto Garcia <berto@igalia.com>
  Date:   2020-08-26 (Wed, 26 Aug 2020)

  Changed paths:
    A tests/qemu-iotests/271
    A tests/qemu-iotests/271.out
    M tests/qemu-iotests/group

  Log Message:
  -----------
  iotests: Add tests for qcow2 images with extended L2 entries

Signed-off-by: Alberto Garcia <berto@igalia.com>
Reviewed-by: Max Reitz <mreitz@redhat.com>
Message-Id: 
<e6dd0429cafe84ca603179c298a8703bddca2904.1594396418.git.berto@igalia.com>
[mreitz: Use env in shebang line]
Signed-off-by: Max Reitz <mreitz@redhat.com>


  Commit: 25f6dc28a3a8dd231c2c092a0e65bd796353c769
      
https://github.com/qemu/qemu/commit/25f6dc28a3a8dd231c2c092a0e65bd796353c769
  Author: Peter Maydell <peter.maydell@linaro.org>
  Date:   2020-08-26 (Wed, 26 Aug 2020)

  Changed paths:
    M block/qcow2-cluster.c
    M block/qcow2-refcount.c
    M block/qcow2.c
    M block/qcow2.h
    M block/trace-events
    M docs/interop/qcow2.txt
    M docs/qcow2-cache.txt
    M include/block/block_int.h
    M qapi/block-core.json
    M tests/qemu-iotests/031.out
    M tests/qemu-iotests/036.out
    M tests/qemu-iotests/049.out
    M tests/qemu-iotests/060.out
    M tests/qemu-iotests/061
    M tests/qemu-iotests/061.out
    M tests/qemu-iotests/065
    M tests/qemu-iotests/082.out
    M tests/qemu-iotests/085.out
    M tests/qemu-iotests/144.out
    M tests/qemu-iotests/182.out
    M tests/qemu-iotests/185.out
    M tests/qemu-iotests/198
    M tests/qemu-iotests/206.out
    M tests/qemu-iotests/242.out
    M tests/qemu-iotests/255.out
    A tests/qemu-iotests/271
    A tests/qemu-iotests/271.out
    M tests/qemu-iotests/274.out
    M tests/qemu-iotests/280.out
    M tests/qemu-iotests/291.out
    M tests/qemu-iotests/302.out
    M tests/qemu-iotests/303.out
    M tests/qemu-iotests/common.filter
    M tests/qemu-iotests/group

  Log Message:
  -----------
  Merge remote-tracking branch 'remotes/maxreitz/tags/pull-block-2020-08-26' 
into staging

Block patches:
- qcow2 subclusters (extended L2 entries)

# gpg: Signature made Wed 26 Aug 2020 08:37:04 BST
# gpg:                using RSA key 91BEB60A30DB3E8857D11829F407DB0061D5CF40
# gpg:                issuer "mreitz@redhat.com"
# gpg: Good signature from "Max Reitz <mreitz@redhat.com>" [full]
# Primary key fingerprint: 91BE B60A 30DB 3E88 57D1  1829 F407 DB00 61D5 CF40

* remotes/maxreitz/tags/pull-block-2020-08-26: (34 commits)
  iotests: Add tests for qcow2 images with extended L2 entries
  qcow2: Assert that expand_zero_clusters_in_l1() does not support subclusters
  qcow2: Allow preallocation and backing files if extended_l2 is set
  qcow2: Add the 'extended_l2' option and the QCOW2_INCOMPAT_EXTL2 bit
  qcow2: Add prealloc field to QCowL2Meta
  qcow2: Add subcluster support to qcow2_measure()
  qcow2: Add subcluster support to qcow2_co_pwrite_zeroes()
  qcow2: Add subcluster support to handle_alloc_space()
  qcow2: Clear the L2 bitmap when allocating a compressed cluster
  qcow2: Update L2 bitmap in qcow2_alloc_cluster_link_l2()
  qcow2: Add subcluster support to check_refcounts_l2()
  qcow2: Add subcluster support to discard_in_l2_slice()
  qcow2: Add subcluster support to zero_in_l2_slice()
  qcow2: Add subcluster support to qcow2_get_host_offset()
  qcow2: Add subcluster support to calculate_l2_meta()
  qcow2: Handle QCOW2_SUBCLUSTER_UNALLOCATED_ALLOC
  qcow2: Replace QCOW2_CLUSTER_* with QCOW2_SUBCLUSTER_*
  qcow2: Add cluster type parameter to qcow2_get_host_offset()
  qcow2: Add qcow2_cluster_is_allocated()
  qcow2: Add qcow2_get_subcluster_range_type()
  ...

Signed-off-by: Peter Maydell <peter.maydell@linaro.org>


Compare: https://github.com/qemu/qemu/compare/78dca230c97e...25f6dc28a3a8