[Qemu-devel] [RFC V6 00/33] QCOW2 deduplication core functionality

[Benoît Canet]

This patchset create the core infrastructure for deduplication and enable it.

One can compile and install https://github.com/wernerd/Skein3Fish and use the
--enable-skein-dedup configure option in order to use the faster skein HASH.

Images must be created with "-o dedup=[skein|sha256]" in order to activate the
deduplication in the image.

Deduplication is now fast enough to be usable.
Nice side effect is that duplicated writes are faster than native QCOW2:

v6:
    Fix typo in "Drop hash for a given cluster..." commits message [Eric]
    Fix spurious whitespace change in "qcow2: Add qcow2_dedup_read_mis..." 
[Eric]
    Fix spelling mistake in "qcow2: Add qcow2_dedup_read_mis..." [Eric]
    Make #defines for deduplication strategies and use them [Eric]
    Specify that refcount_order must be >= 4 in qcow2 spec [Eric]
    Remove LAZY_REFCOUNT [Benoît]
    Do not modifigy L2 Size [Benoît]

v5:
    Move qemu-io-test dedup patch [Eric]
    Reserve some room at the end of the QCOW header extensions. [Eric]
    Fix the specification. [Eric]
    Now overflow deduplication refcount at 2^16/2 [Stefan]
    Increase L2 table size and deduplication block hash size.

v4: Fix and complete qcow2 spec [Stefan]
    Hash the hash_algo field in the header extension [Stefan]
    Fix qcow2 spec [Eric]
    Remove pointer to hash and simplify hash memory management [Stefan]
    Rename and move qcow2_read_cluster_data to qcow2.c [Stefan]
    Document lock dropping behaviour of the previous function [Stefan]
    cleanup qcow2_dedup_read_missing_cluster_data [Stefan]
    rename *_offset to *_sect [Stefan]
    add a ./configure check for ssl [Stefan]
    Replace openssl by gnutls [Stefan]
    Implement Skein hashes
    Rewrite pretty every qcow2-dedup.c commits after Add
       qcow2_dedup_read_missing_and_concatenate to simplify the code
    Use 64KB deduplication hash block to reduce allocation flushes
    Use 64KB l2 tables to reduce allocation flushes [breaks compatibility]
    Use lazy refcounts to avoid qcow2_cache_set_dependency loops resultings
       in frequent caches flushes
    Do not create and load dedup RAM structures when bdrs->read_only is true

v3: make it work barely
    replace kernel red black trees by gtree.
*** BLURB HERE ***

Benoît Canet (33):
  qcow2: Add deduplication to the qcow2 specification.
  qmp: Add DedupStatus enum.
  qcow2: Add deduplication structures and fields.
  qcow2: Add qcow2_dedup_read_missing_and_concatenate
  qcow2: Make update_refcount public.
  qcow2: Create a way to link to l2 tables when deduplicating.
  qcow2: Add qcow2_dedup and related functions
  qcow2: Add qcow2_dedup_store_new_hashes.
  qcow2: Implement qcow2_compute_cluster_hash.
  qcow2: Extract qcow2_dedup_grow_table
  qcow2: Add qcow2_dedup_grow_table and use it.
  qcow2: Makes qcow2_alloc_cluster_link_l2 mark to deduplicate
    clusters.
  qcow2: make the deduplication forget a cluster hash when a cluster is
    to dedupe
  qcow2: Create qcow2_is_cluster_to_dedup.
  qcow2: Load and save deduplication table header extension.
  qcow2: Extract qcow2_do_table_init.
  qcow2-cache: Allow to choose table size at creation.
  qcow2: Extract qcow2_add_feature and qcow2_remove_feature.
  block: Add qcow2_dedup format and image creation code.
  qcow2: Add a deduplication boolean to update_refcount.
  qcow2: Drop hash for a given cluster when dedup makes refcount >
    2^16/2.
  qcow2: Remove hash when cluster is deleted.
  qcow2: Add qcow2_dedup_is_running to probe if dedup is running.
  qcow2: Integrate deduplication in qcow2_co_writev loop.
  qcow2: Serialize write requests when deduplication is activated.
  qcow2: Add verification of dedup table.
  qcow2: Adapt checking of QCOW_OFLAG_COPIED for dedup.
  qcow2: Add check_dedup_l2 in order to check l2 of dedup table.
  qcow2: Do not overwrite existing entries with QCOW_OFLAG_COPIED.
  qcow2: Integrate SKEIN hash algorithm in deduplication.
  qcow: Set large dedup hash block size.
  qemu-iotests: Filter dedup=on/off so existing tests don't break.
  qcow2: Add qcow2_dedup_init and qcow2_dedup_close.

 block/Makefile.objs          |    1 +
 block/qcow2-cache.c          |   12 +-
 block/qcow2-cluster.c        |  180 +++++--
 block/qcow2-dedup.c          | 1118 ++++++++++++++++++++++++++++++++++++++++++
 block/qcow2-refcount.c       |  157 ++++--
 block/qcow2.c                |  427 ++++++++++++++--
 block/qcow2.h                |  141 +++++-
 configure                    |   55 +++
 docs/specs/qcow2.txt         |  105 +++-
 include/block/block_int.h    |    1 +
 qapi-schema.json             |   18 +
 tests/qemu-iotests/common.rc |    3 +-
 12 files changed, 2092 insertions(+), 126 deletions(-)
 create mode 100644 block/qcow2-dedup.c

-- 
1.7.10.4