[Qemu-block] 2018-07-09 Incremental Backup Status Report

[John Snow]

On 07/02/2018 07:47 PM, John Snow wrote:
> I'm tracking the following series for Incremental Backup right now; if
> it's not on this list, I don't know about it:
> 
> [Qemu-devel] [PATCH v4 0/2] block: formalize and test fleecing​

Merged.

> [Qemu-devel] [PATCH v2 0/3] image fleecing

Dropped in favor of the above, but we've then added:

[Qemu-devel] [PATCH v5 0/4] fix image fleecing

Which should have the same effect as preventing race conditions between
serializations that exist across two nodes.

> [Qemu-devel] [PATCH] migration: Appease coverity, skip empty block trees

I need to spin a v2.

> [Qemu-devel] [PATCH v2 0/2] test NBD bitmap export

Merged.

> [Qemu-devel] [PATCH 0/6] fix persistent bitmaps migration logic​

Needs review.

> [Qemu-devel] [PATCH 0/2] dirty bitmap fixes​

Merged.

> [Qemu-devel] [PATCH v2] qcow2: add overlap check for bitmap directory

Merged.

> [Qemu-devel] [PATCH 0/1] blockdev: implement n-ary bitmap merge

Needs review; it's not essential but I suspect it's likely nice to have
for libvirt.

> [Qemu-devel] [PATCH v2 00/10] qemu-img: add bitmap info output​
> 

On hold pending "fix persistent bitmaps migration logic."


New series:
- [PATCH v2 0/4] transaction support for bitmap merge



New status list looks like this, then:

[Qemu-devel] [PATCH] migration: Appease coverity, skip empty block trees
[Qemu-devel] [PATCH v5 0/4] fix image fleecing
[Qemu-devel] [PATCH 0/6] fix persistent bitmaps migration logic​
[Qemu-devel] [PATCH 0/1] blockdev: implement n-ary bitmap merge
[Qemu-devel] [PATCH v2 00/10] qemu-img: add bitmap info output​
[Qemu-devel] [PATCH v2 0/4] transaction support for bitmap merge

I'll be looking at #3, #6 and #1 tomorrow.

Thanks,
--js


> 
> Now, in a little more detail below;
> 
> 
> # Image fleecing:
> 
> In order to support pull-model semantics for incremental backup, we need
> working point in time NBD export of live disk images. This was thought
> supported/finished for quite some time, but we've had no iotests.
> 
> There are two series in flight here:
> 
> - [Qemu-devel] [PATCH v4 0/2] block: formalize and test fleecing​
> - [Qemu-devel] [PATCH v2 0/3] image fleecing
> 
> The first listed there simply loosens restrictions on blockdev-backup
> and adds an iotest that this approach works in the simple case.
> 
> The second series adds a fleecing filter to combat deficiencies in the
> IO serialization that may allow the fleecing node to sometimes see new,
> live data instead of the PIT data. See
> https://lists.gnu.org/archive/html/qemu-devel/2018-07/msg00150.html for
> details.
> 
> Our current plan is to merge the first series and to continue looking
> into the fleecing filter during freeze as a potential bugfix.
> 
> 
> # NBD:
> 
> - [Qemu-devel] [PATCH v2 0/2] test NBD bitmap export
> 
> Eric's series that adds a quick hack to our NBD client and an iotest 223
> that demos that the bitmap support in our NBD server is working
> correctly. Notably, we don't have an iotest that combines this with the
> image fleecing workflow, so there's discussion there that I've started
> for how to combine the two, comments appreciated:
> https://lists.gnu.org/archive/html/qemu-devel/2018-07/msg00476.html
> 
> 
> # Migration/Persistence:
> 
> - [Qemu-devel] [PATCH 0/6] fix persistent bitmaps migration logic​
> 
> Needs review, but can go in after freeze as a fix.
> 
> 
> # Bitmap manipulation:
> 
> - [Qemu-devel] [PATCH 0/1] blockdev: implement n-ary bitmap merge​
> 
> Possibly important to have if we wish to add merge into transactions, as
> this will greatly simplify the commands needed to be issued to QEMU. Not
> yet reviewed, but I might submit a PR for this tomorrow in anticipation
> of its usefulness over the 1-bitmap version.
> 
> - [Qemu-devel] [PATCH v2 00/10] qemu-img: add bitmap info output​
> 
> On hold until persistence is settled, due to interactions with
> attempting to cache the bitmap metadata. "offline" bitmap manipulation
> will need to occur via QMP on QEMU with no running guest in the
> meantime. We will still need a patch for `qemu-img check -r` that allows
> us to delete corrupted bitmaps, however. That can be a bugfix for 3.0.
> 
> There is a follow-up series that's also on hold because it's blocked on
> this series, that adds the actual add/delete/clear offline functionality.
> 
> 
> # Miscellaneous:
> 
> - [Qemu-devel] [PATCH] migration: Appease coverity, skip empty block trees
> 
> Needs a respin based on Stefan's feedback. Only important so that it
> doesn't keep annoying Peter Maydell, does not actually fix a bug.
> 
> - [Qemu-devel] [PATCH 0/2] dirty bitmap fixes​
> 
> Staged: https://github.com/jnsnow/qemu/commits/bitmaps
> 
> - [Qemu-devel] [PATCH v2] qcow2: add overlap check for bitmap directory
> 
> Needs a V3 after review from Max. Not crucial for release, but nice to have.
> 
> 
> 
> 
> So in order to put together a demo from QEMU's point-of-view that both
> push and pull model are working, I think we still need:
> 
> - iotest 222 and 223 merged, with their respective enablement patches.
> - A new iotest (224?) that combines functionality of both; which makes
> use of perhaps some new features that allow us to tie bitmap
> manipulation to the start of the node fleecing.
> - All of the various bugfixes above reviewed and merged.
> 
> 
> 
> Please let me know if this roadmap is missing anything.
> 
> Thanks,
> --js
>