Re: [Qemu-block] [Qemu-devel] [PATCH for-2.12 0/4] qmp dirty bitmap API

[John Snow]

On 12/11/2017 07:18 AM, Vladimir Sementsov-Ogievskiy wrote:
> 11.12.2017 14:15, Kevin Wolf wrote:
>> Am 09.12.2017 um 01:57 hat John Snow geschrieben:
>>> Here's an idea of what this API might look like without revealing
>>> explicit merge/split primitives.
>>>
>>> A new bitmap property that lets us set retention:
>>>
>>> :: block-dirty-bitmap-set-retention bitmap=foo slices=10
>>>
>>> Or something similar, where the default property for all bitmaps is
>>> zero -- the current behavior: no copies retained.
>>>
>>> By setting it to a non-zero positive integer, the incremental backup
>>> mode will automatically save a disabled copy when possible.
>> -EMAGIC
>>
>> Operations that create or delete user-visible objects should be
>> explicit, not automatic. You're trying to implement management layer
>> functionality in qemu here, but incomplete enough that the artifacts of
>> it are still visible externally. (A complete solution within qemu
>> wouldn't expose low-level concepts such as bitmaps on an external
>> interface, but you would expose something like checkpoints.)
>>
>> Usually it's not a good idea to have a design where qemu implements
>> enough to restrict management tools to whatever use case we had in mind,
>> but not enough to make the management tool's life substantially easier
>> (by not having to care about some low-level concepts).
>>
>>> "What happens if we exceed our retention?"
>>>
>>> (A) We push the last one out automatically, or
>>> (B) We fail the operation immediately.
>>>
>>> A is more convenient, but potentially unsafe if the management tool or
>>> user wasn't aware that was going to happen.
>>> B is more annoying, but definitely more safe as it means we cannot lose
>>> a bitmap accidentally.
>> Both mean that the management layer has not only to deal with the
>> deletion of bitmaps as it wants to have them, but also to keep the
>> retention counter somewhere and predict what qemu is going to do to the
>> bitmaps and whether any corrective action needs to be taken.
>>
>> This is making things more complex rather than simpler.
>>
>>> I would argue for B with perhaps a force-cycle=true|false that defaults
>>> to false to let management tools say "Yes, go ahead, remove the old one"
>>> with additionally some return to let us know it happened:
>>>
>>> {"return": {
>>>    "dropped-slices": [ {"bitmap0": 0}, ...]
>>> }}
>>>
>>> This would introduce some concept of bitmap slices into the mix as ID'd
>>> children of a bitmap. I would propose that these slices are numbered and
>>> monotonically increasing. "bitmap0" as an object starts with no slices,
>>> but every incremental backup creates slice 0, slice 1, slice 2, and so
>>> on. Even after we start deleting some, they stay ordered. These numbers
>>> then stand in for points in time.
>>>
>>> The counter can (must?) be reset and all slices forgotten when
>>> performing a full backup while providing a bitmap argument.
>>>
>>> "How can a user make use of the slices once they're made?"
>>>
>>> Let's consider something like mode=partial in contrast to
>>> mode=incremental, and an example where we have 6 prior slices:
>>> 0,1,2,3,4,5, (and, unnamed, the 'active' slice.)
>>>
>>> mode=partial bitmap=foo slice=4
>>>
>>> This would create a backup from slice 4 to the current time α. This
>>> includes all clusters from 4, 5, and the active bitmap.
>>>
>>> I don't think it is meaningful to define any end point that isn't the
>>> current time, so I've omitted that as a possibility.
>> John, what are you doing here? This adds option after option, and even
>> additional slice object, only complicating an easy thing more and more.
>> I'm not sure if that was your intention, but I feel I'm starting to
>> understand better how Linus's rants come about.
>>
>> Let me summarise what this means for management layer:
>>
>> * The management layer has to manage bitmaps. They have direct control
>>    over creation and deletion of bitmaps. So far so good.
>>
>> * It also has to manage slices in those bitmaps objects; and these
>>    slices are what contains the actual bitmaps. In order to identify a
>>    bitmap in qemu, you need:
>>
>>      a) the node name
>>      b) the bitmap ID, and
>>      c) the slice number
>>
>>    The slice number is assigned by qemu and libvirt has to wait until
>>    qemu tells it about the slice number of a newly created slice. If
>>    libvirt doesn't receive the reply to the command that started the
>>    block job, it needs to be able to query this information from qemu,
>>    e.g. in query-block-jobs.
>>
>> * Slices are automatically created when you start a backup job with a
>>    bitmap. It doesn't matter whether you even intend to do an incremental
>>    backup against this point in time. qemu knows better.
>>
>> * In order to delete a slice that you don't need any more, you have to
>>    create more slices (by doing more backups), but you don't get to
>>    decide which one is dropped. qemu helpfully just drops the oldest one.
>>    It doesn't matter if you want to keep an older one so you can do an
>>    incremental backup for a longer timespan. Don't worry about your
>>    backup strategy, qemu knows better.
>>
>> * Of course, just creating a new backup job doesn't mean that removing
>>    the old slice works, even if you give the respective option. That's
>>    what the 'dropped-slices' return is for. So once again wait for
>>    whatever qemu did and reproduce it in the data structures of the
>>    management tool. It's also more information that needs to be exposed
>>    in query-block-jobs because libvirt might miss the return value.
>>
>> * Hmm... What happens if you start n backup block jobs, with n > slices?
>>    Sounds like a great way to introduce subtle bugs in both qemu and the
>>    management layer.
>>
>> Do you really think working with this API would be fun for libvirt?
>>
>>> "Does a partial backup create a new point in time?"
>>>
>>> If yes: This means that the next incremental backup must necessarily be
>>> based off of the last partial backup that was made. This seems a little
>>> inconvenient. This would mean that point in time α becomes "slice 6."
>> Or based off any of the previous points in time, provided that qemu
>> didn't helpfully decide to delete it. Can't I still create a backup
>> starting from slice 4 then?
>>
>> Also, a more general question about incremental backup: How does it play
>> with snapshots? Shouldn't we expect that people sometimes use both
>> snapshots and backups? Can we restrict the backup job to considering
>> bitmaps only from a single node or should we be able to reference
>> bitmaps of a backing file as well?
>>
>>> If no: This means that we lose the point in time when we made the
>>> partial and we cannot chain off of the partial backup. It does mean that
>>> the next incremental backup will work as normally expected, however.
>>> This means that point in time α cannot again be referenced by the
>>> management client.
>>>
>>> This mirrors the dynamic between "incremental" and "differential"
>>> backups.
>>>
>>> ..hmmm..
>>>
>>> You know, incremental backups are just a special case of "partial" here
>>> where slice is the last recorded slice... Let's look at an API like
>>> this:
>>>
>>> mode=<incremental|differential> bitmap=<name> [slice=N]
>>>
>>> Incremental: We create a new slice if the bitmap has room for one.
>>> Differential: We don't create a new slice. The data in the active bitmap
>>> α does not get cleared after the bitmap operation.
>>>
>>> Slice:
>>> If not specified, assume we want only the active slice. This is the
>>> current behavior in QEMU 2.11.
>>> If specified, we create a temporary merge between bitmaps [N..α] and use
>>> that for the backup operation.
>>>
>>> "Can we delete slices?"
>>>
>>> Sure.
>>>
>>> :: block-dirty-bitmap-slice-delete bitmap=foo slice=4
>>>
>>> "Can we create a slice without making a bitmap?"
>>>
>>> It would be easy to do, but I'm not sure I see the utility. In using it,
>>> it means if you don't specify the slice manually for the next backup
>>> that you will necessarily be getting something not usable.
>>>
>>> but we COULD do it, it would just be banking the changes in the active
>>> bitmap into a new slice.
>> Okay, with explicit management this is getting a little more reasonable
>> now. However, I don't understand what slices buy us then compared to
>> just separate bitmaps.
>>
>> Essentially, bitmaps form a second kind of backing chain. Backup always
>> wants to use the combined bitmaps of some subchain. I see two easy ways
>> to do this: Either pass an array of bitmaps to consider to the job, or
>> store the "backing link" in the bitmap so that we can just specify a
>> "base bitmap" like we usually do with normal backing files.
>>
>> The backup block job can optionally append a new bitmap to the chain
>> like external snapshots do for backing chains. Deleting a bitmap in the
>> chain is the merge operation, similar to a commit block job for backing
>> chains.
>>
>> We know these mechanism very well because the block layer has been using
>> them for ages.
>>
>>>> I also have another idea:
>>>> implement new object: point-in-time or checkpoint. The should have
>>>> names, and the simple add/remove API.
>>>> And they will be backed by dirty bitmaps. so checkpoint deletion is
>>>> bitmap merge (and delete one of them),
>>>> checkpoint creation is disabling of active-checkpoint-bitmap and
>>>> starting new active-checkpoint-bitmap.
>>> Yes, exactly! I think that's pretty similar to what I am thinking of
>>> with slices.
>>>
>>> This sounds a little safer to me in that we can examine an operation to
>>> see if it's sane or not.
>> Exposing checkpoints is a reasonable high-level API. The important part
>> then is that you don't expose bitmaps + slices, but only checkpoints
>> without bitmaps. The bitmaps are an implementation detail.
>>
>>>> Then we can implement merging of several bitmaps (from one of
>>>> checkpoints to current moment) in
>>>> NBD meta-context-query handling.
>>>>
>>> Note:
>>>
>>> I should say that I've had discussions with Stefan in the past over
>>> things like differential mode and the feeling I got from him was that he
>>> felt that data should be copied from QEMU precisely *once*, viewing any
>>> subsequent copying of the same data as redundant and wasteful.
>> That's a management layer decision. Apparently there are users who want
>> to copy from qemu multiple times, otherwise we wouldn't be talking about
>> slices and retention.
>>
>> Kevin
> 
> You didn't touch storing-to-qcow2 and migration, which we will have to
> implement for
> these bitmap-like objects.
> 
> By exposing low level merge/disable/enable we solve all the problem by
> less code and
> without negotiation the architecture. We avoid:
>  - implementing new job
>  - implementing new objects (checkpoints) and new interface for them
>  - saving these new objects to qcow2
>  - migration of them
> 
> Also, even if we open merge/disable/enable interface, we can implement
> checkpoints or slices
> in future, if we will be _sure_ that we need them.
> 
> Also, I don't see what is unsafe with implementing this simple low-level
> API.
> We already have bitmaps deletion. Is it safe?
> 
> 

Just scrap the whole thought.