Re: [PATCH v5 2/5] qcow2: Expose bitmaps' size during measure

[Nir Soffer]

On Thu, May 21, 2020 at 6:09 PM Vladimir Sementsov-Ogievskiy
<address@hidden> wrote:
>
> 21.05.2020 16:29, Nir Soffer wrote:
> > On Thu, May 21, 2020 at 2:40 PM Vladimir Sementsov-Ogievskiy
> > <address@hidden> wrote:
> >>
> >> 21.05.2020 02:00, Nir Soffer wrote:
> >>> On Thu, May 21, 2020 at 1:01 AM Eric Blake <address@hidden> wrote:
> >>>>
> >>>> It's useful to know how much space can be occupied by qcow2 persistent
> >>>> bitmaps, even though such metadata is unrelated to the guest-visible
> >>>> data.  Report this value as an additional QMP field, present when
> >>>> measuring an existing image and output format that both support
> >>>> bitmaps.  Update iotest 178 and 190 to updated output, as well as new
> >>>> coverage in 190 demonstrating non-zero values made possible with the
> >>>> recently-added qemu-img bitmap command.
> >>>>
> >>>> On the command-line side, 'qemu-img measure' gains a new --bitmaps
> >>>> flag.  When present, the bitmap size is rolled into the two existing
> >>>> measures (or errors if either the source image or destination format
> >>>> lacks bitmaps); when absent, there is never an error (for
> >>>> back-compat), but the output will instead include a new line item for
> >>>> bitmaps (which you would have to manually add), with that line being
> >>>> omitted in the same cases where passing --bitmaps would error.
> >>>
> >>> Supporting 2 ways to measure, one by specifying --bitmaps, and another
> >>> by adding bitmaps key is not a good idea. We really need one way.
> >>>
> >>> Each one has advantages. adding --bitmaps flag is consistent with
> >>> "qemu-img convert"
> >>> and future extensions that may require  new flag, and adding "bitmaps"
> >>> key is consistent
> >>> with "qmeu-img info", showing bitmaps when they exist.
> >>>
> >>> Adding a "bitmaps" key has an advantage that we can use it to test if 
> >>> qemu-img
> >>> supports measuring and copying bitmaps (since both features are expected 
> >>> to
> >>> be delivered at the same time). So we can avoid checking --help learn 
> >>> about
> >>> the capabilities.
> >>>
> >>> I'm ok with both options, can we have only one?
> >>
> >> Hi! What are your scenarios? Are you using qemu-img by hand, or it is used 
> >> from some management tool? For management tool, I'd recommend to use qmp 
> >> interface, which is a lot more strict, reliable and stable, and 
> >> documented. You just need to run qemu binary in stopped mode for it.
> >
> > The use case is oVirt - it is a management system but it uses qemu-img
> > to perform various
> > operations, like copying disks around. The specific use case is
> > cloning qcow2 image chain
> > from one server to another, or cloning on the same server.
> >
> > In the case of qcow2 images on logical volumes, we need to create a
> > large enough  logical
> > volume to copy an image, and for this we use "qemu-img measure". With
> > the current patches
> > we will be able to create large enough logical volume and copy the
> > image data and the bitmps
> > to the destination.
> >
> > qmp interface is nice but to use it we have to rewrite all the code
> > using qemu-img to start qemu
> > with the relevant disks and perform operation via qmp or via libvirt.
> > This a huge rewrite and I'm
> > not sure it's worth the effort.
>
> OK, I'm not familiar with oVirt.. But maybe, you don't need to rewrite 
> everything, but just add a possibility to use qmp interface, so, keep old 
> features working on qemu-img, while adding new features using qmp?

This is always the case, we always need to support old code for 2
versions when we add
new code. This is even worse. It makes sense only if this enables
stuff which is not possible
with qemu-img.

One case that we did not handle yet is copying bitmaps down after
"qemu-img commit",
which should be solved now in libvirt when using blockCommit(). If use
libvirt also for
cold-commit by running qemu paused, it makes sense to add this.

Another case we did not handle yet is backing up non-running vms,
which will also work
transparently if we run the vm paused just for the backup. So using
paused vm via
libvirt seems like a good direction.

> > This approach also has limitations, for example using qemu-img we can
> > copy disks in parallel on
> > multiple hosts, while using libvirt and qemu will limit us to using
> > one host since qemu takes locks
> > that we control.
>
> hmm.. do you open same image RW in several qemu-img processes? Or I don't 
> follow, what behavior is blocked by Qemu locks... qemu-img takes locks as 
> well..

We don't copy images in parallel yet. Currently we just copy the image
one by one,
starting at the base image in the chain. If we start to copy in
parallel, we will run multiple
qemu-img processes, each copying one layer in the chain. It should be
possible since
only the destination image is writable,  and qemu does not need to
read the backing file
not on the source or destination. I think we tried this but maybe it
was before qemu-img
added locking.

> > Also using libvirt means that all new features take more time to
> > complete since now we have
> > a new layer between oVirt and qemu.
>
> May be, using qmp directly is still a good option.

We use qmp for testing our nbd client, it is awesome. This is another option
we need to consider.