Re: [Qemu-devel] Incremental drive-backup with dirty bitmaps

[Suman Swaroop]

  Hey, some continuation questions from above discussion,


   1.

   Comments in blockdev-add command section in patch
   https://patchwork.kernel.org/patch/9638133/ says that
   “Note: This command is still a work in progress. It doesn't support all
   block drivers among other things. Stay away from it unless you want to help
   with its development.” Does it work with qcow2 and raw from version 2.3?
   2.

   When new nodes are added in chain, there is an associated backing image
   file as well. Some of these image files become redundant when a node is
   merged with other nodes. While issuing qmp commands to qemu via libvirt tls
   socket, there does not seem to be any functionality available to delete the
   redundant image files. Basically when a node is streamed or committed, the
   node and its backing image file can be deleted as they are not required
   anymore. Is there any qmp command to achieve the same with the constraint
   that we only have access to libvirt tls socket and cannot ssh to the host?
   3.

   In filename path of blockdev-add command or target path of drive-backup
   command can we provide nfs urld directly for the image path i.e
   nfs://<ip>/foo(not a mounted directory)? This is related to the point 2
   above that we do not have root ssh access to the host but only access to
   its tls socket


Suman

On Thu, Jan 24, 2019 at 6:12 PM Bharadwaj Rayala <
address@hidden> wrote:

> Hi Eric, Kashyap,
>
> I work for Rubrik. I am trying to implement first class support for
> protection of rhev instances using rubrik. I want the solution to be
> generic and be a step towards protection of any kvm instance with any
> management layer(standalone kvm/rhel host, rhev/ovirt clusters, openstack
> clusters ...). But first l am just concentrating on the rhev flow. This
> backup program would be a proprietary product.
>
> I looked at the rhev recommended way for backups.
> https://access.redhat.com/documentation/en-us/red_hat_virtualization/4.2/html/administration_guide/sect-backing_up_and_restoring_virtual_machines_using_the_backup_and_restore_api
> This doesnot make use of qemu block jobs for incremental ingest, or dirty
> bitmaps. Entire responsibility is on the backup program to take backups,
> and the best one can do is to do a block by block fingerprint on the whole
> drive and then do an incremental ingest by comparing fp with fp's of the
> base snapshot.
>
> I wanted to build something that directly talks to rhev(or any kvm) hosts
> using libvirt and use block jobs and dirty bitmaps to do cbt like, but push
> based ingest. We need to support 4.1 version of rhev, which contains
> qmeu-kvm-rhev 2.6.0-28.el7_3.9 which has (almost) all the functionality
> that we need. I know that write access through libvirt is not supported by
> rhev right now [1]
> <https://access.redhat.com/documentation/en-us/red_hat_virtualization/4.2/html/product_guide/accessing-rhv>.
> I am trying to find a non invasive way of accessing qemu drive-backup block
> job through libvirt(using qemu-monitor-command, libvirt2.0 in rhev4.1
> doesnot have drive-backup) that does not interfere with user driven
> snapshots/interactions from rhev ui/api. I am not sure if I can continue on
> this path though, with the information Eric has provided on user driven
> snapshots. Please do let me know if you have any comments/better ideas.
>
> Currently i am talking to libvirt hosts over tls socket (virsh -c
> qemu+tls://rhevnode1/system ), by certifying rubrik nodes from which backup
> is triggered using the same CA.key manager uses to certify its own nodes. I
> already read your wikis on incremental live backups and dirty bitmaps. They
> are a great find. TY
>
> Bharadwaj.
>
> On Thu, Jan 24, 2019 at 2:46 PM Kashyap Chamarthy <address@hidden>
> wrote:
>
>> On Wed, Jan 23, 2019 at 01:09:41PM -0600, Eric Blake wrote:
>> > On 1/23/19 12:08 PM, Bharadwaj Rayala wrote:
>>
>> [...] # [Snip Eric's excellent exposition.]
>>
>> > > What do you mean by issues? Do you mean any data/corruption bugs or
>> lack of
>> > > some nice functionality that we are talking here?
>> >
>> > Lack of functionality.  In particular, the 4.0 commands
>> > block-dirty-bitmap-{enable,merge,disable} (or their 3.1 counterparts
>> > x-block-dirty-bitmap-*) are essential to the workflow of differential
>> > backups (without being able to manage bitmaps yourself, you can only get
>> > the weaker incremental backup, and that means qemu itself is clearing
>> > the bitmap out of under your feet on success, and where you are having
>> > to worry about completion-mode=grouped).
>> >
>> > >
>> > > Thanks a lot Eric for spending your time in answering my queries. I
>> dont
>> > > know if you work with Kashyap Chamarthy, but your help and his blogs
>> are
>> > > lifesavers.
>> >
>> > Yes, Kashyap is trying to build solutions on top of the building blocks
>> > that I am working on, so we have collaborated several times on these
>> > types of issues (he does a lot better at blog posts extracted from my
>> > mailing list brain dumps).
>>
>> I haven't kept up with incremental backups lately, as I've been swamped
>> with other work.  But two other documents that I can point to are these
>> [1][2] in the QEMU tree.  And their HTML-rendered versions are
>> here[3][4].  They're generated for 3.0.0; but these docs haven't changed
>> much since then.
>>
>> Along with Eric's last year talk, also check out presentations from
>> previous KVM Forums from other Block Layer maintainers.
>>
>>
>> [1]
>> https://git.qemu.org/?p=qemu.git;a=blob;f=docs/interop/live-block-operations.rst
>> [2] https://git.qemu.org/?p=qemu.git;a=blob;f=docs/interop/bitmaps.rst
>> [3]
>> https://kashyapc.fedorapeople.org/QEMU-Docs-v3.0.0/_build/html/docs/interop/live-block-operations.html
>> [4]
>> https://kashyapc.fedorapeople.org/QEMU-Docs-v3.0.0/_build/html/docs/interop/bitmaps.html
>>
>>
>> --
>> /kashyap
>>
>
>
> --
> [image: photo]
> Bharadwaj Rayala
> Software Engineer at Rubrik
> M  +919618462233  <+919618462233> E  address@hidden
> <address@hidden> W  www.rubrik.com
> <http://www.rubrik.com?utm_source=WiseStamp&utm_medium=email&utm_term=&utm_content=&utm_campaign=signature>
>
>