Re: [Qemu-devel] While testing `drive-backup` with 'dirty-bitmap' syc mode via QMP. . .

[John Snow]

Kashyap Chamarthy sent me a question asking for some clarifications on 
testing my incremental backup series. I responded off-list but he has 
asked me to reproduce my answer for the archives :)

On 04/21/2015 08:35 AM, Kashyap Chamarthy wrote:
> Hi John,
>
> I would have asked this question on #qemu on OFTC, but there are three
> to four parallel conversations ongoing at the moment. . .
>
> So, I'm testing on a branch with your "v2.5" tree from
>
>      https://github.com/jnsnow/qemu/commits/incremental-transactions
>
>      $ git checkout -b v2.5-and-v6-inc-backup jsnow/incremental-transactions
>      $ git describe
>      v2.3.0-rc3-37-g6932a32
>
> Test
> ----
>
> (1) I have the QEMU invocation w/ QMP of a CirrOS disk image, that way:
>
>    /home/kashyapc/tinker-space/qemu-upstream/x86_64-softmmu/qemu-system-x86_64 \
>    . . .
>    -drive file=/export/full_backup.img,if=none,id=drive-ide0-0-0,format=qcow2 \
>    -qmp tcp:localhost:4444,server
>
> So, it's waiting on the server:
>
>    $ ./2-min-qemu-invoc-qmp.sh
>    char device redirected to /dev/pts/31 (label charserial0)
>    QEMU waiting for connection on: disconnected:tcp:localhost:4444,server
>
>
> (2) Then, when I execute the `drive-backup` QMP command:
> -----------------------------------
> #!/bin/bash
> set -x
> # Test incremental backup with `drive-backup`
>
> exec 3<>/dev/tcp/localhost/4444
> echo -e "{ 'execute': 'qmp_capabilities' }" >&3
> read response <&3
> echo $response
> echo -e "{ 'execute': 'drive-backup', 'arguments':
>        { 'device': 'drive-ide0-0-0', 'bitmap': 'bitmap0', 'sync': 
> 'dirty-bitmap', 'target':
>        
> '/home/kashyapc/work/virt/qemu/incremental-backup-test-qemu/tests/incremental.0.img', 
> 'mode': 'existing', 'format': 'qcow2' } }" >&3
> read response <&3
> echo $response
> -----------------------------------
>
> (3) Invoke it:
>
> $ ./invoke-drive-backup.sh
> {"QMP": {"version": {"qemu": {"micro": 93, "minor": 2, "major": 2}, "package": ""}, 
> "capabilities": []}}
> {"return": {}}
>
>
> Now, I don't see any feedback notification on the QMP server as a result
> of the `drive-bacup` execution from step (3) unlike from a slightly old
> test of regular `drive-backup` command[1].
>
> NOTE: I tested with and without pre-creating the 'incremental.0.img'
> file, to no avail.
>
> Am I missing anything obvious?
>
>
> [1] https://kashyapc.fedorapeople.org/virt/test-qmp-drive-backup.txt

(1) It doesn't appear as though you've created a bitmap in this example?

(2) Incremental backup is designed to work basically exclusively with 
'mode': 'existing' because it relies on the user (or the management
utility/api) to keep track of what the "last backup" was.

(I had actually attempted to add that convenience feature into QEMU, but 
the sense was that the complexities of that were best left to libvirt, 
and I agreed.)

Here's the workflow (if the VM has not begun guest execution)
  (A) Create a bitmap ("bitmap0" is my usual choice)
  (B) Create an "anchor" or "tail" backup:
      This is a full backup of the drive, performed as usual.
  (C) Create the image to be used, using e.g. tail.qcow2 as
      the backing file.
  (D) Ensure that <bitmap0> is clear by issuing a clear command. Look
      at bitmaps.md for an example.
  (E) You may optionally start the VM to allow the disk to change
      at this point.
  (F) You may now issue an incremental backup command:
      { 'execute': 'drive-backup',
        'arguments': {
          'device': 'drive-ide0-0-0',
          'bitmap': 'bitmap0',
          'sync': 'dirty-bitmap',
          'target': .../tail.qcow2,
          'mode': 'existing',
          'format': 'qcow2'
        }
      }

The workflow is different if the VM has already started,
for data safety reasons. (We need extra precautions to ensure
the coherency of the incremental backup.)

  (A) Create a bitmap. ("bitmap0")
  (B) Using a single transaction, you must accomplish the following:
    (i) Create a full backup of 'drive-ide0-0-0'
    (ii) Clear bitmap0.
         This ensures that the bitmap tracks only writes that have
         occurred after the full backup was made. This is why the
         transactions are important.

         It's OK to let QEMU create the tail.qcow2 file here,
         so we don't need mode:existing.

         { 'execute': 'transaction',
           'arguments': {
             'actions': [
               { 'type': 'block-dirty-bitmap-clear',
                 'data': {
                   "node": 'drive-ide0-0-0',
                   "name": 'bitmap0'
                 }
               },
               { 'type': 'drive-backup',
                 'data': {
                   'device': 'drive-ide0-0-0',
                   'bitmap': 'bitmap0',
                   'sync': 'full',
                   'target': .../tail.qcow2,
                   'format': 'qcow2'
                 }
               }
             ]
           }
         }

  (C) We need to create the incremental backup image outside of QEMU;
      # qemu-img create -f qcow2 -F qcow2 -b .../tail.qcow2
  (D) Now that we have a full backup and a synced bitmap, we can
      issue a standard incremental backup command at any time.
      { 'execute': 'drive-backup',
        'arguments': {
          'device': 'drive-ide0-0-0',
          'bitmap': 'bitmap0',
          'sync': 'dirty-bitmap',
          'target': .../tail.qcow2,
          'mode': 'existing',
          'format': 'qcow2'
        }
      }

(3) You are probably not seeing any return from this script because it 
looks as if you only read from the QMP stream once, which will just give 
you the reply from accepting the QMP command, but the event that tells 
you if it completed successfully or not occurs later, so you need one 
more read. If you use bash to send/receive QMP commands, you need to 
poll the fd to get event responses that occur later.

--
—js