Re: [Qemu-devel] [PATCH v13 00/10] Block replication for continuous checkpoints

[Wen Congyang]

On 01/27/2016 07:03 PM, Dr. David Alan Gilbert wrote:
> Hi,
>   I've got a block error if I kill the secondary.
> 
> Start both primary & secondary
> kill -9 secondary qemu
> x_colo_lost_heartbeat on primary
> 
> The guest sees a block error and the ext4 root switches to read-only.
> 
> I gdb'd the primary with a breakpoint on quorum_report_bad; see
> backtrace below.
> (This is based on colo-v2.4-periodic-mode of the framework
> code with the block and network proxy merged in; so it could be my
> merging but I don't think so ?)
> 
> 
> (gdb) where
> #0  quorum_report_bad (node_name=0x7f2946a0892c "node0", ret=-5, 
> acb=0x7f2946cb3910, acb=0x7f2946cb3910)
>     at /root/colo/jan-2016/qemu/block/quorum.c:222
> #1  0x00007f2943b23058 in quorum_aio_cb (opaque=<optimized out>, 
> ret=<optimized out>)
>     at /root/colo/jan-2016/qemu/block/quorum.c:315
> #2  0x00007f2943b311be in bdrv_co_complete (acb=0x7f2946cb3f60) at 
> /root/colo/jan-2016/qemu/block/io.c:2122
> #3  0x00007f2943ae777d in aio_bh_call (bh=<optimized out>) at 
> /root/colo/jan-2016/qemu/async.c:64
> #4  aio_bh_poll (address@hidden) at /root/colo/jan-2016/qemu/async.c:92
> #5  0x00007f2943af5090 in aio_dispatch (ctx=0x7f2945b771d0) at 
> /root/colo/jan-2016/qemu/aio-posix.c:305
> #6  0x00007f2943ae756e in aio_ctx_dispatch (source=<optimized out>, 
> callback=<optimized out>, 
>     user_data=<optimized out>) at /root/colo/jan-2016/qemu/async.c:231
> #7  0x00007f293b84a79a in g_main_context_dispatch () from 
> /lib64/libglib-2.0.so.0
> #8  0x00007f2943af3a00 in glib_pollfds_poll () at 
> /root/colo/jan-2016/qemu/main-loop.c:211
> #9  os_host_main_loop_wait (timeout=<optimized out>) at 
> /root/colo/jan-2016/qemu/main-loop.c:256
> #10 main_loop_wait (nonblocking=<optimized out>) at 
> /root/colo/jan-2016/qemu/main-loop.c:504
> #11 0x00007f29438529ee in main_loop () at /root/colo/jan-2016/qemu/vl.c:1945
> #12 main (argc=<optimized out>, argv=<optimized out>, envp=<optimized out>) 
> at /root/colo/jan-2016/qemu/vl.c:4707
> 
> (gdb) p s->num_children
> $1 = 2
> (gdb) p acb->success_count
> $2 = 0
> (gdb) p acb->is_read
> $5 = false

Sorry for the late reply.
What it the value of acb->count?

If secondary host is down, you should remove quorum's children.1. Otherwise, 
you will get
I/O error event.

Thanks
Wen Congyang

> 
> (qemu) info block
> colo-disk0 (#block080): json:{"children": [{"driver": "raw", "file": 
> {"driver": "file", "filename": "/root/colo/bugzilla.raw"}}, {"driver": 
> "replication", "mode": "primary", "file": {"port": "8889", "host": "ibpair", 
> "driver": "nbd", "export": "colo-disk0"}}], "driver": "quorum", "blkverify": 
> false, "rewrite-corrupted": false, "vote-threshold": 1} (quorum)
>     Cache mode:       writeback, direct
> 
> Dave
> 
> * Changlong Xie (address@hidden) wrote:
>> Block replication is a very important feature which is used for
>> continuous checkpoints(for example: COLO).
>>
>> You can get the detailed information about block replication from here:
>> http://wiki.qemu.org/Features/BlockReplication
>>
>> Usage:
>> Please refer to docs/block-replication.txt
>>
>> This patch series is based on the following patch series:
>> 1. http://lists.nongnu.org/archive/html/qemu-devel/2015-12/msg04570.html
>>
>> You can get the patch here:
>> https://github.com/Pating/qemu/tree/changlox/block-replication-v13
>>
>> You can get the patch with framework here:
>> https://github.com/Pating/qemu/tree/changlox/colo_framework_v12
>>
>> TODO:
>> 1. Continuous block replication. It will be started after basic functions
>>    are accepted.
>>
>> Changs Log:
>> V13:
>> 1. Rebase to the newest codes
>> 2. Remove redundant marcos and semicolon in replication.c 
>> 3. Fix typos in block-replication.txt
>> V12:
>> 1. Rebase to the newest codes
>> 2. Use backing reference to replcace 'allow-write-backing-file'
>> V11:
>> 1. Reopen the backing file when starting blcok replication if it is not
>>    opened in R/W mode
>> 2. Unblock BLOCK_OP_TYPE_BACKUP_SOURCE and BLOCK_OP_TYPE_BACKUP_TARGET
>>    when opening backing file
>> 3. Block the top BDS so there is only one block job for the top BDS and
>>    its backing chain.
>> V10:
>> 1. Use blockdev-remove-medium and blockdev-insert-medium to replace backing
>>    reference.
>> 2. Address the comments from Eric Blake
>> V9:
>> 1. Update the error messages
>> 2. Rebase to the newest qemu
>> 3. Split child add/delete support. These patches are sent in another 
>> patchset.
>> V8:
>> 1. Address Alberto Garcia's comments
>> V7:
>> 1. Implement adding/removing quorum child. Remove the option non-connect.
>> 2. Simplify the backing refrence option according to Stefan Hajnoczi's 
>> suggestion
>> V6:
>> 1. Rebase to the newest qemu.
>> V5:
>> 1. Address the comments from Gong Lei
>> 2. Speed the failover up. The secondary vm can take over very quickly even
>>    if there are too many I/O requests.
>> V4:
>> 1. Introduce a new driver replication to avoid touch nbd and qcow2.
>> V3:
>> 1: use error_setg() instead of error_set()
>> 2. Add a new block job API
>> 3. Active disk, hidden disk and nbd target uses the same AioContext
>> 4. Add a testcase to test new hbitmap API
>> V2:
>> 1. Redesign the secondary qemu(use image-fleecing)
>> 2. Use Error objects to return error message
>> 3. Address the comments from Max Reitz and Eric Blake
>>
>> Wen Congyang (10):
>>   unblock backup operations in backing file
>>   Store parent BDS in BdrvChild
>>   Backup: clear all bitmap when doing block checkpoint
>>   Allow creating backup jobs when opening BDS
>>   docs: block replication's description
>>   Add new block driver interfaces to control block replication
>>   quorum: implement block driver interfaces for block replication
>>   Implement new driver for block replication
>>   support replication driver in blockdev-add
>>   Add a new API to start/stop replication, do checkpoint to all BDSes
>>
>>  block.c                    | 145 ++++++++++++
>>  block/Makefile.objs        |   3 +-
>>  block/backup.c             |  14 ++
>>  block/quorum.c             |  78 +++++++
>>  block/replication.c        | 545 
>> +++++++++++++++++++++++++++++++++++++++++++++
>>  blockjob.c                 |  11 +
>>  docs/block-replication.txt | 227 +++++++++++++++++++
>>  include/block/block.h      |   9 +
>>  include/block/block_int.h  |  15 ++
>>  include/block/blockjob.h   |  12 +
>>  qapi/block-core.json       |  33 ++-
>>  11 files changed, 1089 insertions(+), 3 deletions(-)
>>  create mode 100644 block/replication.c
>>  create mode 100644 docs/block-replication.txt
>>
>> -- 
>> 1.9.3
>>
>>
>>
> --
> Dr. David Alan Gilbert / address@hidden / Manchester, UK
> 
> 
> .
>