"guest-reset" and "invalid runstate transition" in COLO SVM

[Jing-Wei Su]

Hello,

I'm testing  COLO in qemu-4.2.0 with the commit
https://github.com/qemu/qemu/commit/f51d0b4178738bba87d796eba7444f6cdb3aa0fd.

The qmp of SVM sometimes show the following errors ("guest-reset"
or/and "invalid runstate transition") .
Does any have idea about this?

{"timestamp": {"seconds": 1584435907, "microseconds": 610964},
"event": "RESUME"}
{"timestamp": {"seconds": 1584435927, "microseconds": 553683}, "event": "STOP"}
{"timestamp": {"seconds": 1584435980, "microseconds": 533344},
"event": "RESUME"}
{"timestamp": {"seconds": 1584435980, "microseconds": 579256},
"event": "RESET", "data": {"guest": true, "reason": "guest-reset"}}
{"timestamp": {"seconds": 1584435980, "microseconds": 588350}, "event": "STOP"}
{"timestamp": {"seconds": 1584435980, "microseconds": 801483},
"event": "RESUME"}
{"timestamp": {"seconds": 1584435980, "microseconds": 802061}, "event": "STOP"}
{"timestamp": {"seconds": 1584435980, "microseconds": 803988},
"event": "RESET", "data": {"guest": true, "reason": "guest-reset"}}
qemu-system-x86_64: invalid runstate transition: 'colo' -> 'prelaunch'
secondary-nonshared.sh: line 25: 23457 Aborted                 (core
dumped) qemu-system-x86_64 -name secondary -enable-kvm -cpu
qemu64,+kvmclock -m 2048 -global kvm-apic.vapic=false -netdev
tap,id=hn0,vhost=off,helper=/usr/local/libexec/qemu-bridge-helper
-device e1000,id=e0,netdev=hn0 -chardev
socket,id=red0,host=$primary_ip,port=9003,reconnect=1 -chardev
socket,id=red1,host=$primary_ip,port=9004,reconnect=1 -object
filter-redirector,id=f1,netdev=hn0,queue=tx,indev=red0 -object
filter-redirector,id=f2,netdev=hn0,queue=rx,outdev=red1 -object
filter-rewriter,id=rew0,netdev=hn0,queue=all -drive
if=none,id=parent0,file.filename=$imagefolder/secondary.qcow2,driver=qcow2
-drive 
if=none,id=childs0,driver=replication,mode=secondary,file.driver=qcow2,top-id=colo-disk0,file.file.filename=$imagefolder/secondary-active.qcow2,file.backing.driver=qcow2,file.backing.file.filename=$imagefolder/secondary-hidden.qcow2,file.backing.backing=parent0
-drive 
if=ide,id=colo-disk0,driver=quorum,read-pattern=fifo,vote-threshold=1,children.0=childs0
-qmp unix:/tmp/qmp-svm-sock,server,nowait -qmp stdio -vnc :5 -incoming
tcp:0.0.0.0:9998

My PVM and SVM are on the same PC.
Here are the steps to setup my testing
(1) Start PVM
qemu-system-x86_64 -name primary -enable-kvm -cpu qemu64,+kvmclock -m
2048 -global kvm-apic.vapic=false \
-netdev tap,id=hn0,vhost=off,helper=/usr/local/libexec/qemu-bridge-helper \
-device e1000,id=e0,netdev=hn0 \
-drive 
if=ide,id=colo-disk0,driver=quorum,read-pattern=fifo,vote-threshold=1,children.0.file.filename=$imagefolder/primary.qcow2,children.0.driver=qcow2
\
-qmp stdio -vnc :4

(2) Add chardevs to PVM via qmp
{'execute': 'qmp_capabilities'}
{'execute': 'chardev-add', 'arguments':{ 'id': 'mirror0', 'backend':
{'type': 'socket', 'data': {'addr': { 'type': 'inet', 'data': {
'host': '0.0.0.0', 'port': '9003' } }, 'server': true } } } }
{'execute': 'chardev-add', 'arguments':{ 'id': 'compare1', 'backend':
{'type': 'socket', 'data': {'addr': { 'type': 'inet', 'data': {
'host': '0.0.0.0', 'port': '9004' } }, 'server': true } } } }
{'execute': 'chardev-add', 'arguments':{ 'id': 'compare0', 'backend':
{'type': 'socket', 'data': {'addr': { 'type': 'inet', 'data': {
'host': '127.0.0.1', 'port': '9001' } }, 'server': true } } } }
{'execute': 'chardev-add', 'arguments':{ 'id': 'compare0-0',
'backend': {'type': 'socket', 'data': {'addr': { 'type': 'inet',
'data': { 'host': '127.0.0.1', 'port': '9001' } }, 'server': false } }
} }
{'execute': 'chardev-add', 'arguments':{ 'id': 'compare_out',
'backend': {'type': 'socket', 'data': {'addr': { 'type': 'inet',
'data': { 'host': '127.0.0.1', 'port': '9005' } }, 'server': true } }
} }
{'execute': 'chardev-add', 'arguments':{ 'id': 'compare_out0',
'backend': {'type': 'socket', 'data': {'addr': { 'type': 'inet',
'data': { 'host': '127.0.0.1', 'port': '9005' } }, 'server': false } }
} }

(3) Start SVM
primary_ip=127.0.0.1
qemu-system-x86_64 -name secondary -enable-kvm -cpu qemu64,+kvmclock
-m 2048 -global kvm-apic.vapic=false \
-netdev tap,id=hn0,vhost=off,helper=/usr/local/libexec/qemu-bridge-helper \
-device e1000,id=e0,netdev=hn0 \
-chardev socket,id=red0,host=$primary_ip,port=9003,reconnect=1 \
-chardev socket,id=red1,host=$primary_ip,port=9004,reconnect=1 \
-object filter-redirector,id=f1,netdev=hn0,queue=tx,indev=red0 \
-object filter-redirector,id=f2,netdev=hn0,queue=rx,outdev=red1 \
-object filter-rewriter,id=rew0,netdev=hn0,queue=all \
-drive 
if=none,id=parent0,file.filename=$imagefolder/secondary.qcow2,driver=qcow2
\
-drive 
if=none,id=childs0,driver=replication,mode=secondary,file.driver=qcow2,top-id=colo-disk0,file.file.filename=$imagefolder/secondary-active.qcow2,file.backing.driver=qcow2,file.backing.file.filename=$imagefolder/secondary-hidden.qcow2,file.backing.backing=parent0
\
-drive 
if=ide,id=colo-disk0,driver=quorum,read-pattern=fifo,vote-threshold=1,children.0=childs0
\
-qmp stdio -vnc :5 -incoming tcp:0.0.0.0:9998

(4) Start NBD server of SVM
{'execute':'qmp_capabilities'}
{'execute': 'nbd-server-start', 'arguments': {'addr': {'type': 'inet',
'data': {'host': '0.0.0.0', 'port': '9999'} } } }
{'execute': 'nbd-server-add', 'arguments': {'device': 'parent0',
'writable': true } }

(5) Invoke drive-mirror in PVM side
{'execute': 'drive-mirror', 'arguments':{ 'device': 'colo-disk0',
'job-id': 'resync', 'target': 'nbd://127.0.0.2:9999/parent0', 'mode':
'existing', 'format': 'raw', 'sync': 'full'} }

Wait until disk is synced, then:
{'execute': 'stop'}
{'execute': 'block-job-cancel', 'arguments':{ 'device': 'resync'} }

(6) Add Filters and Start COLO Migrate
{'execute': 'human-monitor-command', 'arguments':{ 'command-line':
'drive_add -n buddy
driver=replication,mode=primary,file.driver=nbd,file.host=127.0.0.1,file.port=9999,file.export=parent0,node-name=replication0'}}
{'execute': 'x-blockdev-change', 'arguments':{ 'parent': 'colo-disk0',
'node': 'replication0' } }
{'execute': 'object-add', 'arguments':{ 'qom-type': 'filter-mirror',
'id': 'm0', 'props': { 'netdev': 'hn0', 'queue': 'tx', 'outdev':
'mirror0' } } }
{'execute': 'object-add', 'arguments':{ 'qom-type':
'filter-redirector', 'id': 'redire0', 'props': { 'netdev': 'hn0',
'queue': 'rx', 'indev': 'compare_out' } } }
{'execute': 'object-add', 'arguments':{ 'qom-type':
'filter-redirector', 'id': 'redire1', 'props': { 'netdev': 'hn0',
'queue': 'rx', 'outdev': 'compare0' } } }
{'execute': 'object-add', 'arguments':{ 'qom-type': 'iothread', 'id':
'iothread1' } }
{'execute': 'object-add', 'arguments':{ 'qom-type': 'colo-compare',
'id': 'comp0', 'props': { 'primary_in': 'compare0-0', 'secondary_in':
'compare1', 'outdev': 'compare_out0', 'iothread': 'iothread1' } } }
{'execute': 'migrate-set-capabilities', 'arguments':{ 'capabilities':
[ {'capability': 'x-colo', 'state': true } ] } }
{'execute': 'migrate', 'arguments':{ 'uri': 'tcp:127.0.0.1:9998' } }

Thanks!
Sincerely,
Jing-Wei