[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
[Qemu-devel] hitting intermittent issue with live migration from qemu-kv
From: |
Chris Friesen |
Subject: |
[Qemu-devel] hitting intermittent issue with live migration from qemu-kvm-ev 2.3.0 to qemu-kvm-ev 2.6.0 |
Date: |
Fri, 31 Mar 2017 14:12:36 -0600 |
User-agent: |
Mozilla/5.0 (X11; Linux x86_64; rv:31.0) Gecko/20100101 Thunderbird/31.3.0 |
Hi,
I'm running into an issue with live-migrating a guest from a host running
qemu-kvm-ev 2.3.0-31 to a host running qemu-kvm-ev 2.6.0-27.1. This is a
libvirt-tunnelled migration, in the context of upgrading an OpenStack install to
newer software. The source host is running CentOS 7.2.1511, while the dest host
is running CentOS 7.3.1611.
I'll include the qemu commandlines for the source/dest at the bottom.
Initially we have a bunch of guests running on compute-2 (which is running
qemu-kvm-ev 2.3.0). We then started live-migrating them one at a time to
compute-0 (which is running qemu-kvm-ev 2.6.0). Three of them migrated
successfully. The fourth (which was essentially identical in configuration to
the first three) failed, as per the following logs in
/var/log/libvirt/qemu/instance-0000000e.log:
2017-03-29T06:38:37.886940Z qemu-kvm: VQ 2 size 0x80 < last_avail_idx 0x47b -
used_idx 0x47c
2017-03-29T06:38:37.886974Z qemu-kvm: error while loading state for instance 0x0
of device '0000:00:07.0/virtio-balloon'
2017-03-29T06:38:37.888684Z qemu-kvm: load of migration failed: Operation not
permitted
2017-03-29 06:38:37.896+0000: shutting down
Does anyone know of an existing bug report covering this issue? (I took a look
and didn't see anything obviously related.)
The qemu commandline on the source compute node is:
/usr/libexec/qemu-kvm -c 0x00000000000000000000000000000001 -n 4
--proc-type=secondary --file-prefix=vs -- -enable-dpdk -name instance-0000000e
-S -machine pc-i440fx-rhel7.2.0,accel=kvm,usb=off -m 512 -realtime mlock=off
-smp 1,sockets=1,cores=1,threads=1 -object
memory-backend-file,id=ram-node0,prealloc=yes,mem-path=/mnt/huge-2048kB/libvirt/qemu,share=yes,size=536870912,host-nodes=1,policy=bind
-numa node,nodeid=0,cpus=0,memdev=ram-node0 -uuid
57ae849f-aa66-422a-90a2-62db6c59db29 -smbios type=1,manufacturer=Fedora
Project,product=OpenStack
Nova,version=13.0.0-0.tis.4,serial=4c8121f1-d927-424e-8712-88b1de45be37,uuid=57ae849f-aa66-422a-90a2-62db6c59db29,family=Virtual
Machine -no-user-config -nodefaults -chardev
socket,id=charmonitor,path=/var/lib/libvirt/qemu/domain-instance-0000000e/monitor.sock,server,nowait
-mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc,driftfix=slew
-global kvm-pit.lost_tick_policy=discard -no-hpet -no-shutdown -boot
reboot-timeout=5000,strict=on -device
piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive
file=/dev/disk/by-path/ip-192.168.205.6:3260-iscsi-iqn.2010-10.org.openstack:volume-ac57fcaa-7ecd-4d3b-8671-3bc740337a42-lun-0,if=none,id=drive-virtio-disk0,format=raw,serial=ac57fcaa-7ecd-4d3b-8671-3bc740337a42,cache=none,aio=native
-device
virtio-blk-pci,scsi=off,bus=pci.0,addr=0x6,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1
-chardev
socket,id=charnet0,path=/var/run/vswitch/usvhost-9e574d3c-32dd-4d39-97e6-447b15fb00b4
-netdev type=vhost-user,id=hostnet0,chardev=charnet0 -device
virtio-net-pci,netdev=hostnet0,id=net0,mac=fa:16:3e:b0:59:a9,bus=pci.0,addr=0x3
-chardev
socket,id=charnet1,path=/var/run/vswitch/usvhost-7bc48d91-f215-4394-99ff-eb7f20d9ff1e
-netdev type=vhost-user,id=hostnet1,chardev=charnet1 -device
virtio-net-pci,netdev=hostnet1,id=net1,mac=fa:16:3e:8b:6f:09,bus=pci.0,addr=0x4
-chardev
socket,id=charnet2,path=/var/run/vswitch/usvhost-c32e2d0d-9ed4-4f4b-abc9-539a12a86008
-netdev type=vhost-user,id=hostnet2,chardev=charnet2 -device
virtio-net-pci,netdev=hostnet2,id=net2,mac=fa:16:3e:07:ca:a0,bus=pci.0,addr=0x5
-chardev
file,id=charserial0,path=/etc/nova/instances/57ae849f-aa66-422a-90a2-62db6c59db29/console.log
-device isa-serial,chardev=charserial0,id=serial0 -chardev pty,id=charserial1
-device isa-serial,chardev=charserial1,id=serial1 -device usb-tablet,id=input0
-vnc 0.0.0.0:11 -k en-us -device cirrus-vga,id=video0,bus=pci.0,addr=0x2
-incoming fd:25 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x7 -msg
timestamp=on
The complete instance-0000000e.log file on the destination is:
2017-03-29 06:38:35.962+0000: starting up libvirt version: 2.0.0, package:
10.el7_3.2.tis.24 (Unknown, 2017-03-15-14:59:22, yow-dsulliva-lx-vm1.wrs.com),
qemu version: 2.6.0 (qemu-kvm-ev-2.6.0-27.1.el7.tis.31), hostname: compute-0
LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin
QEMU_AUDIO_DRV=none /usr/libexec/qemu-kvm '-c
0x00000000000000000000000000000001' '-n 4' --proc-type=secondary
--file-prefix=vs -- -enable-dpdk -name guest=instance-0000000e,debug-threads=on
-S -object
secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-10-instance-0000000e/master-key.aes
-machine pc-i440fx-rhel7.2.0,accel=kvm,usb=off -m 512 -realtime mlock=off -smp
1,sockets=1,cores=1,threads=1 -object
memory-backend-file,id=ram-node0,prealloc=yes,mem-path=/mnt/huge-2048kB/libvirt/qemu,share=yes,size=536870912,host-nodes=0,policy=bind
-numa node,nodeid=0,cpus=0,memdev=ram-node0 -uuid
57ae849f-aa66-422a-90a2-62db6c59db29 -smbios 'type=1,manufacturer=Fedora
Project,product=OpenStack
Nova,version=13.0.0-0.tis.4,serial=4c8121f1-d927-424e-8712-88b1de45be37,uuid=57ae849f-aa66-422a-90a2-62db6c59db29,family=Virtual
Machine' -no-user-config -nodefaults -chardev
socket,id=charmonitor,path=/var/lib/libvirt/qemu/domain-10-instance-0000000e/monitor.sock,server,nowait
-mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc,driftfix=slew
-global kvm-pit.lost_tick_policy=discard -no-hpet -no-shutdown -boot
reboot-timeout=5000,strict=on -device
piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive
file=/dev/disk/by-path/ip-192.168.205.6:3260-iscsi-iqn.2010-10.org.openstack:volume-ac57fcaa-7ecd-4d3b-8671-3bc740337a42-lun-0,format=raw,if=none,id=drive-virtio-disk0,serial=ac57fcaa-7ecd-4d3b-8671-3bc740337a42,cache=none,aio=native
-device
virtio-blk-pci,scsi=off,bus=pci.0,addr=0x6,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1
-chardev
socket,id=charnet0,path=/var/run/vswitch/usvhost-9e574d3c-32dd-4d39-97e6-447b15fb00b4
-netdev type=vhost-user,id=hostnet0,chardev=charnet0 -device
virtio-net-pci,netdev=hostnet0,id=net0,mac=fa:16:3e:b0:59:a9,bus=pci.0,addr=0x3
-chardev
socket,id=charnet1,path=/var/run/vswitch/usvhost-7bc48d91-f215-4394-99ff-eb7f20d9ff1e
-netdev type=vhost-user,id=hostnet1,chardev=charnet1 -device
virtio-net-pci,netdev=hostnet1,id=net1,mac=fa:16:3e:8b:6f:09,bus=pci.0,addr=0x4
-chardev
socket,id=charnet2,path=/var/run/vswitch/usvhost-c32e2d0d-9ed4-4f4b-abc9-539a12a86008
-netdev type=vhost-user,id=hostnet2,chardev=charnet2 -device
virtio-net-pci,netdev=hostnet2,id=net2,mac=fa:16:3e:07:ca:a0,bus=pci.0,addr=0x5
-add-fd set=0,fd=51 -chardev file,id=charserial0,path=/dev/fdset/0,append=on
-device isa-serial,chardev=charserial0,id=serial0 -chardev pty,id=charserial1
-device isa-serial,chardev=charserial1,id=serial1 -device usb-tablet,id=input0
-vnc 0.0.0.0:9 -k en-us -device cirrus-vga,id=video0,bus=pci.0,addr=0x2
-incoming defer -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x7 -msg
timestamp=on
Domain id=10 is tainted: high-privileges
EAL:eal_memory.c:1591: WARNING: Address Space Layout Randomization (ASLR) is
enabled in the kernel.
EAL:eal_memory.c:1593: This may cause issues with mapping memory into
secondary processes
char device redirected to /dev/pts/9 (label charserial1)
2017-03-29T06:38:37.886940Z qemu-kvm: VQ 2 size 0x80 < last_avail_idx 0x47b -
used_idx 0x47c
2017-03-29T06:38:37.886974Z qemu-kvm: error while loading state for instance 0x0
of device '0000:00:07.0/virtio-balloon'
2017-03-29T06:38:37.888684Z qemu-kvm: load of migration failed: Operation not
permitted
2017-03-29 06:38:37.896+0000: shutting down
For what it's worth, the differences between the two qemu command lines are as
follows:
source:
-name instance-0000000e -chardev
file,id=charserial0,path=/etc/nova/instances/57ae849f-aa66-422a-90a2-62db6c59db29/console.log
-vnc 0.0.0.0:9 -incoming fd:25
destination:
-name guest=instance-0000000e,debug-threads=on -object
secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-10-instance-0000000e/master-key.aes
-add-fd set=0,fd=51 -chardev file,id=charserial0,path=/dev/fdset/0,append=on
-vnc 0.0.0.0:11 -incoming defer
Thanks,
Chris
[Prev in Thread] |
Current Thread |
[Next in Thread] |
- [Qemu-devel] hitting intermittent issue with live migration from qemu-kvm-ev 2.3.0 to qemu-kvm-ev 2.6.0,
Chris Friesen <=