Re: [Qemu-devel] HEAD is failing virt-test on migration tests

[Lucas Meneghel Rodrigues]

From what the log says, after a round of migrations 'info migrate' does not respond after 4 minutes, timing out. Virt Test then shuts down the VM. When it tries to check the qcow2 image, it is corrupted. I'm checking out the latest master to see how reproducible this problem is.

On Thu, Feb 12, 2015 at 8:12 PM, Juan Quintela <address@hidden> wrote:

Hi

while testing my changes I noticed that virt-test was failing.  I
check-out master, and failures are there.

This is one extract of the log after the 1st failure.  Notice that it
fails randomly, not every time.

I have to go to bed right now, so if anybody beats me with a fix, I
would be happy when I wakeup.

Thanks, Juan.


22:54:07 DEBUG| (monitor hmp1) Response to 'info migrate'
22:54:07 DEBUG| (monitor hmp1)    capabilities: xbzrle: off rdma-pin-all: off auto-converge: off zero-blocks: off
22:54:07 DEBUG| (monitor hmp1)    Migration status: active
22:54:07 DEBUG| (monitor hmp1)    total time: 2003 milliseconds
22:54:07 DEBUG| (monitor hmp1)    expected downtime: 300 milliseconds
22:54:07 DEBUG| (monitor hmp1)    setup: 3 milliseconds
22:54:07 DEBUG| (monitor hmp1)    transferred ram: 67619 kbytes
22:54:07 DEBUG| (monitor hmp1)    throughput: 268.61 mbps
22:54:07 DEBUG| (monitor hmp1)    remaining ram: 103056 kbytes
22:54:07 DEBUG| (monitor hmp1)    total ram: 1065796 kbytes
22:54:07 DEBUG| (monitor hmp1)    duplicate: 224304 pages
22:54:07 DEBUG| (monitor hmp1)    skipped: 0 pages
22:54:07 DEBUG| (monitor hmp1)    normal: 16380 pages
22:54:07 DEBUG| (monitor hmp1)    normal bytes: 65520 kbytes
22:54:07 DEBUG| (monitor hmp1)    dirty sync count: 0
22:54:09 DEBUG| Waiting for migration to complete (4.006475 secs)
22:54:09 DEBUG| (monitor hmp1) Sending command 'info migrate'
22:54:09 DEBUG| Send command: info migrate
22:54:09 DEBUG| (monitor hmp1) Response to 'info migrate'
22:54:09 DEBUG| (monitor hmp1)    capabilities: xbzrle: off rdma-pin-all: off auto-converge: off zero-blocks: off
22:54:09 DEBUG| (monitor hmp1)    Migration status: active
22:54:09 DEBUG| (monitor hmp1)    total time: 4008 milliseconds
22:54:09 DEBUG| (monitor hmp1)    expected downtime: 300 milliseconds
22:54:09 DEBUG| (monitor hmp1)    setup: 3 milliseconds
22:54:09 DEBUG| (monitor hmp1)    transferred ram: 131397 kbytes
22:54:09 DEBUG| (monitor hmp1)    throughput: 268.57 mbps
22:54:09 DEBUG| (monitor hmp1)    remaining ram: 31392 kbytes
22:54:09 DEBUG| (monitor hmp1)    total ram: 1065796 kbytes
22:54:09 DEBUG| (monitor hmp1)    duplicate: 226311 pages
22:54:09 DEBUG| (monitor hmp1)    skipped: 0 pages
22:54:09 DEBUG| (monitor hmp1)    normal: 32289 pages
22:54:09 DEBUG| (monitor hmp1)    normal bytes: 129156 kbytes
22:54:09 DEBUG| (monitor hmp1)    dirty sync count: 0
22:54:11 DEBUG| Waiting for migration to complete (6.011556 secs)
22:54:11 DEBUG| (monitor hmp1) Sending command 'info migrate'
22:54:11 DEBUG| Send command: info migrate
22:54:32 WARNI| virt-tests-vm1 is not alive. Can not query the register status
22:58:11 DEBUG| Destroying VM virt-tests-vm1 (PID 10880)
22:58:11 DEBUG| Ending VM virt-tests-vm1 process (monitor)
22:58:11 INFO | [qemu output] (Process terminated with status 0)
22:58:11 DEBUG| VM virt-tests-vm1 down (monitor)
22:58:11 DEBUG| Host does not support OpenVSwitch: Missing command: ovs-vswitchd
22:58:11 DEBUG| Destroying VM virt-tests-vm1 (PID 10763)
22:58:11 DEBUG| Shutting down VM virt-tests-vm1 (shell)
22:58:11 DEBUG| Login command: 'ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -o PreferredAuthentications=password -p 5000 address@hidden'
22:58:11 DEBUG| virt-tests-vm1 alive now. Used to failed to get register info from guest 9 times
22:58:13 INFO | [qemu output] (Process terminated with status 0)
22:58:13 DEBUG| VM virt-tests-vm1 down (shell)
22:58:14 DEBUG| Host does not support OpenVSwitch: Missing command: ovs-vswitchd
22:58:14 DEBUG| Checking image file /mnt/kvm/src/virt-test/shared/data/images/jeos-20-64.qcow2
22:58:14 DEBUG| Running '/bin/qemu-img info /mnt/kvm/src/virt-test/shared/data/images/jeos-20-64.qcow2'
22:58:14 DEBUG| Running '/bin/qemu-img check /mnt/kvm/src/virt-test/shared/data/images/jeos-20-64.qcow2'
22:58:14 ERROR| [stdout]
22:58:14 ERROR| [stdout] 1 errors were found on the image.
22:58:14 ERROR| [stdout] Data may be corrupted, or further writes to the image may corrupt it.
22:58:14 ERROR| [stdout] 13495/163840 = 8.24% allocated, 0.03% fragmented, 0.00% compressed clusters
22:58:14 ERROR| [stdout] Image end offset: 885129216
22:58:14 ERROR| [stderr] ERROR cluster 13505 refcount=1 reference=2
22:58:14 ERROR| Errors found on image: '/mnt/kvm/src/virt-test/shared/data/images/jeos-20-64.qcow2'
22:58:14 WARNI| virt-tests-vm1 is not alive. Can not query the register status
22:58:14 DEBUG| Thread quit. Used to failed to get register info from guest 20150212-225320-Mb1E4VV7 for 1 times.

--

Lucas