[Qemu-devel] [Bug 1833204] [NEW] VM failed to start in nested virtualization with error "KVM: entry failed, hardware error 0x0"

[Wei, Yu]

Public bug reported:

Hi,

I have 3 ubuntu nodes provisioned by IaaS. 
Then I tried to launch VM again in my ubuntu nodes.
It's a little strange that VM could be started successfully in two nodes. 
And always failed in one nodes with error "KVM: entry failed, hardware error 
0x0". 

When using virsh to resume the VM, it failed with following error,
virsh # list
 Id   Name                State
----------------------------------
 1    default_vm-cirros   paused

virsh # resume default_vm-cirros
error: Failed to resume domain default_vm-cirros
error: internal error: unable to execute QEMU command 'cont': Resetting the 
Virtual Machine is required


The detailed log from /var/log/libvirt/qemu/default_vm-cirros.log is as below.
```
2019-06-18 09:55:52.397+0000: starting up libvirt version: 5.0.0, package: 
1.fc28 (Unknown, 2019-01-22-08:04:34, 64723eea657e48d296e6beb0b1be9c4c), qemu 
version: 3.1.0qemu-3.1.0-4.fc28, kernel: 4.15.0-47-generic, hostname: vm-cirros
LC_ALL=C \
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin \
HOME=/root \
QEMU_AUDIO_DRV=none \
/usr/bin/qemu-system-x86_64 \
-name guest=default_vm-cirros,debug-threads=on \
-S \
-object 
secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-1-default_vm-cirros/master-key.aes
 \
-machine pc-q35-3.1,accel=kvm,usb=off,dump-guest-core=off \
-cpu 
Broadwell-IBRS,vme=on,ss=on,vmx=on,f16c=on,rdrand=on,hypervisor=on,arat=on,tsc_adjust=on,mpx=on,avx512f=on,avx512cd=on,ssbd=on,xsaveopt=on,abm=on,invpcid=off
 \
-m 489 \
-realtime mlock=off \
-smp 1,sockets=1,cores=1,threads=1 \
-object iothread,id=iothread1 \
-uuid 0d2a2043-41c0-59c3-9b17-025022203668 \
-no-user-config \
-nodefaults \
-chardev socket,id=charmonitor,fd=22,server,nowait \
-mon chardev=charmonitor,id=monitor,mode=control \
-rtc base=utc \
-no-shutdown \
-boot strict=on \
-device 
pcie-root-port,port=0x10,chassis=1,id=pci.1,bus=pcie.0,multifunction=on,addr=0x2
 \
-device pcie-root-port,port=0x11,chassis=2,id=pci.2,bus=pcie.0,addr=0x2.0x1 \
-device pcie-root-port,port=0x12,chassis=3,id=pci.3,bus=pcie.0,addr=0x2.0x2 \
-device pcie-root-port,port=0x13,chassis=4,id=pci.4,bus=pcie.0,addr=0x2.0x3 \
-device pcie-root-port,port=0x14,chassis=5,id=pci.5,bus=pcie.0,addr=0x2.0x4 \
-device virtio-serial-pci,id=virtio-serial0,bus=pci.2,addr=0x0 \
-drive 
file=/var/run/kubevirt-ephemeral-disks/container-disk-data/default/vm-cirros/disk_containerdisk/disk-image.raw,format=raw,if=none,id=drive-ua-containerdisk,cache=none
 \
-device 
virtio-blk-pci,scsi=off,bus=pci.3,addr=0x0,drive=drive-ua-containerdisk,id=ua-containerdisk,bootindex=1,write-cache=on
 \
-drive 
file=/var/run/kubevirt-ephemeral-disks/cloud-init-data/default/vm-cirros/noCloud.iso,format=raw,if=none,id=drive-ua-cloudinitdisk,cache=none
 \
-device 
virtio-blk-pci,scsi=off,bus=pci.4,addr=0x0,drive=drive-ua-cloudinitdisk,id=ua-cloudinitdisk,write-cache=on
 \
-netdev tap,fd=24,id=hostua-default,vhost=on,vhostfd=25 \
-device 
virtio-net-pci,host_mtu=1430,netdev=hostua-default,id=ua-default,mac=16:57:38:cd:57:cb,bus=pci.1,addr=0x0
 \
-chardev socket,id=charserial0,fd=26,server,nowait \
-device isa-serial,chardev=charserial0,id=serial0 \
-chardev socket,id=charchannel0,fd=27,server,nowait \
-device 
virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=org.qemu.guest_agent.0
 \
-vnc 
vnc=unix:/var/run/kubevirt-private/3b22a138-91af-11e9-af36-0016ac101123/virt-vnc
 \
-device VGA,id=video0,vgamem_mb=16,bus=pcie.0,addr=0x1 \
-sandbox 
on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny \
-msg timestamp=on
KVM: entry failed, hardware error 0x0
EAX=00000000 EBX=00000000 ECX=00000000 EDX=000306d2
ESI=00000000 EDI=00000000 EBP=00000000 ESP=00000000
EIP=0000fff0 EFL=00000002 [-------] CPL=0 II=0 A20=1 SMM=0 HLT=0
ES =0000 00000000 0000ffff 00009300
CS =f000 ffff0000 0000ffff 00009b00
SS =0000 00000000 0000ffff 00009300
DS =0000 00000000 0000ffff 00009300
FS =0000 00000000 0000ffff 00009300
GS =0000 00000000 0000ffff 00009300
LDT=0000 00000000 0000ffff 00008200
TR =0000 00000000 0000ffff 00008b00
GDT=     00000000 0000ffff
IDT=     00000000 0000ffff
CR0=60000010 CR2=00000000 CR3=00000000 CR4=00000000
DR0=0000000000000000 DR1=0000000000000000 DR2=0000000000000000 
DR3=0000000000000000 
DR6=00000000ffff0ff0 DR7=0000000000000400
EFER=0000000000000000
Code=06 66 05 00 00 01 00 8e c1 26 66 a3 14 f5 66 5b 66 5e 66 c3 <ea> 5b e0 00 
f0 30 36 2f 32 33 2f 39 39 00 fc 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
```

Ubuntu node version as follow,
cat /etc/os-release 
NAME="Ubuntu"
VERSION="18.04.2 LTS (Bionic Beaver)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 18.04.2 LTS"
VERSION_ID="18.04"
HOME_URL="https://www.ubuntu.com/";
SUPPORT_URL="https://help.ubuntu.com/";
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/";
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy";
VERSION_CODENAME=bionic
UBUNTU_CODENAME=bionic

Output of `uname -a` is:
4.15.0-47-generic #50-Ubuntu SMP Wed Mar 13 10:44:52 UTC 2019 x86_64 x86_64 
x86_64 GNU/Linux


Any additional information needed, please let me know.
Thx.

** Affects: qemu
     Importance: Undecided
         Status: New


** Tags: kvm qemu

-- 
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.
https://bugs.launchpad.net/bugs/1833204

Title:
  VM failed to start in nested virtualization with error "KVM: entry
  failed, hardware error 0x0"

Status in QEMU:
  New

Bug description:
  Hi,

  I have 3 ubuntu nodes provisioned by IaaS. 
  Then I tried to launch VM again in my ubuntu nodes.
  It's a little strange that VM could be started successfully in two nodes. 
  And always failed in one nodes with error "KVM: entry failed, hardware error 
0x0". 

  When using virsh to resume the VM, it failed with following error,
  virsh # list
   Id   Name                State
  ----------------------------------
   1    default_vm-cirros   paused

  virsh # resume default_vm-cirros
  error: Failed to resume domain default_vm-cirros
  error: internal error: unable to execute QEMU command 'cont': Resetting the 
Virtual Machine is required

  
  The detailed log from /var/log/libvirt/qemu/default_vm-cirros.log is as below.
  ```
  2019-06-18 09:55:52.397+0000: starting up libvirt version: 5.0.0, package: 
1.fc28 (Unknown, 2019-01-22-08:04:34, 64723eea657e48d296e6beb0b1be9c4c), qemu 
version: 3.1.0qemu-3.1.0-4.fc28, kernel: 4.15.0-47-generic, hostname: vm-cirros
  LC_ALL=C \
  PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin \
  HOME=/root \
  QEMU_AUDIO_DRV=none \
  /usr/bin/qemu-system-x86_64 \
  -name guest=default_vm-cirros,debug-threads=on \
  -S \
  -object 
secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-1-default_vm-cirros/master-key.aes
 \
  -machine pc-q35-3.1,accel=kvm,usb=off,dump-guest-core=off \
  -cpu 
Broadwell-IBRS,vme=on,ss=on,vmx=on,f16c=on,rdrand=on,hypervisor=on,arat=on,tsc_adjust=on,mpx=on,avx512f=on,avx512cd=on,ssbd=on,xsaveopt=on,abm=on,invpcid=off
 \
  -m 489 \
  -realtime mlock=off \
  -smp 1,sockets=1,cores=1,threads=1 \
  -object iothread,id=iothread1 \
  -uuid 0d2a2043-41c0-59c3-9b17-025022203668 \
  -no-user-config \
  -nodefaults \
  -chardev socket,id=charmonitor,fd=22,server,nowait \
  -mon chardev=charmonitor,id=monitor,mode=control \
  -rtc base=utc \
  -no-shutdown \
  -boot strict=on \
  -device 
pcie-root-port,port=0x10,chassis=1,id=pci.1,bus=pcie.0,multifunction=on,addr=0x2
 \
  -device pcie-root-port,port=0x11,chassis=2,id=pci.2,bus=pcie.0,addr=0x2.0x1 \
  -device pcie-root-port,port=0x12,chassis=3,id=pci.3,bus=pcie.0,addr=0x2.0x2 \
  -device pcie-root-port,port=0x13,chassis=4,id=pci.4,bus=pcie.0,addr=0x2.0x3 \
  -device pcie-root-port,port=0x14,chassis=5,id=pci.5,bus=pcie.0,addr=0x2.0x4 \
  -device virtio-serial-pci,id=virtio-serial0,bus=pci.2,addr=0x0 \
  -drive 
file=/var/run/kubevirt-ephemeral-disks/container-disk-data/default/vm-cirros/disk_containerdisk/disk-image.raw,format=raw,if=none,id=drive-ua-containerdisk,cache=none
 \
  -device 
virtio-blk-pci,scsi=off,bus=pci.3,addr=0x0,drive=drive-ua-containerdisk,id=ua-containerdisk,bootindex=1,write-cache=on
 \
  -drive 
file=/var/run/kubevirt-ephemeral-disks/cloud-init-data/default/vm-cirros/noCloud.iso,format=raw,if=none,id=drive-ua-cloudinitdisk,cache=none
 \
  -device 
virtio-blk-pci,scsi=off,bus=pci.4,addr=0x0,drive=drive-ua-cloudinitdisk,id=ua-cloudinitdisk,write-cache=on
 \
  -netdev tap,fd=24,id=hostua-default,vhost=on,vhostfd=25 \
  -device 
virtio-net-pci,host_mtu=1430,netdev=hostua-default,id=ua-default,mac=16:57:38:cd:57:cb,bus=pci.1,addr=0x0
 \
  -chardev socket,id=charserial0,fd=26,server,nowait \
  -device isa-serial,chardev=charserial0,id=serial0 \
  -chardev socket,id=charchannel0,fd=27,server,nowait \
  -device 
virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=org.qemu.guest_agent.0
 \
  -vnc 
vnc=unix:/var/run/kubevirt-private/3b22a138-91af-11e9-af36-0016ac101123/virt-vnc
 \
  -device VGA,id=video0,vgamem_mb=16,bus=pcie.0,addr=0x1 \
  -sandbox 
on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny \
  -msg timestamp=on
  KVM: entry failed, hardware error 0x0
  EAX=00000000 EBX=00000000 ECX=00000000 EDX=000306d2
  ESI=00000000 EDI=00000000 EBP=00000000 ESP=00000000
  EIP=0000fff0 EFL=00000002 [-------] CPL=0 II=0 A20=1 SMM=0 HLT=0
  ES =0000 00000000 0000ffff 00009300
  CS =f000 ffff0000 0000ffff 00009b00
  SS =0000 00000000 0000ffff 00009300
  DS =0000 00000000 0000ffff 00009300
  FS =0000 00000000 0000ffff 00009300
  GS =0000 00000000 0000ffff 00009300
  LDT=0000 00000000 0000ffff 00008200
  TR =0000 00000000 0000ffff 00008b00
  GDT=     00000000 0000ffff
  IDT=     00000000 0000ffff
  CR0=60000010 CR2=00000000 CR3=00000000 CR4=00000000
  DR0=0000000000000000 DR1=0000000000000000 DR2=0000000000000000 
DR3=0000000000000000 
  DR6=00000000ffff0ff0 DR7=0000000000000400
  EFER=0000000000000000
  Code=06 66 05 00 00 01 00 8e c1 26 66 a3 14 f5 66 5b 66 5e 66 c3 <ea> 5b e0 
00 f0 30 36 2f 32 33 2f 39 39 00 fc 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  ```

  Ubuntu node version as follow,
  cat /etc/os-release 
  NAME="Ubuntu"
  VERSION="18.04.2 LTS (Bionic Beaver)"
  ID=ubuntu
  ID_LIKE=debian
  PRETTY_NAME="Ubuntu 18.04.2 LTS"
  VERSION_ID="18.04"
  HOME_URL="https://www.ubuntu.com/";
  SUPPORT_URL="https://help.ubuntu.com/";
  BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/";
  
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy";
  VERSION_CODENAME=bionic
  UBUNTU_CODENAME=bionic

  Output of `uname -a` is:
  4.15.0-47-generic #50-Ubuntu SMP Wed Mar 13 10:44:52 UTC 2019 x86_64 x86_64 
x86_64 GNU/Linux


  Any additional information needed, please let me know.
  Thx.

To manage notifications about this bug go to:
https://bugs.launchpad.net/qemu/+bug/1833204/+subscriptions