[Qemu-devel] [Bug 1797332] Re: qemu nested virtualization is not working with Ubuntu16.04

[Thomas Huth]

** Project changed: qemu => qemu (Ubuntu)

-- 
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.
https://bugs.launchpad.net/bugs/1797332

Title:
  qemu nested virtualization is not working with Ubuntu16.04

Status in qemu package in Ubuntu:
  New

Bug description:
  # 1 What am I trying to do ? #

  I want to use `libvirt` `qemu/KVM` with **nested virtualization** like 
described
  in [1] and [2].
  **But it does not work with Ubuntu16.04.** It worked some times ago, but not
  anymore.

  
  I want 2 levels of virtualization like this:

  * L0 – the bare metal host, running KVM on `Ubuntu 16.04`
  * L1 – a `Ubuntu 16.04` VM running on L0; also called the "guest hypervisor" 
    — as it itself is capable of running KVM
  * L2 – a `Ubuntu 16.04` VM running on L1, also called the "nested guest"

  
  [1] 
https://docs.fedoraproject.org/en-US/quick-docs/using-nested-virtualization-in-kvm/
  [2] https://www.linux-kvm.org/page/Nested_Guests

  
  My goal is to deploy an `OpenStack` environnement on top of VMs rather than on
  bare metal hosts for convenience for a lab experiment. As a result, the 
  `OpenStack` nodes are L1 VMs. Compute nodes are L1 VMs as well and the VMs 
  created with `OpenStack` and wich are running on the compute nodes are L2 VMs.



  
  # 2 What is my problem ? #

  I can **not** run my 2nd levels of virtualization in 16.04:

  * L0 is just fine: running `Ubuntu 16.04.5 LTS`, installed with the `.iso` 
image
  * L1: I install `libvirt` + `KVM` on L0. I can run VMs like the `Ubuntu16.04` 
    cloud image on L0.
  * L2: I install `libvirt` + `KVM` on L1 as well. But I **can not** run VMs on 
    L1: I get `kernel panic` or `general protection fault`.

  
  **But if I do the same with Ubuntu18.04** (on the same hardware) instead of 
  `Ubuntu16.04`, it works without faults.
  I don't change the configuration or `virt-install scripts` (other than using 
  the 18.04 .iso and cloud image).



  
  # 3 My libvirt installation for Ubuntu16.04 #

  I install `libvir KVM` in both L0 and L1 using a custom repository [3] from 
  `OpenStack` team, because their version of libvirt in this repo is newer than 
 
  the one on Ubuntu 16.04 official repo and it match the version of `libvirt` 
  in Ubuntu 18.04.

  [3] https://wiki.ubuntu.com/OpenStack/CloudArchive



  
  # 4 hardware and CPU #

  CPU is:
  > Intel(R) Xeon(R) CPU E5-2650 v4 @ 2.20GHz
  > Intel virt is enable in the bios/uefi.

  The rest is standard HDD, standard I/O...



  
  # 5 .iso and cloud image #

  I download .iso for L0 bare metal server and cloud image 
  for L1/L2 VMs from official repository:

  Ubuntu 16.04
   * http://releases.ubuntu.com/16.04/
   * https://cloud-images.ubuntu.com/releases/16.04/release/

  Ubuntu 18.04
   * http://releases.ubuntu.com/bionic/
   * https://cloud-images.ubuntu.com/releases/18.04/release/



  
  # 6 Details #

  ## Details about L0 Ubuntu 16.04 bare metal host ##
  L0 is running `Ubuntu 16.04.5 LTS` installed with the .iso.

  
  **kernel**
  ```
  address@hidden:~$ uname -a
  Linux L0 4.4.0-137-generic #163-Ubuntu SMP Mon Sep 24 13:14:43 UTC 2018 
x86_64 x86_64 x86_64 GNU/Linux
  ```

  **libvirt version** running on L0
  ```
  address@hidden:~$ virsh version
  Compiled against library: libvirt 4.0.0
  Using library: libvirt 4.0.0
  Using API: QEMU 4.0.0
  Running hypervisor: QEMU 2.11.1
  ```

  **qemu version detail**
  ```
  address@hidden:~$ qemu-system-x86_64 --version
  QEMU emulator version 2.11.1(Debian 1:2.11+dfsg-1ubuntu7.5~cloud0)
  Copyright (c) 2003-2017 Fabrice Bellard and the QEMU Project developers
  ```

  **KVM acceleration**
  ```
  address@hidden:~$ kvm-ok 
  INFO: /dev/kvm exists
  KVM acceleration can be used
  ```

  **nested parameter**
  ```
  address@hidden:~$ cat /sys/module/kvm_intel/parameters/nested
  Y
  ```

  **number of CPU**
  ```
  address@hidden:~$ egrep -c '(vmx|svm)' /proc/cpuinfo
  48
  ```


  ## Details about a L1 Ubuntu 16.04 VM ##
  A VM in L1 (which is running on L0) which is running `Ubuntu 16.04.5 LTS` 
  installed by a cloud image.

  **kernel**
  ```
  address@hidden:~$ uname -a
  Linux L1 4.4.0-137-generic #163-Ubuntu SMP Mon Sep 24 13:14:43 UTC 2018 
x86_64 x86_64 x86_64 GNU/Linux
  ```

  **libvirt version** running on the L1 VM
  ```
  address@hidden:~$ sudo virsh version
  Compiled against library: libvirt 4.0.0
  Using library: libvirt 4.0.0
  Using API: QEMU 4.0.0
  Running hypervisor: QEMU 2.11.1
  ```

  **qemu version detail**
  ```
  address@hidden:~$ qemu-system-x86_64 --version
  QEMU emulator version 2.11.1(Debian 1:2.11+dfsg-1ubuntu7.5~cloud0)
  Copyright (c) 2003-2017 Fabrice Bellard and the QEMU Project developers
  ```

  **KVM acceleration**
  ```
  address@hidden:~$ kvm-ok 
  INFO: /dev/kvm exists
  KVM acceleration can be used
  ```

  **nested parameter**
  ```
  address@hidden:~$ cat /sys/module/kvm_intel/parameters/nested
  Y
  ```

  **number of CPU**, which are vCPU given by L0 to the L1 VM
  I give 20 vCPU.
  ```
  address@hidden:~$ egrep -c '(vmx|svm)' /proc/cpuinfo
  20
  ```


  ## L1 VM virt-install script parameter ##
  If you want to reproduce an L1 VM, I followed this [4]:

  ```
  virt-install \
      --connect=qemu:///system \
      --name $VMName \
      --memory $RAM \
      --vcpus $VCPUS \
      --cpu host \
      --metadata description=$DESCRIPTION \
      --os-type linux \
      --os-variant ubuntu16.04 \
      --disk $DISK_PATH/$VMName.$DISK_FORMAT,size=$DISK_SIZE,bus=virtio \
      --disk $CFGIMG_PATH/config_$VMName.$DISK_FORMAT,device=cdrom \
      --network bridge=virbr0 \
      --graphics none \
      --console pty,target_type=serial \
      --hvm
  ```

  [4] https://youth2009.org/post/kvm-with-ubuntu-cloud-image/


  ## Details about a L2 VM ##

  I want to create a L2 `Ubuntu 16.04.5 LTS` VM installed by a cloud image VM 
  within my L1 `KVM` VM. But whatever I do, my L2 VM crash before finishing to 
be 
  instantiated. I get `kernel panic` or `general protection fault`.

  
  Here is the log of an L2 VM after the instanciation failed:
  ```
  address@hidden:~$ less /var/log/libvirt/qemu/VMNAME.log

  2018-10-11T07:40:45.837151Z qemu-system-x86_64: -chardev pty,id=charserial0: 
char device redirected to /dev/pts/1 (label charserial0)
  2018-10-11T07:40:45.844279Z qemu-system-x86_64: warning: host doesn't support 
requested feature: CPUID.07H:EBX.invpcid [bit 10]
  2018-10-11T07:40:45.848532Z qemu-system-x86_64: warning: host doesn't support 
requested feature: CPUID.07H:EBX.invpcid [bit 10]
  ```

  
  If you want to reproduce an L2 VM running on L1, follow [4].

  
  **However** a Cirros OS image can run on a L1 VM !



  
  # 7 Thoughts #
  I think this is a bug in either `Ubuntu16.04` or `libvirt`.
  All the information are here to reproduce the bug, I think.

  
  If I do the same with `Ubuntu 18.04`, on the same hardware, following the 
same 
  steps but with Ubuntu 18.04 .iso and cloud image, it works.

  It works if:

  * L0 = Ubuntu18.04 (.iso) + qemu/KVM
  * L1 = Ubuntu18.04 (cloud image) + qemu/KVM
  * L2 = Ubuntu18.04 (cloud image)

  
  It also works if:

  * L0 = Ubuntu18.04 (.iso) + qemu/KVM
  * L1 = Ubuntu18.04 (cloud image) + qemu/KVM
  * L2 = Ubuntu16.04 (cloud image)


  
  Thank you for your time reading !
  --
  nico

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/qemu/+bug/1797332/+subscriptions