Re: [Qemu-devel] [PATCH v2 4/4] hw/i386: Introduce the microvm machine type

[Sergio Lopez]

Sergio Lopez <address@hidden> writes:

> Paolo Bonzini <address@hidden> writes:
>
>> On 02/07/19 12:52, Sergio Lopez wrote:
>>> As I said, I'm also in favor of microvm supporting booting from
>>> firmware in the future, as long we keep the simple mode too.
>>
>> The simple mode adds code to QEMU's x86 target that only exists to
>> support microvm.  It should be motivated by a clear win in boot times.
>
> OK. When I'm back from my PTO, I'll work on adding the firmware
> support to microvm. I'll run and share some numbers to see whether the
> simple mode makes sense or we can just rely on qboot for lower boot
> times plus SeaBIOS for compatibility.

I've just added support for starting the machine from SeaBIOS (Stefan
Hajnoczi pointed in another thread that it can be as fast as qboot, and
given that the latter doesn't support mptables, I just tested this
one). I tried to keep it as minimalistic as possible, but it still
required an RTC (mc146818), which dragged in an ISA BUS, and this one a
KVM PIT.

I ran some numbers using Stefano Garzarella's qemu-boot-time scripts
[1] on a server with 2xIntel Xeon Silver 4114 2.20GHz, using the
upstream QEMU (474f3938d79ab36b9231c9ad3b5a9314c2aeacde) built with
minimal features [2]. The VM boots a minimal kernel [3] without initrd,
using a kata container image as root via virtio-blk (though this isn't
really relevant, as we're just taking measurements until the kernel is
about to exec init).

 ---------------------
 | QEMU with SeaBIOS |
 ---------------------

Command line:

./x86_64-softmmu/qemu-system-x86_64 -m 512m -enable-kvm -M microvm,legacy 
-kernel /root/src/images/vmlinux-5.2 -append "console=hvc0 reboot=k panic=1 
root=/dev/vda quiet virtio_mmio.device=512@0xd0000600:15 
virtio_mmio.device=512@0xd0000400:14" -smp 1 -nodefaults -no-user-config 
-chardev pty,id=virtiocon0,server -device virtio-serial-device -device 
virtconsole,chardev=virtiocon0 -drive 
id=test,file=/usr/share/kata-containers/kata-containers.img,format=raw,if=none 
-device virtio-blk-device,drive=test -monitor stdio

Average boot times after 10 consecutive runs:

 qemu_init_end: 65.958714
 linux_start_kernel: 77.735803 (+11.777089)
 linux_start_user: 127.360739 (+49.624936)

Exposed I/O ports and MMIOs:

 address-space: memory
  0000000000000000-ffffffffffffffff (prio 0, i/o): system
    0000000000000000-000000001fffffff (prio 0, i/o): alias ram-below-4g 
@microvm.ram 0000000000000000-000000001fffffff
    00000000000e0000-00000000000fffff (prio 1, i/o): alias isa-bios @pc.bios 
0000000000000000-000000000001ffff
    00000000d0000000-00000000d00001ff (prio 0, i/o): virtio-mmio
    00000000d0000200-00000000d00003ff (prio 0, i/o): virtio-mmio
    00000000d0000400-00000000d00005ff (prio 0, i/o): virtio-mmio
    00000000d0000600-00000000d00007ff (prio 0, i/o): virtio-mmio
    00000000fee00000-00000000feefffff (prio 4096, i/o): kvm-apic-msi
    00000000fffe0000-00000000ffffffff (prio 0, ram): pc.bios

 address-space: I/O
  0000000000000000-000000000000ffff (prio 0, i/o): io
    0000000000000020-0000000000000021 (prio 0, i/o): kvm-pic
    0000000000000040-0000000000000043 (prio 0, i/o): kvm-pit
    0000000000000070-0000000000000071 (prio 0, i/o): rtc
      0000000000000070-0000000000000070 (prio 0, i/o): rtc-index
    000000000000007e-000000000000007f (prio 0, i/o): kvmvapic
    00000000000000a0-00000000000000a1 (prio 0, i/o): kvm-pic
    00000000000004d0-00000000000004d0 (prio 0, i/o): kvm-elcr
    00000000000004d1-00000000000004d1 (prio 0, i/o): kvm-elcr
    0000000000000510-0000000000000511 (prio 0, i/o): fwcfg
    0000000000000514-000000000000051b (prio 0, i/o): fwcfg.dma


 -------------------
 | QEMU direct PVH |
 -------------------

Command line:

 ./x86_64-softmmu/qemu-system-x86_64 -m 512m -enable-kvm -M microvm -kernel 
/root/src/images/vmlinux-5.2 -append "console=hvc0 reboot=k panic=1 
root=/dev/vda quiet" -smp 1 -nodefaults -no-user-config -chardev 
pty,id=virtiocon0,server -device virtio-serial-device -device 
virtconsole,chardev=virtiocon0 -drive 
id=test,file=/usr/share/kata-containers/kata-containers.img,format=raw,if=none 
-device virtio-blk-device,drive=test -monitor stdio

Average boot times after 10 consecutive runs:

 qemu_init_end: 64.043264
 linux_start_kernel: 65.481782 (+1.438518)
 linux_start_user: 114.938353 (+49.456571)

Exposed I/O ports and MMIOs:

 address-space: memory
  0000000000000000-ffffffffffffffff (prio 0, i/o): system
    0000000000000000-000000001fffffff (prio 0, i/o): alias ram-below-4g 
@microvm.ram 0000000000000000-000000001fffffff
    00000000d0000000-00000000d00001ff (prio 0, i/o): virtio-mmio
    00000000d0000200-00000000d00003ff (prio 0, i/o): virtio-mmio
    00000000d0000400-00000000d00005ff (prio 0, i/o): virtio-mmio
    00000000d0000600-00000000d00007ff (prio 0, i/o): virtio-mmio
    00000000fec00000-00000000fec00fff (prio 0, i/o): kvm-ioapic
    00000000fee00000-00000000feefffff (prio 4096, i/o): kvm-apic-msi

 address-space: I/O
  0000000000000000-000000000000ffff (prio 0, i/o): io
    000000000000007e-000000000000007f (prio 0, i/o): kvmvapic


 --------------
 | Comparison |
 --------------

Average boot time:

 * Relying on SeaBIOS, when compared with direct PVH boot, as a total
   average overhead of ~12ms. The cost of initializing QEMU increases in
   ~2ms (probably due to need to instantiate more devices), while the
   other ~10ms is the SeaBIOS overhead.

Exposed I/O ports and MMIOs:

 * The following 8 I/O ports are only present in the version relying on SeaBIOS:

    0000000000000020-0000000000000021 (prio 0, i/o): kvm-pic
    0000000000000040-0000000000000043 (prio 0, i/o): kvm-pit
    0000000000000070-0000000000000071 (prio 0, i/o): rtc
      0000000000000070-0000000000000070 (prio 0, i/o): rtc-index
    00000000000000a0-00000000000000a1 (prio 0, i/o): kvm-pic
    00000000000004d0-00000000000004d0 (prio 0, i/o): kvm-elcr
    00000000000004d1-00000000000004d1 (prio 0, i/o): kvm-elcr
    0000000000000510-0000000000000511 (prio 0, i/o): fwcfg
    0000000000000514-000000000000051b (prio 0, i/o): fwcfg.dma

 * The following MMIO region is only present in the direct boot version:

    00000000fec00000-00000000fec00fff (prio 0, i/o): kvm-ioapic


 ---------------
 | Conclusions |
 ---------------

Objectively, the version that boots directly the kernel using PVH is 10%
faster and has a slightly larger exposed surface. Whether this is enough
to justify its existence is quite subjective.

In my opinion, not only I think it makes sense to have it, but I also
think there's little reason to have the firmware reliant version, given
the nature and purpose of microvm.

Sergio.

[1] https://github.com/stefano-garzarella/qemu-boot-time
[2] https://paste.fedoraproject.org/paste/YZ9Ok-dJtQrc0xxctFm-nw
[3] https://paste.fedoraproject.org/paste/sck0jfioAJdMq51HH6wkmA

signature.asc
Description: PGP signature