Re: [Qemu-devel] Slow kernel/initrd loading via fw_cfg; Was Re: Hack integrating SeaBios / LinuxBoot option rom with QEMU trace backends

[Blue Swirl]

On Tue, Oct 11, 2011 at 8:23 AM, Daniel P. Berrange <address@hidden> wrote:
> On Mon, Oct 10, 2011 at 09:01:52PM +0200, Alexander Graf wrote:
>>
>> On 10.10.2011, at 20:53, Anthony Liguori wrote:
>>
>> > On 10/10/2011 12:08 PM, Daniel P. Berrange wrote:
>> >> With the attached patches applied to QEMU and SeaBios, the attached
>> >> systemtap script can be used to debug timings in QEMU startup.
>> >>
>> >> For example, one execution of QEMU produced the following log:
>> >>
>> >>   $ stap qemu-timing.stp
>> >>   0.000 Start
>> >>   0.036 Run
>> >>   0.038 BIOS post
>> >>   0.180 BIOS int 19
>> >>   0.181 BIOS boot OS
>> >>   0.181 LinuxBoot copy kernel
>> >>   1.371 LinuxBoot copy initrd
>> >
>> > Yeah, there was a thread a bit ago about the performance
>> > of the interface to read the kernel/initrd.  I think at it
>> > was using single byte access instructions and there were
>> > patches to use string accessors instead?  I can't remember
>> > where that threaded ended up.
>
> There was initially a huge performance problem, which was
> fixed during the course of the thread, getting to the current
> state where it still takes a few seconds to load large blobs.
> The thread continued with many proposals & counter proposals
> but nothing further really came out of it.
>
>   https://lists.gnu.org/archive/html/qemu-devel/2010-08/msg00133.html
>
> One core point to take away though, is that -kernel/-initrd is
> *not* just for ad-hoc testing by qemu/kernel developers. It is
> critical functionality widely used by users of QEMU in production
> scenarios and performance of it does matter, in some cases, alot.
>
>> IIRC we're already using string accessors, but are still
>> slow. Richard had a nice patch cooked up to basically have
>> the fw_cfg interface be able to DMA its data to the guest.
>> I like the idea. Avi did not.
>
> That's here:
>
>  https://lists.gnu.org/archive/html/qemu-devel/2010-07/msg01037.html
>
>> And yes, bad -kernel performance does hurt in some workloads. A lot.
>
> Let me recap the 3 usage scenarios I believe are most common:
>
>  - Most Linux distro installs done with libvirt + virt-manager/virt-install
>   are done by directly booting the distro's PXE kernel/initrd files.
>   The kernel images are typically < 5 MB, while the initrd images may
>   be as large as 150 MB.  Both are compressed already. An uncompressed
>   initrd image would be more like 300 MB,  so these are avoided for
>   obvious reasons.
>
>   Performance is not really an issue, within reason, since the overall
>   distro installation time will easily dominate, but loading should
>   still be measured in seconds, not minutes.
>
>   The reason for using a kernel/initrd instead of a bootable ISO is
>   to be able to set kernel command line arguments for the installer.
>
>  - libguestfs directly boots its appliance using the regular host's
>   kernel image and a custom built initrd image. The initrd does
>   not contain the entire appliance, just enough to boot up and
>   dynamically read files in from the host OS on demand. This is
>   a so called "supermin appliance".
>
>   The kernel is < 5 MB, while the initrd is approx 100MB. The initrd
>   image is used uncompressed, because decompression time needs to be
>   eliminated from bootup.  Performance is very critical for libguestfs.
>   100's of milliseconds really do make a big difference for it.
>
>   The reason for using a kernel/initrd instead of bootable ISO is to
>   avoid the time required to actually build the ISO, and to avoid
>   having more disks visible in the guest, which could confuse apps
>   using libguestfs which enumerate disks.
>
>  - Application sandbox, directly boots the regular host's kernel and
>   a custom initrd image. The initrd does not contain any files except
>   for the 9p kernel modules and a custom init binary, which mounts
>   the guest root FS from a 9p filesystem export.
>
>   The kernel is < 5 MB, while the initrd is approx 700 KB compressed,
>   or 1.4 MB compressed. Performance for the sandbox is even more
>   critical than for libguestfs. Even 10's of milliseconds make a
>   difference here. The commands being run in the sandbox can be
>   very short lived processes, executed reasonably frequently. The
>   goal is to have end-to-end runtime overhead of < 2 seconds. This
>   includes libvirt guest startup, qemu startup/shutdown, bios time,
>   option ROM time, kernel boot & shutdown time.
>
>   The reason for using a kerenl/initrd instead of a bootable ISO,
>   is that building an ISO requires time itself, and we need to be
>   able to easily pass kernel boot arguments via -append.
>
>
> I'm focusing on the last use case, and if the phase of the moon
> is correct, I can currently executed a sandbox command with a total
> overhead of 3.5 seconds (if using a compressed initrd) of which
> the QEMU execution time is 2.5 seconds.
>
> Of this, 1.4 seconds is the time required by LinuxBoot to copy the
> kernel+initrd. If I used an uncompressed initrd, which I really want
> to, to avoid decompression overhead, this increases to ~1.7 seconds.
> So the LinuxBoot ROM is ~60% of total QEMU execution time, or 40%
> of total sandbox execution overhead.
>
> For comparison I also did a test building a bootable ISO using ISOLinux.
> This required 700 ms for the boot time, which is appoximately 1/2 the
> time reqiured for direct kernel/initrd boot. But you have to then add
> on time required to build the ISO on every boot, to add custom kernel
> command line args. So while ISO is faster than LinuxBoot currently
> there is still non-negligable overhead here that I want to avoid.
>
> For further comparison I tested with Rich Jones' patches which add a
> DMA-like inteface to fw_cfg. With this the time spent in the LinuxBoot
> option ROM was as close to zero as matters.
>
> So obviously, my preference is for -kernel/-initrd to be made very fast
> using the DMA-like patches, or any other patches which could achieve
> similarly high performance for -kernel/-initd.

I don't understand why PC can't use the same way of loading initrd by
QEMU to guest memory before boot as Sparc32 uses. It should even be
possible to deduplicate the kernel and initrd images: improve the
loader to use mmap() for loading so that several guests would use the
same pages. Preloaded kernel and initrd are paravirtual anyway, there
could be even guest visible changes if ever needed (e.g. map
kernel/initrd pages outside of normal RAM areas).