Re: [Qemu-devel] Modularizing QEMU RFC

[Daniel P. Berrange]

On Mon, Aug 03, 2015 at 11:09:06AM +0800, Fam Zheng wrote:
> On Fri, 07/31 17:45, Marc Marí wrote:
> > Hi everyone
> > 
> > I propose improving the current modular driver system for QEMU so it
> > can benefit everybody in speed and flexibility. I'm looking for other
> > ideas, comments, critics, etc.
> > 
> > - Background -
> > In order to speed up QEMU, I'm looking at the high number of libraries
> > and dependencies that it loads. I've generated a QEMU image that needs
> > 145 shared libraries to start, and there were still some options
> > disabled. This is a lot, and this means that it is slow.
> > 
> > So I've been looking at actual module system. Yes, QEMU does have a
> > module system, but disabled by default. The problem is, the modules get
> > loaded always during startup. This means, booting with modules enabled
> > is even slower, because loading at runtime is slower that letting the
> > linker do all the work at the start. At this point, I doubt of the
> > benefits of the actual modular system.
> > 
> > But, if disabling the actual block modules (iscsi, curl, rbd, gluster,
> > ssh and dmg) gives 40 ms of speedup, I think is worth an effort of
> > improving modules, and modularizing new parts
> > 
> > - Current module flow -
> > The current module system is based on shared libraries. Each of these
> > libraries has a constructor that registers the BlockDriver structure to
> > the global bdrv_drivers list. This list is later searched by the type,
> > the protocol, etc.
> > 
> > - Proposals -
> > I have in mind two ideas, which are not mutually exclusive:
> > 
> > -- A --
> > Having a huge lookup table with the names of the drivers that are in
> > modules, and the name of the modules where it can be found. When
> > some type is not found in the driver lists, this table can be traversed
> > to find the library and load it.
> > 
> > This requires an effort by the driver developer to fill the tables. But
> > the refactoring work can stay localized in the drivers and the modules,
> > it is (possibly) not necessary to touch any other part of the QEMU
> > system.
> > 
> > I don't specially like this huge manual-edited table.

You don't neccessarily need to have a look up table at all - just use a
1-1 mapping between driver name and the .so module file on disk, so you
can just directly load it when needed.

> 
> Yeah, the way to fill the tables is an (implementation) question, but there 
> are
> still more questions if we go this way...
> 
> bdrv_find_protocol is easy, the protocol name is extracted from image uri or
> blockdev options, we can load the module by the name.
> 
> bdrv_probe_all is harder. If we modularize a format driver, its .bdrv_probe
> code will be in the module. If we want to do the format detection, we need to
> load all format drivers. This means if the command line has an unspecified
> format, we'll still need to load all drivers at starting phase. (I wish all
> formats are probed according to magic bytes at offset 0, so we can simplify 
> the
> .bdrv_probe logic and do it with data matching in block.c like the protocol
> case, but that's not true for VMDK :( )

In general we recommend apps to not rely on probing of disk formats
because of the inherant security problems which have led to multiple
CVEs for many apps over the years. As such I think you could make the
argument that probing is a problem that is not worth putting much
effort into solving.

As such I think it would be fine to say that if 'bdrv_probe' is invoked,
QEMU should just load all modules it can find. Or you could have an
/etc/qemu/block-probe.conf file which lists the modules that are
acceptable to auto-load for probing. Keep it simple on the basis that
most apps should not rely on this.

> > -- B --
> > The same --enable-X that is done at compile time, but directly on the
> > command line when running QEMU or even in hot at the monitor.
> > 
> > This solution requires work on the monitor, the command line
> > processing, the modules and the drivers system. It is also less
> > transparent to the final user.
> 
> I'm afraid this breaks backward compatibility. :(

Requiring an --enable-X command line arg also exposes external apps to
an implementation detail of QEMU's module system. I think that is a bad
idea in general, even ignoring backwards compat issues. Any loading of
modules should be more or less invisible from the POV of an application.

Regards,
Daniel
-- 
|: http://berrange.com      -o-    http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org              -o-             http://virt-manager.org :|
|: http://autobuild.org       -o-         http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org       -o-       http://live.gnome.org/gtk-vnc :|