Re: [Qemu-devel] runtime Block driver modules (was Re: [PATCH v3 2/2] rbd: link and load librbd dynamically)

[Josh Durgin]

On 04/10/2013 01:09 AM, Stefan Hajnoczi wrote:
> On Wed, Apr 10, 2013 at 2:05 AM, Josh Durgin <address@hidden> wrote:
> > This allows the rbd block driver to detect symbols in the installed
> > version of librbd, and enable or disable features appropriately.  This
> > obviates the #ifdefs regarding librbd versions.
> >
> > Loading librbd dynamically also makes the rbd block driver easier to
> > install and package, since it removes the dependency on librbd at
> > build time.
> >
> > Add structures containing the necessary function pointer signatures
> > and types from librbd, and fill them in the first time the rbd module
> > is used. Use glib's g_module interface so we don't preclude future
> > portability, and don't have to deal with odd dlopen behavior directly.
> >
> > Internally, librbd and some libraries it depends on use C++ templates,
> > which mean that they each contain a defined weak symbol for their
> > definition.  Due to the way the linker resolves duplicate symbols, the
> > libraries loaded by librbd end up using the template definitions from
> > librbd, creating a circular dependency. This means that once librbd is
> > loaded, it won't be unloaded. Changing this behavior might work with a
> > Sun ld, but there doesn't seem to be a portable (or even working with
> > GNU ld) way to hide these C++ symbols correctly. Instead, never unload
> > librbd, and explicitly make it resident.
> >
> > Signed-off-by: Josh Durgin <address@hidden>
> > ---
> >   block/Makefile.objs |    3 +-
> >   block/rbd.c         |  292 ++++++++++++++++++++++++++++++++++++---------------
> >   block/rbd_types.h   |   95 +++++++++++++++++
> >   configure           |   41 +-------
> >   4 files changed, 309 insertions(+), 122 deletions(-)
> >   create mode 100644 block/rbd_types.h
>
> NACK
>
> I think we're solving the problem at the wrong level.  Writing our own
> dynamic linker and adding boilerplate to juggle function pointers
> every time we use a library dependency is ugly.

I wouldn't call this writing our own dynamic linker. To load a library
at runtime requires some boilerplate and using function pointers,
although there could be cleaner ways of setting it up.

> There are two related problems here:
>
> 1. Packagers do not want to enable niche dependencies since users will
> complain that the package is bloated and pulls in too much stuff.
>
> 2. QEMU linked against a newer library version fails to run on hosts
> that have an older library.

Related to 2, there's also a need to rebuild QEMU to be able to access
new functionality in a newer version of a library than QEMU was linked
against. This would be easier to do if the devices were modules
themselves, since only the module that used the library would need to
be rebuilt.

> Problem #1 has several solutions:
>
> 1. Let packagers take care of it.  For example, vim is often shipped
> in several packages that have various amounts of dependencies
> (vim-tiny, vim-gtk, etc).  Packagers create the specialized packages
> for specific groups of users to meet their demands without dragging in
> too many dependencies.
>
> 2. Make QEMU modular - host devices should be shared libraries that
> are loaded at runtime.  There should be no stable API so that
> development stays flexible and we discourage binary-only modules.
> This lets packagers easily ship a qemu-rbd package, for example, that
> drops in a .so file that QEMU can load at runtime.
>
> Problem #2 is already solved:
>
> The dynamic linker will refuse to load the program if there are
> missing symbols.  It's not possible to mix and match binaries across
> environments while downgrading their library dependencies.  With
> effort, this could be doable but it's not an interesting use case that
> many users care about - they get their binaries from a distro or build
> them from source with correct dependencies.
>
> Maybe it's time to move block drivers and other components into modules?

I think that would be great. I'm not too familiar with the other
components, so I'll discuss block drivers as a first step.

I could see this happening in at a few different ways. These could be
incremental steps or final implementations, increasing in complexity
and effort:

1) It seems like it would be possible to achieve an initial version of
runtime loading with minimal driver changes by loading modules instead
of using block_init(), and using the existing driver-specific
block_init() methods (renamed to be the same across modules) as the
only function. This would avoid the function pointer ugliness since
it would be hidden by the existing BlockDriver structure.

2) The existing BlockDriver methods could become the module interface,
with a couple extra functions for defining the current data members
(format_name, protocol_name, instance_size, and create_options).
The block layer could build up a BlockDriver for each module based
on the functions it exports, detected by g_module_symbol().

3) An entirely different module interface from what exists today in the 
block drivers.

Thoughts?

Josh