Re: [Qemu-devel] [RFC] QCFG: a new mechanism to replace QemuOpts and option handling

[Markus Armbruster]

Anthony Liguori <address@hidden> writes:

> On 03/17/2011 10:22 AM, Markus Armbruster wrote:
>>
>>> void qcfg_handle_vnc(VncConfig *option, Error **errp)
>>> {
>>> }
>>>
>>> And that's it.  You can squirrel away the option such that they all
>>> can be processed later, you can perform additional validation and
>>> return an error, or you can implement the appropriate logic.
>>>
>>> The VncConfig structure is a proper C data structure.  The advantages
>>> of this approach compared to QemuOpts are similar to QAPI:
>>>
>>> 1) Strong typing means less bugs with lack of command line validation.
>>> In many cases, a bad command line results in a SEGV today.
>> Static typing, you mean.
>
> Both.  We don't consistently check error returns when dealing with
> QemuOpts so while it does implement dynamic typing, the dynamic typing
> is not consistently enforced.

Yes.  Dynamic typing in C takes a lot of discipline.

>>> 2) Every option is formally specified and documented in a way that is
>>> both rigorous and machine readable.  This means we can generate high
>>> quality documentation in a variety of formats.
>> You make it sound like QemuOpts wouldn't specify options.  It does.
>> Where your proposal differs from QemuOpts, in my opinion, is that it
>> uses C types rather than QemuOpts' very own ad hoc type system, and is
>> more expressive, see your 6) below.
>
> My point in that bullet is that the documentation is machine readable
> and highly structured.
>
>>> 3) The command line parameters support full introspection.  This
>>> should provide the same functionality as Dan's earlier introspection
>>> patches.
>> Again, this isn't something QemuOpts could not do.  We just neglected to
>> make its option specification available outside QEMU, mostly because for
>> lack of consensus on what we want to expose there.
>
> It's all just a matter of bits, right :-)
>
> But QemuOpts is just about parsing a string, it doesn't really
> explicitly tie to a command line.  Another layer would be needed to
> clearly map that option foo is associated with this QemuOpts.
>
> So yeah, it could grow in that direction but there's a lot more to it
> then just returning a list of names and ties given a qemu opts
> definition.
>
>
>>> 5) Very complex data types can be implemented.  We had some discussion
>>> of supporting nested structures with -blockdev.  This wouldn't work
>>> with QemuOpts but I've already implemented it with QCFG (blockdev
>>> syntax is my test case right now).  The syntax I'm currently using is
>>> -blockdev
>>> cache=none,id=foo,format.qcow.protocol.nbd.hostname=localhost where
>>> .' is used to reference sub structures.
>> We might want some syntactic sugar so that users don't have to repeat
>> format.qcow.protocol.nbd.FOO for every FOO they want to configure.
>
> -set will still function in this model as key/value parsing and
> translation to a C structure are two separate steps.
>
> But yeah, I think we ought to brain storm ways to simplify blockdev
> because the structure is very complex.  But that's a separate
> discussion.

Fair enough.

>>> Initial code is in my QAPI tree.
>>>
>>> I'm not going to start converting things until we get closer to the
>>> end of 0.15 and QAPI is fully merged.  My plan is to focus on this for
>>> 0.16 and do a full conversion for the 0.16 time frame using the same
>>> approach as QAPI.  That means that for 0.16, we would be able to set
>>> all command line options via QMP in a programmatic fashion with full
>>> support for introspection.
>>>
>>> I'm haven't yet closed on how to bridge this to qdev.  qdev is a big
>>> consumer of QemuOpts today.  I have some general ideas about what I'd
>>> like to do but so far, I haven't written anything down.
>> Glad you mention qdev.
>>
>> qdev properties describe the configurable parts of qdev state objects.
>> A state object is a C struct.  The description is C data.  Poor man's
>> reflection.
>
> Right.  The problem is it's very hard to reflect C even if you parse
> it without additional annotations.  For instance:
>
> typedef struct Foo {
>     Bar *bar;
> } Foo;
>
> What the type of bar is is ambigious.  It could be a pointer to a list
> of Bar's (if bar has an embedded pointer), it could be an array of
> Bars that is terminated using a field within Bar, it could be a
> pointer to a fixed size array of Bars, or it could be just a pointer
> to a single Bar object.
>
> So you end up needing additional annotations like:
>
> typedef struct Foo {
>    size_t n_bar;
>    Bar *bar sizeis(n_bar);
> } Foo;
>
> This is what most IDLs that use C style syntax do.

We currently use a more low-tech approach: define the struct in plain C,
and the data describing the struct in plain C as well.

Information about the type is in two places and in two formats (C type
declaration and C data intializer).  There's a bit of redundancy.
Ensuring consistency requires preprocessor hackery.

The data doesn't have to describe all struct members.  I'm inclined to
regard that as a feature.

Avoids generating source code.

>> qdev needs to parse and print a device's properties.  It uses QemuOpts
>> to parse NAME=VALUE,... option strings into a list of (NAME, VALUE), and
>> callbacks to parse the VALUEs.  Awkward, especially when you go QMP ->
>> option string ->  qdev struct.
>
> Indeed.  You can do very weird things like pass a float through qdev
> and it appears as a (valid) string type with QemuOpts.
>
> But qdev properties are a little odder semantically.  qdev properties
> are construct-only and read-only.   They are construct-only because
> they are set implicitly in the object before its created as a sort of
> default value.
>
> It's easy to bridge QCFG/QMP to this but I've been thinking of going a
> step further.  I've been thinking of taking the current qdev
> properties and making them proper construction properties and removing
> the magic involved in setting their default value.  This would require
> the code generator to create a construction function that is called
> such as:
>
> DeviceState *isa_serial_init(int iobase, int irq, CharDriverState *chr)
> {
>      ISADeviceState *dev = qdev_alloc(&isa_serial_desc);
>
>      dev->iobase = iobase;
>      dev->irq = irq;
>      dev->chr = chr;
>      // ...
> }
>
> There would be a separate interface for getting/setting properties.
> It might even be something as simple as, you have to implement:
>
> int isa_serial_get_iobase(ISADeviceState *dev);
> int isa_serial_get_irq(ISADeviceState *dev);
> ...

To be honest, I'm wary of generating lots of C source.  Prone to become
a pain to debug and change.  Generated code tends to be much worse than
data in that regard.

If you generate significant parts of your program with other programs,
chances are that you're not using the right language for the job.

> This ends up being a powerful interface because you can easily get
> these properties within QEMU, but you also (transparently) map these
> to the wire.  It also extends really nicely for setting properties
> which becomes an interesting way to implement dynamic device logic
> (like setting the link status of a network card).

Well, the traditional way to "get properties" within QEMU is the ol' ->
operator.

I'm not sure I get the rest of the paragraph.

>> Yet another one: vmstate, which describes migratable parts of qdev state
>> objects.
>
> Yes, for now, I'm ignoring vmstate.  To really solve vmstate, I think
> you need to describe the complete object instead of just it's
> properties.

Would be nice to have a solution that can cover all our reflection
needs, including vmstate.  But we need to watch out not to get bogged
down in the Generality Swamps.

>> Unlike these two, QCFG doesn't describe C structs, it generates them
>> from JSON specs.  If I understand your proposal correctly.  Hmm.
>
> Correct.
>
>> Can we avoid defining our data types in JSON rather than C?
>
> I didn't start with describing them in JSON.  I started by describing
> them in Python with the idea that I'd write a IDL parser using a C
> style syntax that would then generate the Python structures.  Using
> the fixed Python data structures, I could play with code generation
> without all the trouble of writing a full blown IDL parser.
>
> But I never got around to writing that parser and have actually become
> fond of the Python/JSON style syntax.

Well, let's say "fondness" isn't what I'd expect ordinary people to feel
when confonted with the idea to define half our structs in Python rather
than C ;)

>> Can we adopt *one* reflection mechanism?
>
> Yes, I think we should start with that being the JSON syntax I'm using
> for QAPI/QCFG and then down the road, we can introduce a totally new
> syntax if need be.  But I think we need a central schema that
> describes all externally visible interfaces.  I think this is really
> the key idea here.

Yes, self-describing external interfaces are desirable.

Speaking of external interfaces: we need to be careful when exposing
internal data types externally.  If we couple our external interfaces
too closely to internals, we risk calcifying our internal interfaces.

qdev has gotten that partly right: you can change the state structs
pretty freely without affecting the externally visible properties.

[...]