Re: Making QEMU easier for management tools and applications

[John Snow]

On 1/24/20 2:59 AM, Markus Armbruster wrote:
> Digression: JSON is a poor choice for configuration files.  YAML is a
> complex and confusing beast (it's spec is printed 116 pages, and
> yaml-0.2.2/src is ~7kSLOC).  XML is XML, 'nuff said.  TOML is much
> simpler than either of the two.
JSON has a built-in Python parser. YAML has a widely availably PyYAML
parser module. TOML is not yet in widespread use, but does offer a
python parser.

No XML parser I have ever used appears to be feature-complete to the XML
spec. Most modern toolings support JSON and/or YAML.

I personally like the idea of YAML as it was intended for human-written
configuration files without brackets and other such visual confetti. It
even allows for comments as a first-class citizen of the specification.
This would be useful to documentation efforts.

Hearsay: most of the existing high-level users of the virt stack are in
Go or Python. Python users are quite comfortable with whitespace-markup
formats. Go developers are likely working in the cloud space, which
overwhelmingly uses YAML.


Suggestion:

(1) We use YAML and use Python's built-in YAML parser to build a machine
specification format defined by QAPI.

We build a "configuration" struct in QAPI, and extend from there.

(2) We offer "--config myconfig.yaml" as an option for specifying
options. This precludes the use of *any* traditional command line flags.
Security wonkery permitting, we may also allow QEMU to take a YAML file
on STDIN at boot, if this is easier for cloud management programs to script.

(3) We offer "--writeconfig out.yaml" to convert to the new format... if
possible. It very well may NOT be, given the nature of our wild west
parsing system. I consider this an optional olive branch, not a necessity.

(4) Just scrap the existing config file system entirely. It was never
finished and does not help solve the existing problem that we do not
have a schema for our configuration syntax.

(5) Deprecate the traditional CLI configuration syntax. Remove it only
on a major version transition. Advertise this as our "2.0" break. Use
this opportunity to do heavier refactoring of our worst design
transgressions as desired. Make no effort to add additional features to
the CLI if it doesn't come for free.

(Bonus: If we work with libvirt to adapt to the new format, libvirt
users will enjoy a seamless transition. This is an insanely good
marketing point for libvirt's usefulness, allowing QEMU to adapt as
necessary to remove baggage with minimal interruption to end-users.)

(Downside: Any libvirt XML passing CLI flags directly will lose that
ability. They will need to specify YAML object overrides that plug into
specific parts of the hierarchy instead.)

(Upside to the downside: These overrides are often not supported by
downstream vendors anyway.)


Benefits:

- The entirety of the ugly parsing code in vl.c and strewn about
submodules goes away. Many of the .hx files go away.

- QemuOpts goes away. Everything is handled at the QAPI level via YAML
and QMP parsers. Input errors are handled at the schema level.

- Extending QMP and the CLI occurs with one schema definition change.

- We don't have to write a new python parser; The complexity of YAML
itself poses no particular concern here.

- YAML is in wide use. Support in emacs/vim is widespread. Third party
tools for manipulating YAML documents are widely available and in-use
due to usage in the cloud infrastructure world.

- Once deserialized, JSON and YAML configuration statements are
identical in-memory in Python. Many of the same tools and infrastructure
can be recycled.

- PyYaml uses libYAML, a C library. That same library can be used to
build the same structures that we already use for QMP.

- YAML examples can be trivially generated from the QAPI spec.

- YAML allows for in-line comments, which would be a tremendous feature
for writing documentation and annotating machine configuration files.


Downsides:

- We WOULD need a new YAML parsing layer in QEMU, generated by QAPI.
Supporting two deserialization layers could lead to strange
discrepancies between the two formats at runtime. So, admittedly, using
JSON would be *even easier*, as we could re-use the same parsers already
battle-tested in QEMU. We could get *MAXIMUM* code re-use this way.

- YAML does not allow duplicate keys. On its face, this is not an issue,
we can't support parallel universe configurations anyway. If it's
important to allow "overrides" of base configurations, this might intuit
the need for cascading YAML files. (--config base.yaml --config
overlay.yaml)

- We will break compatibility with our existing CLI. People will not be
happy about this, especially, perhaps, embedded board and TCG developers
who use fairly minimal command lines regularly.

- A lot of iotests and python scripts will need serious love to bring
forward into the new world.



I guess at the end of writing this out, I wind up making a slightly
better case for JSON in terms of implementation difficulty, but still
have a personal bias and fondness for YAML, in part for the in-line
comments and in part for bringing us closer to cloud projects which use
YAML heavily. Not sure if it's worth the pain in the end, but I haven't
given up hope yet.


Some things I think are a given:

- We want a config file. We don't know what format it should be in, but
it should be defined by the QAPI schema in some way. Declarative and
data driven is the most beautiful of all programming paradigms.

- Our existing CLI syntax is large, unknowable, and poorly documented.
The work it would take to turn it into a schema-defined language is
likely as-complex as it would be to develop a new format from first
principles anyway.

- Due to the ad-hoc nature of how each top-level CLI flag/option parses
its own argument data, making the CLI regular is going to involve a lot
of broken backwards compatibility and unhappy users anyway.

- So we may as well go for broke and make one hell of a config file
format, whatever language it ends up being in.


--js