Re: Redesign of QEMU startup & initial configuration

[Mark Burton]

> On 16 Dec 2021, at 16:40, Daniel P. Berrangé <berrange@redhat.com> wrote:
> 
> On Thu, Dec 16, 2021 at 04:28:29PM +0100, Paolo Bonzini wrote:
>> On 12/16/21 11:24, Markus Armbruster wrote:
>>>> Not really, in particular the startup has been mostly reworked already
>>>> and I disagree that it is messy.  softmmu/vl.c is messy, sure: it has
>>>> N different command line parser for command line options, magic
>>>> related to default devices, and complicated ordering of -object
>>>> creation.
>>>> 
>>>> But the building of emulator data structures is not messy; only the
>>>> code that transforms the user's instructions into startup commands.
>>>> The messy parts are almost entirely contained within softmmu/vl.c.
>>> 
>>> In my opinion, the worst mess is the reordering and the (commonly
>>> unstated, sometimes unknown) dependencies that govern it.
>>> The reordering is part of the stable interface.  Its finer points
>>> basically nobody understands, at least not without staring at the code.
>> 
>> Then we agree, because that's what I meant by "the messy parts are almost
>> entirely contained within softmmu/vl.c".
>> 
>>>> The one (and important, but fixable) exception is backends for
>>>> on-board devices: serial_hd, drive_get, and nd_table.
>>> 
>>> Practical ideas on fixing it?
>> 
>> What you did with pflash, turned up to 11.
>> 
>>>>> * A new binary sidesteps the need to manage incompatible change.
>>>> 
>>>> More precisely, a new binary sidesteps the need to integrate an
>>>> existing mechanism with a new transport, and deal with the
>>>> incompatibilities that arise.
>>> 
>>> I'm not sure I got this.
>> 
>> Configuring the VM part in CLI and part in QMP is not something anybody
>> needs nor should desire.  A new binary can use the CLI only for things that
>> really have to be done before QMP is up, for example the configuration of
>> sandboxing or tracing; and even that is only nice-to-have and not absolutely
>> necessary.
> 
> I wouldn't special case sandboxing at least. It should be something that
> can be activated via a QMP command too. That way you can blockdev-add
> a bunch of disks and other backends while you are still relatively
> unconfined, and then lock it down with a sandbox before starting vCPUs.
> 
> Or you can perhaps start with a coarse sandbox and then switch to
> a stronger sandbox part way through config (though can't remember
> offhand if that's possible with seccomp, or whether the first
> seccomp profile applied, prevents you adding stronger ones later).
> 
> Anyway sandboxing doesn't need to be active before QMP IMHO, because
> our primary goal with it is to mitigate guest exploits from untrusted
> code - the initial startup is largely trustworthy. Starting guest CPUs,
> or reading VMState is where it becomes dangerous generally.
> 
> I think you can probably argue that even tracing doesn't hugely
> need special casing if you can get QMP started early enough that
> there's little else that can go wrong before you get a chance to
> send a QMP 'trace' command.
> 
>> 
>>>> The problem is that CLI and HMP, being targeted to humans (and as you
>>>> say below humans matter), are not necessarily trivial transports.  If
>>>> people find the trivial transport unusable, we will not be able to
>>>> retire the old CLI.
>>> 
>>> Yes, a trivial CLI transport alone may not suffice to retire the old
>>> CLI.  By itself, that doesn't mean trivial transports must be avoided.
>>> 
>>> Do I have to argue the benefits of a trivial configuration file
>>> transport?
>> 
>> I think you do; I'm not sure that a trivial configuration file transport is
>> useful.  It's a middle ground in sophistication that I'm not sure would
>> serve many people.  The only source of bug reports for -readconfig has been
>> lxd, and I think we agree that they would be served better by QMP.
>> 
>>> Do I have to argue the benefits of a trivial CLI transport for use by
>>> relatively unsophisticated programs / relatively sophisticated humans
>>> (such as developers)? Can we agree these benefits are not zero?
>> 
>> We can.  But again I think you're misunderstanding the pain that the
>> existing CLI inflicts on users.  Most users do not find the ordering
>> complicated in practice; they do not even know that the issue exists. The
>> problem that users have is the 100+ character command lines, and that can be
>> fixed in two ways:
>> 
>> - with a trivial configuration file transport
>> 
>> - with a management tool such as virt-manager or virsh.
>> 
>> And I maintain that most users would be better served by the latter.  In
>> fact, I constantly wonder how much we're overestimating the amount of people
>> that are using QEMU.  In every post (Reddit, HN, wherever) that mentions
>> QEMU being too complex and not having a front-end like VirtualBox, there's
>> always someone that mentions virt-manager.

I totally agree with Paolo of course - thats what I was saying before. You can 
call “libvirt” somebody else’s problem if you wish, but it seems to me a more 
sensible route….

> 
> An important thing to note is that while libvirt is theoretically
> general purpose for any aspect of QEMU, practically speaking our
> coverage of QEMU features is very much skewed to virtualization
> use cases. The emulation use cases don't get anywher near as much
> love & attention, especially for architectures lacking KVM, or for
> machine types not used with KVM.

Totally agree on this (of course).

Thats why I’m here - I care about the people who care about emulation  :-)

In general, what we are working on is exactly the ability to service the 
‘complex’ emulation use case. No CLI, nor single ‘config’ file will be good 
enough, in all cases we will need to drive directly into QMP.

Cheers
Mark.



> 
> Also although libvirt tries to focus on just exposing a mechanism
> and not imposing a policy there are some aspects where we don't
> achieve that due to our architecture design. For example, if you
> want QEMU connected to stdio that's not possible with libvirt
> directly. There's always a level of indirection there because
> we unconditionally daemonize QEMU.
> 
>>>> Bad CLI is also very hard to deprecate because, unlike QMP (for which
>>>> you can delegate the workarounds to Libvirt & friends) and HMP (for
>>>> which people can just learn the new thing and type it), it is baked in
>>>> countless scripts.  People hate it when scripts break.
>>> 
>>> I assure you that bad QMP is plenty hard to deprecate, even when libvirt
>>> can be updated to cope.
>> 
>> Right, and CLI is worse. :)
>> 
>>>> The issues with the CLI then can be completely self-contained within
>>>> softmmu/vl.c, and will not influence the innards of QEMU.
>>> 
>>> The issues with the CLI will still influence its users. Can we
>>> agree that human users deserve something better than the current
>>> CLI?
>> 
>> Deserve, yes.  Need, not sure.  Do you agree that a lot of clients of QEMU
>> would be better served by Libvirt (programs) and virt-manager (humans)?
>> 
>> So, if I have to choose between better QMP now and better CLI now, I choose
>> better QMP now.  Exactly to avoid the "somebody could" trap and focus on
>> something that "we can" do now.
>> 
>>> I think we can learn from our experience with HMP/QMP.
>>> 
>>> Good:
>>> 
>>> * Separate interfaces for machines and for humans
>>> * Layering the human interface on top of the machine interface: HMP
>>>   commands implemented on top of QMP's internal C interface.
>> 
>> Agreed.  CLI should likewise be implemented on top of QMP's internal C
>> interface, the same way non-preconfig mode concludes startup with
>> qmp_x_exit_preconfig(NULL).  Second choice (inferior but sometimes
>> acceptable): implement it on top of the same QOM interfaces as the QOM
>> commands.
>> 
>>> * The stable interface is clear: QMP unless explicitly marked unstable.
>> 
>> Agreed, the problem here is that CLI is harder to evolve.
>> 
>> Paolo
>> 
> 
> Regards,
> Daniel
> -- 
> |: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
> |: https://libvirt.org         -o-            https://fstop138.berrange.com :|
> |: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|
>