Re: [Qemu-devel] [RFC] QMP: add query-hotpluggable-cpus

[Markus Armbruster]

David Gibson <address@hidden> writes:

> On Fri, Feb 19, 2016 at 10:51:11AM +0100, Markus Armbruster wrote:
>> David Gibson <address@hidden> writes:
>> 
>> > On Thu, Feb 18, 2016 at 11:37:39AM +0100, Igor Mammedov wrote:
>> >> On Thu, 18 Feb 2016 14:39:52 +1100
>> >> David Gibson <address@hidden> wrote:
>> >> 
>> >> > On Tue, Feb 16, 2016 at 11:36:55AM +0100, Igor Mammedov wrote:
>> >> > > On Mon, 15 Feb 2016 20:43:41 +0100
>> >> > > Markus Armbruster <address@hidden> wrote:
>> >> > >   
>> >> > > > Igor Mammedov <address@hidden> writes:
>> >> > > >   
>> >> > > > > it will allow mgmt to query present and possible to hotplug CPUs
>> >> > > > > it is required from a target platform that wish to support
>> >> > > > > command to set board specific MachineClass.possible_cpus() hook,
>> >> > > > > which will return a list of possible CPUs with options
>> >> > > > > that would be needed for hotplugging possible CPUs.
>> >> > > > >
>> >> > > > > For RFC there are:
>> >> > > > >    'arch_id': 'int' - mandatory unique CPU number,
>> >> > > > >                       for x86 it's APIC ID for ARM it's MPIDR
>> >> > > > >    'type': 'str' - CPU object type for usage with device_add
>> >> > > > >
>> >> > > > > and a set of optional fields that would allows mgmt tools
>> >> > > > > to know at what granularity and where a new CPU could be
>> >> > > > > hotplugged;
>> >> > > > > [node],[socket],[core],[thread]
>> >> > > > > Hopefully that should cover needs for CPU hotplug porposes for
>> >> > > > > magor targets and we can extend structure in future adding
>> >> > > > > more fields if it will be needed.
>> >> > > > >
>> >> > > > > also for present CPUs there is a 'cpu_link' field which
>> >> > > > > would allow mgmt inspect whatever object/abstraction
>> >> > > > > the target platform considers as CPU object.
>> >> > > > >
>> >> > > > > For RFC purposes implements only for x86 target so far.    
>> >> > > > 
>> >> > > > Adding ad hoc queries as we go won't scale.  Could this be solved 
>> >> > > > by a
>> >> > > > generic introspection interface?  
>> >> > > Do you mean generic QOM introspection?
>> >> > > 
>> >> > > Using QOM we could have '/cpus' container and create QOM links
>> >> > > for exiting (populated links) and possible (empty links) CPUs.
>> >> > > However in that case link's name will need have a special format
>> >> > > that will convey an information necessary for mgmt to hotplug
>> >> > > a CPU object, at least:
>> >> > >   - where: [node],[socket],[core],[thread] options
>> >> > >   - optionally what CPU object to use with device_add command  
>> >> > 
>> >> > Hmm.. is it not enough to follow the link and get the topology
>> >> > information by examining the target?
>> >> One can't follow a link if it's an empty one, hence
>> >> CPU placement information should be provided somehow,
>> >> either:
>> >
>> > Ah, right, so the issue is determining the socket/core/thread
>> > addresses that cpus which aren't yet present will have.
>> >
>> >>  * by precreating cpu-package objects with properties that
>> >>    would describe it /could be inspected via OQM/
>> >
>> > So, we could do this, but I think the natural way would be to have the
>> > information for each potential thread in the package.  Just putting
>> > say "core number" in the package itself assumes more than I'd like
>> > about how packages sit in the heirarchy.  Plus, it means that
>> > management has a bunch of cases to deal with: package has all the
>> > information, package has just a core id, package has just a socket id,
>> > and so forth.
>> >
>> > It is a but clunky that when the package is plugged, this information
>> > will have to sit parallel to the array of actual thread links.
>> >
>> > Markus or Andreas is there a natural way to present a list of (node,
>> > socket, core, thread) tuples in the package object?  Preferably
>> > without having to create a whole bunch of "potential thread" objects
>> > just for the purpose.
>> 
>> I'm just a dabbler when it comes to QOM, but I can try.
>> 
>> I view a concrete cpu-package device (subtype of the abstract
>> cpu-package device) as a composite device containing stuff like actual
>> cores.
>
> So.. the idea is it's a bit more abstract than that.  My intention is
> that the package lists - in some manner - each of the threads
> (i.e. vcpus) it contains / can contain.  Depending on the platform it
> *might* also have internal structure such as cores / sockets, but it
> doesn't have to.  Either way, the contained threads will be listed in
> a common way, as a flat array.
>
>> To create a composite device, you start with the outer shell, then plug
>> in components one by one.  Components can be nested arbitrarily deep.
>> 
>> Perhaps you can define the concrete cpu-package shell in a way that lets
>> you query what you need to know from a mere shell (no components
>> plugged).
>
> Right.. that's exactly what I'm suggesting, but I don't know enough
> about the presentation of basic data in QOM to know quite how to
> accomplish it.
>
>> >> or
>> >>  * via QMP/HMP command that would provide the same information
>> >>    only without need to precreate anything. The only difference
>> >>    is that it allows to use -device/device_add for new CPUs.
>> >
>> > I'd be ok with that option as well.  I'd be thinking it would be
>> > implemented via a class method on the package object which returns the
>> > addresses that its contained threads will have, whether or not they're
>> > present right now.  Does that make sense?
>> 
>> If you model CPU packages as composite cpu-package devices, then you
>> should be able to plug and unplug these with device_add, unless plugging
>> them requires complex wiring that can't be done in qdev / device_add,
>> yet.
>
> There's a whole bunch of issues raised by allowing device_add of
> cpus.  Although they're certainly interesting and probably useful, I'd
> really like to punt on them for the time being, so we can get some
> sort of cpu hotplug working on Power (and s390 and others).

If you make it a device, you can still set
cannot_instantiate_with_device_add_yet to disable -device / device_add
for now, and unset it later, when you're ready for it.

> The idea of the cpu packages is that - at least for now - the user
> can't control their contents apart from the single "present" bit.
> They already know what they can contain.

Composite devices commonly do.  They're not general containers.

The "present" bit sounds like you propose to "pre-plug" all the possible
CPU packages, and thus reduce CPU hot plug/unplug to enabling/disabling
pre-plugged CPU packages.

What if a board can take different kinds of CPU packages?  Do we
pre-plug all combinations?  Then some combinations are non-sensical.
How would we reject them?

For instance, PC machines support a wide range of CPUs in various
arrangements, but you generally need to use a single kind of CPU, and
the kind of CPU restricts the possible arrangements.  How would you
model that?

> There are a bunch of potential use cases this doesn't address, but I
> think it *does* address a useful subset of currently interesting
> cases, without precluding more flexible extensions in future.
>
>> If that's the case, a general solution for "device needs complex wiring"
>> would be more useful than a one-off for CPU packages.
>> 
>> [...]
>>