Re: [Qemu-devel] CPU hotplug

[Eduardo Habkost]

On Mon, Feb 01, 2016 at 04:35:17PM +1100, David Gibson wrote:
> Hi,
> 
> It seems to me we're getting rather bogged down in how to proceed with
> an improved CPU hotplug (and hot unplug) interface, both generically
> and for ppc in particular.
> 
> So here's a somewhat more concrete suggestion of a way forward, to see
> if we can get some consensus.
> 
> The biggest difficulty I think we're grappling with is that device-add
> is actually *not* a great interface to cpu hotplug.  Or rather, it's
> not great as the _only_ interface: in order to represent the many
> different constraints on how cpus can be plugged on various platforms,
> it's natural to use a heirarchy of cpu core / socket / package types
> specific to the specific platform or real-world cpu package being
> modeled.  However, for the normal case of a regular homogenous (and at
> least slightly para-virtualized) server, that interface is nasty for
> management layers because they have to know the right type to
> instantiate.
> 
> To address this, I'm proposing this two layer interface:
> 
> Layer 1: Low-level, device-add based
> 
>     * a new, generic cpu-package QOM type represents a group of 1 or
>       more cpu threads which can be hotplugged as a unit
>     * cpu-package is abstract and can't be instantiated directly
>     * archs and/or individual platforms have specific subtypes of
>       cpu-package which can be instantiated
>     * for platforms attempting to be faithful representations of real
>       hardware these subtypes would match the specific characteristics
>       of the real hardware devices.  In addition to the cpu threads,
>       they may have other on chip devices as sub-objects.
>     * for platforms which are paravirtual - or which have existing
>       firmware abstractions for cpu cores/sockets/packages/whatever -
>       these could be more abstract, but would still be tied to that
>       platform's constraints
>     * Depending on the platform the cpu-package object could have
>       further internal structure (e.g. a package object representing a
>       socket contains package objects representing each core, which in
>       turn contain cpu objects for each thread)
>         * Some crazy platform that has multiple daughterboards each with
>           several multi-chip-modules each with several chips, each
>         with several cores each with several threads could represent
>         that too.

What exactly in this approach makes it depend on device-add? We
could have something very similar based on creation of QOM
objects, for example.

> 
> What would be common to all the cpu-package subtypes is:
>     * A boolean "present" attribute ("realized" might already be
>       suitable, but I'm not certain)

"realized" might be suitable, but I am not even sure we want
cpu-package to be a TYPE_DEVICE subclass. It could be a simple
QOM class or even a QOM interface (machines could choose to
implement it as TYPE_DEVICE, or not).

>     * A generic means of determining the number of cpu threads in the
>       package, and enumerating those

This could be based on QOM links.

>     * A generic means of determining if the package is hotpluggable or
>       not

Isn't this a machine attribute, instead of a package attribute?

>     * They'd get listed in a standard place in the QOM tree

If we allow CPU thread enumeration and package enumeration be
based in QOM links, we can let machines implement those
interfaces without introducing QOM hierarchy requirements.

We have one example where we would need to make this flexible
enough about QOM hierarchy, below (thread-based hotplug in x86).

> 
> This interface is suitable if you want complete control over
> constructing the system, including weird cases like heterogeneous
> machines (either totally different cpu types, or just different
> numbers of threads in different packages).
> 
> The intention is that these objects would never look at the global cpu
> type or sockets/cores/threads numbers.  The next level up would
> instead configure the packages to match those for the common case.
> 
> Layer 2: Higher-level
> 
>     * not all machine types need support this model, but I'd expect
>       all future versions of machine types designed for production use
>       to do so
>     * machine types don't construct cpu objects directly
>     * instead they create enough cpu-package objects - of a subtype
>       suitable for this machine - to provide maxcpus threads
>     * the machine type would set the "present" bit on enough of the
>       cpu packages to provide the base number of cpu threads

Sounds interesting, and very simple for management code. What I
don't see is: what exactly makes it easier to implement just
Layer 1 and not Layer 2?

Implementing Layer 1 looks more difficult to me, because it
requires supporting creation of cpu-package objects on the fly,
using device_add (or whatever mechanism we choose for cpu-package
creation). Layer 2 lets the implementation choose freely when/how
exactly the other objects will be created and how exactly they
will appear in the device tree. They just need to do the right
thing when the "present" property is flipped.

> 
> Management layers can then manage hotplug without knowing platform
> specifics by using qmp to toggle the "present" bit on packages.
> Platforms that allow thread-level pluggability can expose a package
> for every thread, those that allow core-level expose a package per
> core, those that have even less granularity expose a package at
> whatever grouping they can do hotplug on.
> 
> Examples:
> 
> For use with pc (or q35 or whatever) machine type, I'd expect a
> cpu-package subtype called, say "acpi-thread" which represents a
> single thread in the ACPI sense.  Toggling those would trigger ACPI
> hotplug events as cpu_add does now.

You have a good point here: I remember seeing suggestions of
making CPU hotplug tied to the socket/core/thread hierarchy
somehow. But this won't change the fact that x86 allows hotplug
of individual CPU threads.

In other words, if we make a /machine/socket/core/thread QOM
hierarchy, the cpu-packages for x86 won't necessarily represent
CPU sockets (but in other architectures, they might).  The
interface needs to be generic enough to not assume anything about
the CPU topology level where CPU hotplug happens.

> 
> For use with pseries, I'd expect a "papr-core" cpu-package subtype,
> which represents a single (paravirtual) core.  Toggling present on
> this would trigger the PAPR hotplug events.  A property would control
> the number of threads in the core (only settable before enabling
> present).
> 
> For use with the powernv machine type (once ready for merge) I'd
> expect "POWER8-package" type which represents a POWER8 chip / module
> as close to the real hardware as we can get.  It would have a fixed
> number of cores and threads within it as per the real hardware, and
> would also include xscoms and other per-module logic.
> 
> From here to there:
> 
> A suggested order of implementation to get there without too much risk
> of breaking things.
> 
>   1. Fix bugs with creation / removal of CPU objects (Bharata's cpu
>      hotplug series already has this)
>   2. Split creation and realization of CPU objects, so machine types
>      must explicitly perform both steps (Bharata's series has this
>      too)
>   3. Add the abstract cpu-package type, and define the generic
>      interfaces it needs (Bharata's series has something that could be
>      changed to this fairly easily)
>   4. For each machine type we care to convert:
>       4.1. Add platform suitable cpu-package subtypes
>       4.2. Convert the (latest version) machine type to instantiate packages 
> instead of
>            cpu threads directly

Machines could even have the freedom to instantiate CPU threads
directly and then set up package objects for them. Reusing
generic code is useful, but it doesn't even need to be mandatory,
as long as the objects are available at the right place in the
QOM hierarchy.

>       4.3. Add any necessary backwards compat goo
>   5. Teach libvirt how to toggle cpu-packages

This is different from the very flexible QOM object
building/linking approach Andreas was talking about in last KVM
Forum. But while I would love to have the ability to build
arbitrary QOM hierarchies with complex links between CPUs
sockets, cores, threads, etc, I believe we need an interface that
is: 1) generic enough for multiple architectures and machines to
implement them; 2) simple enough so that libvirt can use it
easily without requiring more arch-specific code.

Also, your approach doesn't prevent the simple cpu-package
interface from having a complex QOM hierarchy hidden behind it.

-- 
Eduardo