[Qemu-devel] vCPU hotplug roadmap (was: Minutes for KVM call 2013-01-15)

[Andreas Färber]

Am 15.01.2013 17:16, schrieb Juan Quintela:
> 
> * cpu hot plug
>   - use qdev propierties conected to a set of socket objects (anthony)
>   - cpusets are the wrong interface (anthony)
>   - make a link between cpu <-> socket instead of a propierty?
>   - how far are we from being able to describe a cpu with -device?
>     (didn't heare the answer, andreas?)
>   - perhaps the best approach?
>   - After soft-freeze, exceptions depend on the maintainer
>   - After hard-freeze, no exceptions
>   -device don't require a bus, just an implementation detail, we can change 
> that
>   - use cpuset as an intermediate step until full vision is implemented
>   - several approaches from where we are now, to have something before
>     we get a full solution
> 
> 
> At this point, Andreas agreed to write a better summary of the
> discussion and suggestions O:-)

Got buried, here we go:

== vCPU hot-plug user interfaces ==

=== cpu_set ===

Previously available in qemu-kvm.git:
`cpu_set <n+1> online` via HMP

Pros:
* Hides QOM/qdev implementation details (afaerber)
* Thus: Doesn't depend on QOM CPUState refactoring (imammedo)
* Opens a fast route to implementing vCPU unplug in KVM (imammedo)
* Unintrusive to add and easy to obsolete/remove in future (imammedo)
* Existing virt-test cases (afaerber)
* Supported by libvirt (imammedo)
* Prevents confusing guests by hot-plugging random mix of CPUs (agraf)

Cons:
* Cannot express topologies (ehabkost)

=== device_add ===

`device_add driver=Haswell-x86_64-cpu id=qdevid`
[You can try this today and see it failing / not working.]

Pros:
* QMP/HMP command available today and known to users (afaerber)
* Unified command for device and CPU hot-plug (imammedo)
* Would allow first doing thread-level vCPU hotplug (imammedo)
* Could be extended to support socket-level hot-plug (aliguori/imammedo)

Cons:
* Operates on raw QOM type name unlike -cpu (afaerber)
* Needs support in libvirt for device_add driver=<CPU> (imammedo)
* libvirt needs means to enumerate CPU types (imammedo) => QMP? (AF)

Challenges:
* No CPU qbus (afaerber)
  => "should work" without (aliguori)
* CPU subclasses needed for identifying type name (afaerber/imammedo)
  => "Haswell-x86_64-cpu" does not exist yet, just "x86_64-cpu"
* CPU class_init for -cpu host requires KVM init (imammedo)
  [suggestion by ehabkost to use kvm_arch_vcpu_init, WIP by afaerber]
* Conversion of CPU features to static properties needed (imammedo)
  => device_add driver=foo,level=x,xlevel=y,...
* Alternatively conversion to global properties (imammedo)
* Cements type names - rename for 1.4? (afaerber) => permissable (alig.)
  [patches for arm, m68k, openrisc, unicore32 on list]

=== qom-set ===

`qom-set` via QMP w/ link<CPUSocket> property (aliguori)

Topology represented in QOM:
CPUSocket has-a    CPUCore has-a    CPUThread a.k.a. CPUState, or
CPUSocket links-to CPUCore links-to CPUThread a.k.a. CPUState

Challenges (afaerber):
* No CPUSocket/CPUCore objects yet and may take a while to get there...
  topology fields being moved to CPUState for 1.4 [done, more WIP]
* No decisions on canonical paths for CPUs: CPU? machine? unassigned?
* Duality of thread-level device types and socket-level? (afaerber)
  => fine to have, e.g., quad-core Xeon 500 device (aliguori)
* CPUState is no_user (afaerber)
  => need to generally drop no_user for QOM (aliguori)

=== libvirt ===

libvirt's XML topology modelling is closer to today's -smp than to the
desired QOM modelling:
http://www.libvirt.org/formatcaps.html

`virsh setvcpus <domain> <n>`
http://libvirt.org/sources/virshcmdref/html/sect-setvcpus.html

== qom-cpu course of action (afaerber) ==

It was requested to have vCPU hot-plug in v1.5.

For device_add we need to move code from cpu_init() into QOM facilities.
=> QOM realize support would help [applied by aliguori]
=> cleanups piggy-backed onto CPU realizefn [applied to qom-cpu-next]

Agreement on goal of X86CPU subclasses, but conflicts how to get there:
* Refactor x86_def_t to X86CPUInfo for X86CPUClass class_init? (AF 2012)
* Refactor x86_def_t to X86CPU instance_init as done for arm?
* Refactor x86_def_t to class_inits? (afaerber)
  -> heavy merge conflicts due to bug fixes / cleanups
  Pro: We can get things into a consistent QOM'ish state across targets.
  Con: We will refactor again on top for machine-compat properties.
* Keep x86_def_t within X86CPUClass as done for ppc? (WIP: afaerber)
  => smallest common denominator, separates x86 from cross-target work

APIC ID topology fixes are being reviewed for 1.4. [merged]
X86CPU wave 4 cleanups by Igor are being reviewed for 1.4. [merged]

Rename CPU types according to unified <name>-<arch>-cpu scheme for 1.4?
(aliguori: permissable) [patches on list]

VMState series by Juan being rebased - subset for 1.4, rest for 1.5.
[1.4 part on list, WIP for 1.5]

Remainder is considered 1.5 material, qom-cpu-next avail. during Freeze.

== Common issues (imammedo) ==

- back-port CPU hot-plug ACPI notification
- hot-plug is not allowed on SysBus:
  - APIC that is created by CPU on hot-plug is a SysBus device,
    so essentially APIC is hot-plugged to SysBus anyway which causes
    abort in qdev_try_create() -> qdev_set_parent_bus(SysBus) ->
    bus_add_child() [no hot-plug on SysBus], which is hard to fix
    without allowing hot-plug on SysBus.
    [AF: change the APIC to not be a SysBusDevice (like CPUState) then?]
  - qdev_device_add() and qdev_try_create() assign SysBus by default to
    device if no explicit bus was provided.
    - qbus_find_recursive() in qdev_device_add() always returns SysBus
      if last 2 args are NULL, which is case with busless CPU
    - qdev_try_create() adds device SysBus if no bus was provided in
      1st arg
- vapic is not working with hot-plug in current master
  (do not know why yet)

Experimental CPU hot-add tree using device_add implemented only for
qemu64 CPU:
https://github.com/imammedo/qemu/tree/x86-cpu-hot-add.WIP

---
That completes Igor's and my notes, amended by status updates of mine.
Sorry for the delay, I did try to address some of the discussed issues
in form of patches in the meantime though. ;)

CC'ing s390x, ppc and arm KVM folks that were not on the call for
feedback on their requirements and thoughts.

Regards,
Andreas

-- 
SUSE LINUX Products GmbH, Maxfeldstr. 5, 90409 Nürnberg, Germany
GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer; HRB 16746 AG Nürnberg