Re: [Qemu-devel] [Query] Live Migration between machines with different processor ids

[Andrew Jones]

On Tue, Sep 04, 2018 at 09:16:58AM +0000, Jaggi, Manish wrote:
> > On 31-Aug-2018, at 4:41 PM, Andrew Jones <address@hidden> wrote:
> > I think the sequence should look something like this:
> > 
> >  1) Guest running on Host A with processor a
> >  2) Stop guest and save its state for migration
> >  3) Migrate guest to Host B with processor b (b is "close enough" to a)
> >  4) Restore guest state after migration
> >     If guest is running with '-cpu host'
> >       4.a) Inform KVM of any configuration that impacts invariant registers
> >       4.b) Update the guest's view of all invariant registers to match the
> >            host
> >     EndIf
> >  5) Run guest
> > 
> > 4.a and 4.b require new code both in QEMU and KVM. 4.a may require a new
> > KVM API, unless the existing API can be leveraged.
> > 
> > The definition of "close enough" is left to the users and/or higher layers
> > of the Virt stack.
> > 
> 
> Thanks for detailing the sequence. 
> I got another comment from Juan and David which is not to use -cpu host, see 
> below
> 
> "I really think that the right approach here is not using -cpu host.  You
> do the full work, create a model as David says, and be sure that you car
> run that model on both cpus.  It is a lot of work, but it is the only
> way to make sure that this is going to work long term.”
> 
> Not using -cpu host is orthogonal to the sequence we have been discussing in 
> this thread.
> Use something like -cpu cortex-a57 (this however does not work so far)
> This would avoid close-enough definition, but would need substantial work.
> 
> So which approach should be taken here, whats your take...
> 

Inventing a base-AArch64 cpu model that can then be extended with optional
features is a nice way to extend the migratability of a guest, however
it's hard to do because of errata. Since errata workarounds are enabled
per MIDR, then we'd need to invent our own MIDR and also some way to
communicate which errata we want to enable, possibly through some paravirt
mechanism or through some implementation defined system registers that
KVM would need to reserve and define.

That's not just a ton of work for the entire virt stack (not just KVM and
QEMU, but also all the layers above), but it's possible that it won't be
useful in the end anyway. There's risk that enabling just one erratum
workaround would restrict the guest to hosts of the exact same type
anyway. For each erratum that needs to be enabled, the probability of
enabling an incompatible one goes up, so it may not be likely to do much
better than '-cpu host' in the end. I'm afraid that until errata are
primarily showing up in optional CPU features that can simply be disabled
for the workaround, that we're stuck with '-cpu host'. I'd be happy to
discuss it more though.

In short, I'd go with the proposal above, for now, with possibly one
change. libvirt folk (Andrea Bolognani and Pino Toscano) suggest that
the guest invariant register updating on the destination host only be
done if the user opts-in to it. This is because right now if a user
tries to migrate to a host that is not 100% identical the migration
will fail, which makes the "mistake" clear. If we silently change the
behavior to allow it, then what could have been a mistake, because
the hosts aren't actually "close enough", may go unnoticed. I'm not
100% sure we need another user opt-in flag to be set, though, as I
think the '-cpu host' indicates the user expects the VCPU to look
like the host CPU, and even after migration that expectation should be
met. Simply, users that migrate '-cpu host' VMs need to know what they're
doing.

Thanks,
drew