Re: [Qemu-devel] [Query] Live Migration between machines with different processor ids

[Jaggi, Manish]

> On 31-Aug-2018, at 4:41 PM, Andrew Jones <address@hidden> wrote:
> 
> External Email
> 
> Manish,
> 
> Your mail reader doesn't appear to be providing any quoting, so I'm not
> sure I found all your replies. I've tried to fix up the quoting in order
> to reply here, but you should look into fixing your reader.
> 
Oh. I am using Mail on MacOSx, not sure what is wrong here...

> On Fri, Aug 31, 2018 at 09:52:12AM +0000, Jaggi, Manish wrote:
>> On bleh-bleh, Andrew Jones wrote:
>>> On bleh-bleh, Jaggi, Manish wrote:
>>>> - So in cpu_post_load (Machine B) qemu can lookup whitelist and replace 
>>>> the MIDR with the one at Machine B.
>>>> Sounds good?
>>>> 
>>> It shouldn't be necessary. With '-cpu host' QEMU should probably just read
>>> all the ID registers from the host first, updating the guest's copy to
>>> match the destination host's registers (we're using '-cpu host', the
>>> registers should match the host - including MIDR.) If a user chooses to
>>> migrate a guest that is using '-cpu host', then they need to know what
>>> they are doing. If a whitelist of close-enough processors is possible to
>>> create, then that whitelist should be managed and used at a higher layer
>>> in the virt stack, not down in QEMU.
>> 
>> How would qemu know to that it has to patch? Could not understand your point 
>> here.
>> IIUC qemu needs some parameter for this
> 
> QEMU would unconditionally update the guest's view of the invariant
> registers after migration, when '-cpu host' is used. There's no need
> for a parameter, as the update is unconditional.
> 
>> 
>>> For example, openstack can determine
>>> destination candidates using whatever policy it wants, including a close-
>>> enough processor whitelist.
>>> 
>>> So, I propose blindly updating all invariant registers when migrating
>>> a '-cpu host' guest and leaving it to the user to do these migrations
>>> at their own risk
>>> 
>> yes good point updating all invariant is better, for the case where user is 
>> aware that host and destination have same cpu arch.
>> I can prepare a RFC patch but cpu_post_load will need some flag to replace 
>> these values.,
> 
> I'm not sure what you mean by a flag, but I'll review the RFC when you
> post it.
> 
>>> 
>>> (when migrating to a truly identical host, the blind
>>> update will not change anything. So it would be no worse than what we
>>> do today.) One side note is that we're starting to give QEMU control
>>> over what optional processor features are available to the guest, e.g.
>>> SVE. So before blindly updating all ID registers we'd want to inform
>>> KVM of the guest configuration in order for KVM to return appropriate
>>> ID register values.
>>> 
>> Not sure how to handle this.
> 
> I think the sequence should look something like this:
> 
>  1) Guest running on Host A with processor a
>  2) Stop guest and save its state for migration
>  3) Migrate guest to Host B with processor b (b is "close enough" to a)
>  4) Restore guest state after migration
>     If guest is running with '-cpu host'
>       4.a) Inform KVM of any configuration that impacts invariant registers
>       4.b) Update the guest's view of all invariant registers to match the
>            host
>     EndIf
>  5) Run guest
> 
> 4.a and 4.b require new code both in QEMU and KVM. 4.a may require a new
> KVM API, unless the existing API can be leveraged.
> 
> The definition of "close enough" is left to the users and/or higher layers
> of the Virt stack.
> 

Thanks for detailing the sequence. 
I got another comment from Juan and David which is not to use -cpu host, see 
below

"I really think that the right approach here is not using -cpu host.  You
do the full work, create a model as David says, and be sure that you car
run that model on both cpus.  It is a lot of work, but it is the only
way to make sure that this is going to work long term.”

Not using -cpu host is orthogonal to the sequence we have been discussing in 
this thread.
Use something like -cpu cortex-a57 (this however does not work so far)
This would avoid close-enough definition, but would need substantial work.

So which approach should be taken here, whats your take...

> Thanks,
> drew