Re: [Qemu-devel] [Query] Live Migration between machines with different processor ids

[Juan Quintela]

"Dr. David Alan Gilbert" <address@hidden> wrote:
> * Jaggi, Manish (address@hidden) wrote:
>
>> Just to add what happens in ARM64 case, qemu running on Machine A sends cpu 
>> state information to Machine B.
>> This state contains MIDR value, and so Processor ID value is compared in KVM 
>> and not in qemu (correcting myself).
>> 
>> IIRC, Peter/Eric please point if there is something incorrect in the below 
>> flow...
>> 
>> (Machine B)
>> target/arm/machine.c: cpu_post_load()
>>              - updates cpu->cpreg_values[i] : which includes MIDR (processor 
>> ID register)
>> 
>>              - calls write_list_to_kvmstate(cpu, KVM_PUT_FULL_STATE)
>> 
>>                              target/arm/kvm.c: write_list_to_kvmstate
>>                              - calls => kvm_vcpu_ioctl(cs, KVM_SET_ONE_REG, 
>> &r);
>> 
>>                                      => and it eventually lands up IIRC in 
>> Linux code in 
>> 
>>                                                      => 
>> arch/arm64/kvm/sys_regs.c : set_invariant_sys_reg(u64 id, void __user *uaddr)
>>                                                              /* This is what 
>> we mean by invariant: you can't change it. */
>>                                                              if (r->val != 
>> val)
>>                                                                      return 
>> -EINVAL;
>>                                                              Note: MIDR_EL1 
>> is invariant register.
>> result: Migration fails on Machine B.
>> 
>> A few points:
>> - qemu on arm64 is invoked with -machine virt and -cpu as host. So we don't 
>> explicitly define which cpu. 
>
> Note that even on x86 we don't guarantee as much about '-cpu host', what
> we expect to work for migration is that if you pick a   '-cpu amodel'
> and both hosts support the feature flags required by 'amodel' then it
> should work.

I really think that the right approach here is not using -cpu host.  You
do the full work, create a model as David says, and be sure that you car
run that model on both cpus.  It is a lot of work, but it is the only
way to make sure that this is going to work long term.

>> - In case Machine A and Machine B have almost same Core and the
>> delta may-not have any effect on qemu operation, migration should
>> work by just looking into whitelist.
>> whitelist can be given as a parameter for qemu on machine B.
>> 
>> qemu-system-aarch64 -whitelist <ids separated by commas>
>> 
>> (This is my proposal)
>> 
>> - So in cpu_post_load (Machine B) qemu can lookup whitelist and
>> replace the MIDR with the one at Machine B.
>> Sounds good?
>> 
>> - Juan raised a point about clock speed, I am not sure it will have
>> any effect on arm since qemu is run with -cpu host param.
>> I could be wrong here, Peter/Eric can you please correct me...
>
> Clock speed is only really a problem for things like timestamp counters;
> some cores let you scale them; for those that don't then yes it's a bit
> odd.

This was just one example that I thought on top of my head.  Other thing
that I remember was that at some point, if you migrated to a cpu with
hyperthreading, you lost half of your performance counters.  I still
think that you need to create a proper model, and work from there.

Later, Juan.