Re: [Qemu-devel] vm state save/restore question

[Juan Quintela]

Alexander Graf <address@hidden> wrote:
> On 09.06.2012, at 13:34, Benjamin Herrenschmidt wrote:
>
>> On Sat, 2012-06-09 at 20:53 +1000, Benjamin Herrenschmidt wrote:
>>> Hi folks !
>> 
>> (After some discussion with Andreas ...)
>> 
>>> I'm looking at sorting out the state save/restore of target-ppc (which
>>> means understanding in general how it works in qemu :-)
>>> 
>>> So far I've somewhat figured out that there's the "old way" where we
>>> just provide a "bulk" save/restore function pair, and the "new way"
>>> where we have nicely tagged field lists etc...
>>> 
>>> x86 seems to use the later for the CPU state, ppc is a mess and uses the
>>> former with interesting incompatible format change depending on how qemu
>>> is build :-) So I think that's one area I need to fix.
>> 
>> Ok, so I'm told there are patches to convert ppc, I haven't seen them in
>> my list archives, so if somebody has a pointer, please shoot, that will
>> save me some work :-)

I can send a new version tomorrow.

>>> What I'd need is something in spapr that can be used to "resync" bits of
>>> the cpu state with the external htab that gets run after everything is
>>> loaded and before emulation restarts.
>>> 
>>> Any idea how to do that properly ? I suppose I could also try to iterate
>>> all the vcpu's after loading the hash table & update the fields but not
>>> only that's gross ... I also don't know how to do it :-)
>> 
>> So I did an experiment using the "old style" save/restore (bad boy !)
>> and got that part to work by just iterating the vcpu's.
>> 
>> It's a bit nasty but it's the right way I think, ie, what we have here
>> (the external hash table) is a global object under control/ownership of
>> the platform code for which a pointer is cached in the CPU state (so the
>> mmu emulation gets to it easily), so those cached pointers need to be
>> updated in all CPUs when a new hash table is loaded/allocated.
>> 
>> That leads to another question however... I need to add save/restore to
>> a bunch more stuff such as the xics (interrupt controller), the various
>> spapr devices, etc...
>> 
>> So far the VMState stuff is all nice if you have fixed sized arrays.
>> However I haven't quite found out the right way to use it for things
>> like:
>> 
>> - The hash table (mentioned above). This is just a big chunk of memory
>> (it will routinely be 16M), so I really don't want to start iterating
>> all elements, just a bulk load will do, and the size might actually be
>> variable.

This is going to kill migration download time.  With current setup, we
just sent something like 1-2MB in stage 3 (i.e. after the machine is
down).  Default downtime is 30ms, And 16MB is going to take around 1s on
gigabit ethenet.

Once said that, if you told me the state that you want to sent, I can
take a look.

>> - The XICS (interrupt controller). The actual size of the interrupt
>> state array can vary (the number of interrupt sources can vary, it's
>> fixed today by the machine code but I wouldn't rely too much on that and
>> in any case, from the XICS driver perspective, it's not a constant, it's
>> a variable it gets passed when initializing).

Can you point me at the structure that you want to sent?

>> So in both these cases, I need either code to control the save/load
>> process (old style ? hard to hook into vmstate as far as I can tell) or
>> maybe a way to describe the array so that the array size itself is a
>> pointer to a variable (Andreas mentioned something along those lines).
>> Is there any doco for that stuff btw ? I haven't seen anything
>> detailed...
>
> I'm sure Juan knows more there :)

thanks for pointing me to the discussion O:-)

Later, Juan.