Re: [RFC v3] VFIO Migration

[Stefan Hajnoczi]

On Tue, Nov 17, 2020 at 04:44:52AM -0500, Michael S. Tsirkin wrote:
> On Mon, Nov 16, 2020 at 02:38:12PM +0000, Stefan Hajnoczi wrote:
> > On Wed, Nov 11, 2020 at 03:41:59PM +0000, Dr. David Alan Gilbert wrote:
> > > * Stefan Hajnoczi (stefanha@redhat.com) wrote:
> > > > On Wed, Nov 11, 2020 at 12:56:26PM +0000, Dr. David Alan Gilbert wrote:
> > > > > * Stefan Hajnoczi (stefanha@redhat.com) wrote:
> > > > > > Orchestrating Migrations
> > > > > > ------------------------
> > > > > > In order to migrate a device a *migration parameter list* must 
> > > > > > first be built
> > > > > > on the source. Each migration parameter is added to the list if it 
> > > > > > is in
> > > > > > effect. For example, the migration parameter list for a device with
> > > > > > new-feature=off,num-queues=4 would be num-queues=4 if the 
> > > > > > new-feature migration
> > > > > > parameter was introduced with the off value disabling its effect.
> > > > > 
> > > > > What component builds that list (i.e. what component needs to know the
> > > > > history that new-feature=off was the default - ah I think you answer
> > > > > that below).
> > > > 
> > > > Yep. Thanks for noting this. I'll need to reorder things so it is clear.
> > > > 
> > > > > > The following conditions must be met to establish migration 
> > > > > > compatibility:
> > > > > > 
> > > > > > 1. The source and destination device model strings match.
> > > > > > 
> > > > > > 2. Each migration parameter name from the migration parameter list 
> > > > > > is supported
> > > > > >    by the destination. For example, the destination supports the 
> > > > > > num-queues
> > > > > >    migration parameter.
> > > > > > 
> > > > > > 3. Each migration parameter value from the migration parameter list 
> > > > > > is
> > > > > >    supported by the destination. For example, the destination 
> > > > > > supports
> > > > > >    num-queues=4.
> > > > > 
> > > > > Hmm, are combinations of parameter checks needed - i.e. is it possible
> > > > > that a destination supports    num-queues=4 and  new-feature=on/off -
> > > > > but only supports new-feature=on when num-queues>2 ?
> > > > 
> > > > Yes, it's possible but cannot be expressed in the migration info JSON.
> > > > 
> > > > We need to choose a level of expressiveness that will be useful enough
> > > > without being complex. In the extreme the migration info would contain
> > > > Turing complete validation expressions (e.g. JavaScript) so that any
> > > > relationship can be expressed, but I doubt that complexity is needed.
> > > > The other extreme is just booleans and (opaque) strings for maximum
> > > > simplicity.
> > > > 
> > > > If the syntax is not expressive enough then it's impossible to check
> > > > migration compatibility without actually creating a new device instance
> > > > on the destination. Daniel Berrange raised the requirement of checking
> > > > migration compatibility without creating the device since this helps
> > > > with selecting a migration destination.
> > > 
> > > Right, but my worry isn't the JSON description, it's the set of 3
> > > conditions above; they need to state that only some combinations need to
> > > be valid.
> > 
> > Yes, the proposed syntax is simply not expressive enough. The migration
> > compatibility check will pass and then the destination will refuse to
> > set up the device (before the device state is transferred).
> > 
> > Any suggestions for a syntax without full-blown arithmetic and logic
> > expressions?
> > 
> > > > Any ideas for a better syntax?
> > > 
> > > I'd be happy with a --param name=value   repeatedly, but also know that
> > > some option parsers don't like that.
> > 
> > Another wart, Sphinx considers repeated options an error so you cannot
> > document options using rST option syntax. I remember having this problem
> > when documenting virtiofsd's command-line options :).
> > 
> > If something comes to mind please let me know. I'm not set on a
> > particular syntax, but I'd like to choose the one that is both
> > human-friendly and compatible with option parsers while avoiding
> > namespace collisions with the device implementation's own options.
> > 
> > Stefan
> 
> 
> I think the simplest way is just to include and open-source tool
> for figuring all this out together with qemu.
> Any vendor interested in supporting migration with qemu
> will then just submit a patch for that tool.
> 
> This will also help make sure this interface
> is not just a way to bypass GPL, we can ask that the
> supporting server is opensource.
> 
> And it will help us guide vendors towards supporting migration
> correctly.

Can you describe the tool's command-line interface in more detail? Does
this tool completely replace the VFIO/mdev sysfs and vfio-user
command-line interfaces?

Is it just for vfio-user devices or also for VFIO/mdev devices?

Regarding GPL, I guess you mean the tool maintainers in QEMU would only
accept patches if the corresponding device backend implementation is
open source (GPL, MIT, BSD, Apache, etc)? I'm not sure if that helps
since proprietary vendors ship their own QEMU or can replace this tool
with another binary.

Stefan

signature.asc
Description: PGP signature