Re: Time to introduce a migration protocol negotiation (Re: [PATCH v2 00/25] migration: Postcopy Preemption)

[Peter Xu]

On Tue, Mar 15, 2022 at 11:15:41AM +0000, Daniel P. Berrangé wrote:
> > I still remember you mentioned the upper layer softwares can have
> > assumption on using only 1 pair of socket for migration, I think that makes
> > postcopy-preempt by default impossible.
> > 
> > Why multifd is different here?
> 
> It isn't different. We went through the pain to extending libvirt
> to know how to open many channels for multifd. We'll have todo
> the same with this postcopy-pre-empt. To this day though, management
> apps above libvirt largely don't enable multifd, which is a real
> shame. This is the key reason I think we need to handle this at
> the QEMU level automatically.

But I still don't undertand how QEMU could know about those tunnels, which
should be beyond QEMU's awareness?

The tunneling program can be some admin initiated socat tcp forwarding
programs, which by default may not allow >1 socket pairs.

Or maybe I have mis-understood on what's the tunneling we're discussing?

> 
> > > This post-copy is another case.  We should start off knowing
> > > we can switch to post-copy at any time.
> > 
> > This one is kind of special and it'll be harder, IMHO.
> > 
> > AFAIU, postcopy users will always initiate the migration with at least a
> > full round of precopy, with the hope that all the static guest pages will
> > be migrated.
> 
> I think I didn't explain myself properly here. Today there are
> two parts to postcopy usage in libvirt
> 
>   - Pass the "VIR_MIGRATE_POSTCOPY" when starting the migration.
>     The migration still runs in pre-copy mode. This merely ensures
>     we configure a bi-directional socket, so the app has the option
>     to swtich to postcopy later
> 
>   - Invoke virDomainMigrateStartPostCopy  to flip from pre-copy
>     to post-copy phase. This requires you previously passed
>     VIR_MIGRATE_POSTCOPY to enable its use.
> 
> The first point using 'VIR_MIGRATE_POSTCOPY' should not exist.
> That should be automaticaly negotiated and handled by QEMU.
> 
> Libvirt and mgmt apps should only need to care about whether
> or not they call virDomainMigrateStartPostCopy to flip to
> post-copy mode.

Ah I see.  I think Dave also mentioned it'll be a bit tricky to do so, but
it'll be at least sounds doable.

> 
> > > We should further be able to add pre-emption if we find it available.
> > 
> > Yeah here I have the same question per multifd above.  I just have no idea
> > whether QEMU has such knowledge on making this decision.  E.g., how could
> > QEMU know whether upper app is not tunneling the migration stream?  How
> > could QEMU know whether the upper app could handle multiple tcp sockets
> > well?
> 
> It can't do this today - that's why we need the new migration protocol
> feature negotiation I describe below.
> 
> > > So rather than following our historical practice, anjd adding
> > > yet another migration parameter for a specific feature, I'd
> > > really encourage us to put a stop to it and future proof
> > > ourselves.
> > > 
> > > 
> > > Introduce one *final-no-more-never-again-after-this* migration
> > > capability called "protocol-negotiation".
> > 
> > Let's see how Juan/Dave/others think.. anyway, that's something I always
> > wanted.
> > 
> > IMHO an even simpler term can be as simple as:
> > 
> >   -global migration.handshake=on
> 
> This is just inventing a new migration capability framework. We
> can just use existing QMP for this.

It's not a new one, it's just that a few years ago we exported the
migration capabilities to cmdline too (2081475841fe8), even if it's mostly
for debugging purpose.  In my daily tests it's quite handy.

> 
> > > When that capability is set, first declare that henceforth the
> > > migration transport is REQUIRED to support **multiple**,
> > > **bi-directional** channels.
> > 
> > This new capability will simply need to depend on the return-path
> > capability we already have.  E.g. exec-typed migration won't be able to
> > enable return-path, so not applicable to this one too.
> 
> 'exec' can be made to work if desired. Currently we only create
> a unidirectuional pipe and wire it up to stdin for outgoing
> migration. Nothing stops us declaring 'exec' uses a socketpair
> wired to stdin + stdout, and supprot invoking 'exec' multiple
> times to get many sockets

Yeah sounds working, it's just that we need to have users of it first.  One
point is that exec shouldn't be used in production but for quick hacks or
experiments, so supporting new/perf/enhancement features for it sounds a
bit over-engineering unless explicitly useful.

Thanks,

-- 
Peter Xu