Re: [PATCH RFC 14/15] migration: Postcopy preemption on separate channel

[Dr. David Alan Gilbert]

* Peter Xu (peterx@redhat.com) wrote:
> On Thu, Feb 03, 2022 at 05:45:32PM +0000, Dr. David Alan Gilbert wrote:
> > * Peter Xu (peterx@redhat.com) wrote:
> > > This patch enables postcopy-preempt feature.
> > > 
> > > It contains two major changes to the migration logic:
> > > 
> > >   (1) Postcopy requests are now sent via a different socket from precopy
> > >       background migration stream, so as to be isolated from very high 
> > > page
> > >       request delays
> > > 
> > >   (2) For huge page enabled hosts: when there's postcopy requests, they 
> > > can now
> > >       intercept a partial sending of huge host pages on src QEMU.
> > > 
> > > After this patch, we'll have two "channels" (or say, sockets, because 
> > > it's only
> > > supported on socket-based channels) for postcopy: (1) PRECOPY channel 
> > > (which is
> > > the default channel that transfers background pages), and (2) POSTCOPY
> > > channel (which only transfers requested pages).
> > > 
> > > On the source QEMU, when we found a postcopy request, we'll interrupt the
> > > PRECOPY channel sending process and quickly switch to the POSTCOPY 
> > > channel.
> > > After we serviced all the high priority postcopy pages, we'll switch back 
> > > to
> > > PRECOPY channel so that we'll continue to send the interrupted huge page 
> > > again.
> > > There's no new thread introduced.
> > > 
> > > On the destination QEMU, one new thread is introduced to receive page 
> > > data from
> > > the postcopy specific socket.
> > > 
> > > This patch has a side effect.  After sending postcopy pages, previously 
> > > we'll
> > > assume the guest will access follow up pages so we'll keep sending from 
> > > there.
> > > Now it's changed.  Instead of going on with a postcopy requested page, 
> > > we'll go
> > > back and continue sending the precopy huge page (which can be intercepted 
> > > by a
> > > postcopy request so the huge page can be sent partially before).
> > > 
> > > Whether that's a problem is debatable, because "assuming the guest will
> > > continue to access the next page" doesn't really suite when huge pages are
> > > used, especially if the huge page is large (e.g. 1GB pages).  So that 
> > > locality
> > > hint is much meaningless if huge pages are used.
> > > 
> > > If postcopy preempt is enabled, a separate channel is created for it so 
> > > that it
> > > can be used later for postcopy specific page requests.  On dst node, a
> > > standalone thread is used to receive postcopy requested pages.  The 
> > > thread is
> > > created along with the ram listen thread during POSTCOPY_LISTEN phase.
> > 
> > I think this patch could do with being split into two; the first one that
> > deals with closing/opening channels; and the second that handles the
> > data on the two channels and does the preemption.
> 
> Sounds good, I'll give it a shot on the split.
> 
> > 
> > Another thought is whether, if in the future we allow multifd +
> > postcopy, the multifd code would change - I think it would end up closer
> > to using multiple channels taking different pages on each one.
> 
> Right, so potentially the postcopy channels can be multi-threaded too itself.
> 
> We've had a quick discussion on irc, just to recap: I didn't reuse multifd
> infra because IMO multifd is designed with below ideas in mind:
> 
>   (1) Every multifd thread is equal
>   (2) Throughput oriented
> 
> However I found that postcopy needs something different when they're mixed up
> together with multifd.
> 
> Firstly, we will have some channels sending as much as we could where latency
> is not an issue (aka background pages).  However it's not suitable for page
> requests, so we could also have channels that are servicing page faults fron
> dst.  In short, there're two types of channels/threads we want, and we may 
> want
> to treat them differently.
> 
> The current model is we only have 1 postcopy channel and 1 precopy channel, 
> but
> it should be easier if we want to make it N post + 1 pre base on this series.

It's not clear to me if we need to be able to do N post + M pre, or
whether we have a rule like always at least 1 post, but if there's more
pagefaults in the queue then you can steal all of the pre channels.

> So far all send() is still done in the migration thread so no new sender 
> thread
> but 1 more receiver thread only. If we want to grow that 1->N for postcopy
> channels we may want to move that out too just like what we do with multifd.
> Not sure whether there can be something reused around.  That's where I haven't
> yet explored, but this series should already share a common piece of code on
> refactoring of things like tmp huge page on dst node to be able to receive 
> with
> multiple huge pages.

Right; it makes me think the multifd+postcopy should just use channels.

> This also reminded me that, instead of a new capability, should I simply 
> expose
> a parameter "postcopy-channels=N" to CLI so that we can be prepared with multi
> postcopy channels?

I'm not sure we know enough yet about what configuration it would have;
I'd be tempted to just make it work for the user by enabling both
multifd and preemption and then using this new mechanism rather than
having to add yet another parameter.

Dave

> > 
> > 
> > Do we need to do anything in psotcopy recovery ?
> 
> Yes. It's a todo (in the cover letter), if the whole thing looks sane I'll add
> that together in the non-rfc series.
> 
> Thanks,
> 
> -- 
> Peter Xu
> 
-- 
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK