Re: [PATCH RFC 14/15] migration: Postcopy preemption on separate channel

[Peter Xu]

On Tue, Feb 08, 2022 at 11:24:14AM +0000, Dr. David Alan Gilbert wrote:
> > The current model is we only have 1 postcopy channel and 1 precopy channel, 
> > but
> > it should be easier if we want to make it N post + 1 pre base on this 
> > series.
> 
> It's not clear to me if we need to be able to do N post + M pre, or
> whether we have a rule like always at least 1 post, but if there's more
> pagefaults in the queue then you can steal all of the pre channels.

Right, >1 queue length should easily happen with workloads in real cloud
environment.  Though even with only 1post channel we can already hit at least
<~1ms with this series even if there're 16 pending requests per my test.  I
think that may cover quite some real workloads.

One thing to mention is that we should always assume the pre-channels are
filled up with tons of pages already in the NIC send buffer, so they won't be
good candidate for postcopy requests, IMHO.  So I'm not sure whether we can
mixly use the pre/post channels - we may need to leave the post channels idle.

Then, if we keep some of the multifd channels idle, then it will become some
other thing rather than the existing multifd, since we will start to treat
threads and channels differently and break the "equality" rule in the strict
version of multifd world.

> > This also reminded me that, instead of a new capability, should I simply 
> > expose
> > a parameter "postcopy-channels=N" to CLI so that we can be prepared with 
> > multi
> > postcopy channels?
> 
> I'm not sure we know enough yet about what configuration it would have;
> I'd be tempted to just make it work for the user by enabling both
> multifd and preemption and then using this new mechanism rather than
> having to add yet another parameter.

Let me stick with the current capability bit then, so as to make it 1pre+1post.
And we can leave Npre+1post for later.

Thanks,

-- 
Peter Xu