Re: [Qemu-devel] [PATCH v7 00/12] rdma: migration support

[Michael R. Hines]

On 06/13/2013 10:26 AM, Chegu Vinod wrote:
> > 1. start QEMU with the lock option *first*
> > 2. Then enable x-rdma-pin-all
> > 3. Then perform the migration
> >
> > What happens here? Does pinning "in advance" help you?
>
> Yes it does help by avoiding the freeze time at the start of the 
> pin-all migration.
>
> I already mentioned about this in in my earlier responses as an option 
> to consider for larger guests 
> (https://lists.gnu.org/archive/html/qemu-devel/2013-05/msg00435.html).
>
> But pinning all of guest memory has a few drawbacks...as you may 
> already know.
>
> Just to be sure I just double checked it again with your v7 bits . 
> Started a 64GB/10VCPU guest  (started qemu with the"-realtime 
> mlock=on" option) and as expected the guest startup took about 20 
> seconds longer (i.e. time taken to mlock the 64GB of guest RAM) but 
> the pin-all migration started fine...i.e. didn't observe any freezes 
> at the start of the migration
(CC-ing qemu-devel).

OK, that's good to know. This means that we need to bringup the mlock() 
problem as a "larger" issue in the linux community instead of the QEMU 
community.

In the meantime, how about I make update to the RDMA patch which does 
the following:

1. Solution #1:
       If user requests "x-rdma-pin-all", then
            If QEMU has enabled "-realtime mlock=on"
                   Then, allow the capability
            Else
                  Disallow the capability

2. Solution #2:  Create NEW qemu monitor command which locks memory *in 
advance*
                          before the migrate command occurs, to clearly 
indicate to the user
                          that the cost of locking memory must be paid 
before the migration starts.

Which solution do you prefer? Or do you have alternative idea?


> https://lists.gnu.org/archive/html/qemu-devel/2013-04/msg04161.html
>
> Again this is a generic linux mlock/clearpage related issue and not 
> directly related to your changes.

Do you have any ideas on how linux can be improved to solve this?
Is there any ongoing work that you know of on mlock() performance?

Is there, perhaps, some way for linux to "parallelize" the 
mlock()/clearpage operation?

- Michael