Re: [Qemu-devel] [PATCH v11 14/15] rdma: introduce MIG_STATE_NONE and change MIG_STATE_SETUP state transition

[Paolo Bonzini]

Il 26/06/2013 02:31, Michael R. Hines ha scritto:
> On 06/25/2013 05:06 PM, Paolo Bonzini wrote:
>> Il 25/06/2013 22:56, Michael R. Hines ha scritto:
>>> I was wrong - this does require a protocol extension.
>>>
>>> This is because the RDMA transfers are asynchronous, and thus
>>> we cannot know in advance that it is safe to unregister the memory
>>> associated with each individual transfer before the transfer actually
>>> completes.
>>>
>>> While the destination currently uses the protocol to participate in
>>> *registering* the page, the destination does not participate in the
>>> RDMA transfers themselves, only the source does, and thus would
>>> require a new exchange of messages to block and instruct the
>>> destination to unpin the memory.
>> Yes, that's what I recalled too (really what mst told me :)).  Does it
>> need to be blocking though?  As long as the pinning is blocking, and
>> messages are processed in order, the source can proceed immediately
>> after sending an unpin message.  This assumes of course that the chunk
>> is not being transmitted, and I am not sure how easy the source can
>> determine that.
> 
> No, they're not processed in order. In fact, not only does the device
> write out of order, but also the PCI bus writes out of order.
> This was such a problem in fact, that I fixed several bugs as a result
> a few weeks ago (v7 of the patch with an in-depth description).
> 
> The destination simply cannot assume whatsoever what the ordering
> of the writes are - that's really the whole point of using RDMA in the
> first place so that the software can get out of the way of the transfer
> process to lower the latency of each transfer.

The memory is processed out of order, but what about the messages?
Those must be in order.

Note that I wrote above "This assumes of course that the chunk is not
being transmitted".  Can the source know when an asynchronous transfer
finished, and delay the unpinning until that time?

Paolo

> 
> The only option is to send a blocking message to the other side to
> request the unpinning (in addition to unpinning on the source first upon
> completion of the original transfer).
> 
> As you can expect, this would be very expensive and we must ensure
> that we have *very* good a-priori information that this memory will
> not need to be re-registered anytime in the near future.
> 
> - Michael
> 
> 
>