Re: [Qemu-devel] [RFC PATCH RDMA support v5: 03/12] comprehensive protocol documentation

[Michael S. Tsirkin]

On Thu, Apr 11, 2013 at 04:50:21PM +0200, Paolo Bonzini wrote:
> Il 11/04/2013 16:37, Michael S. Tsirkin ha scritto:
> > 
> > pg1 ->  pin -> req -> res -> rdma -> done
> >         pg2 ->  pin -> req -> res -> rdma -> done
> >                 pg3 -> pin -> req -> res -> rdma -> done
> >                        pg4 -> pin -> req -> res -> rdma -> done
> >                               pg4 -> pin -> req -> res -> rdma -> done
> > 
> > It's like a assembly line see?  So while software does the registration
> > roundtrip dance, hardware is processing rdma requests for previous
> > chunks.
> 
> Does this only affects the implementation, or also the wire protocol?

It affects the wire protocol.

> Does the destination have to be aware that the source is doing pipelining?
> 
> Paolo

Yes. At the moment the protocol assumption is there's only one
outstanding command on the control queue.  So destination has to
prequeue multiple buffers on hardware receive queue, and keep the source
updated about the number of available buffers. Preferably it should do
this using existing responses, maybe a separate ready command
is enough - this needs some thought, since a separate command
consumes buffers itself.

> > 
> > ....
> > 
> > When do you have to stall? when you run out of rx buffer credits so you
> > can not start a new req.  Your protocol has 2 outstanding buffers,
> > so you can only have one req in the air. Do more and
> > you will not need to stall - possibly at all.
> > 
> > One other minor point is that your protocol requires extra explicit
> > ready commands. You can pass the number of rx buffers as extra payload
> > in the traffic you are sending anyway, and reduce that overhead.