Re: [Qemu-devel] [RFC PATCH RDMA support v5: 03/12] comprehensive protocol documentation

[Michael S. Tsirkin]

On Sun, Apr 14, 2013 at 09:10:36PM -0400, Michael R. Hines wrote:
> On 04/14/2013 05:16 PM, Michael S. Tsirkin wrote:
> >On Sun, Apr 14, 2013 at 03:43:28PM -0400, Michael R. Hines wrote:
> >>On 04/14/2013 02:51 PM, Michael S. Tsirkin wrote:
> >>>On Sun, Apr 14, 2013 at 10:31:20AM -0400, Michael R. Hines wrote:
> >>>>On 04/14/2013 04:28 AM, Michael S. Tsirkin wrote:
> >>>>>On Fri, Apr 12, 2013 at 09:47:08AM -0400, Michael R. Hines wrote:
> >>>>>>Second, as I've explained, I strongly, strongly disagree with 
> >>>>>>unregistering
> >>>>>>memory for all of the aforementioned reasons - workloads do not
> >>>>>>operate in such a manner that they can tolerate memory to be
> >>>>>>pulled out from underneath them at such fine-grained time scales
> >>>>>>in the *middle* of a relocation and I will not commit to writing a 
> >>>>>>solution
> >>>>>>for a problem that doesn't exist.
> >>>>>Exactly same thing happens with swap, doesn't it?
> >>>>>You are saying workloads simply can not tolerate swap.
> >>>>>
> >>>>>>If you can prove (through some kind of anaylsis) that workloads
> >>>>>>would benefit from this kind of fine-grained memory overcommit
> >>>>>>by having cgroups swap out memory to disk underneath them
> >>>>>>without their permission, I would happily reconsider my position.
> >>>>>>
> >>>>>>- Michael
> >>>>>This has nothing to do with cgroups directly, it's just a way to
> >>>>>demonstrate you have a bug.
> >>>>>
> >>>>If your datacenter or your cloud or your product does not want to
> >>>>tolerate page registration, then don't use RDMA!
> >>>>
> >>>>The bottom line is: RDMA is useless without page registration. Without
> >>>>it, the performance of it will be crippled. If you define that as a bug,
> >>>>then so be it.
> >>>>
> >>>>- Michael
> >>>No one cares if you do page registration or not.  ulimit -l 10g is the
> >>>problem.  You should limit the amount of locked memory.
> >>>Lots of good research went into making RDMA go fast with limited locked
> >>>memory, with some success. Search for "registration cache" for example.
> >>>
> >>Patches using such a cache would be welcome.
> >>
> >>- Michael
> >>
> >And when someone writes them one day, we'll have to carry the old code
> >around for interoperability as well. Not pretty.  To avoid that, you
> >need to explicitly say in the documenation that it's experimental and
> >unsupported.
> >
> 
> That's what protocols are for.
> 
> As I've already said, I've incorporated this into the design of the protocol
> already.
> 
> The protocol already has a field called "repeat" which allows a user to
> request multiple chunk registrations at the same time.
> If you insist, I can add a capability / command to the protocol
> called "unregister chunk",
> but I'm not volunteering to implement that command as I don't have any data
> showing it to be of any value.

The value would be being able to run your code in qemu as unpriveledged
user.

> That would insulate the protocol against any such future
> "registration cache" design.
> 
> - Michael
>

It won't.  If it's unimplemented it won't be of any use since now your
code does not implement the protocol fully.
 
-- 
MST