[Top][All Lists]

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [Qemu-devel] [PATCH RFC] hw/pvrdma: Proposal of a new pvrdma device

From: Marcel Apfelbaum
Subject: Re: [Qemu-devel] [PATCH RFC] hw/pvrdma: Proposal of a new pvrdma device
Date: Tue, 4 Apr 2017 16:38:40 +0300
User-agent: Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101 Thunderbird/45.1.1

On 04/03/2017 09:23 AM, Leon Romanovsky wrote:
On Fri, Mar 31, 2017 at 06:45:43PM +0300, Marcel Apfelbaum wrote:
On 03/30/2017 11:28 PM, Doug Ledford wrote:
On 3/30/17 9:13 AM, Leon Romanovsky wrote:
On Thu, Mar 30, 2017 at 02:12:21PM +0300, Marcel Apfelbaum wrote:
From: Yuval Shaia <address@hidden>


 General description
 This is a very early RFC of a new RoCE emulated device
 that enables guests to use the RDMA stack without having
 a real hardware in the host.

 The current implementation supports only VM to VM communication
 on the same host.
 Down the road we plan to make possible to be able to support
 inter-machine communication by utilizing physical RoCE devices
 or Soft RoCE.

 The goals are:
 - Reach fast and secure loos-less Inter-VM data exchange.
 - Support remote VMs or bare metal machines.
 - Allow VMs migration.
 - Do not require to pin all VM memory.

 Have a QEMU implementation of the PVRDMA device. We aim to do so without
 any change in the PVRDMA guest driver which is already merged into the
 upstream kernel.

 RFC status
 The project is in early development stages and supports
 only basic send/receive operations.

 We present it so we can get feedbacks on design,
 feature demands and to receive comments from the
 community pointing us to the "right" direction.

If to judge by the feedback which you got from RDMA community
for kernel proposal [1], this community failed to understand:
1. Why do you need new module?

In this case, this is a qemu module to allow qemu to provide a virt rdma device 
to guests that is compatible with the device provided by VMWare's ESX product.  
Right now, the vmware_pvrdma driver
works only when the guest is running on a VMWare ESX server product, this would 
change that.  Marcel mentioned that they are currently making it compatible 
because that's the easiest/quickest thing to
do, but in the future they might extend beyond what VMWare's virt rdma driver 
provides/uses and might then need to either modify it to work with their 
extensions or fork and create their own virt
client driver.

2. Why existing solutions are not enough and can't be extended?

This patch is against the qemu source code, not the kernel.  There is no other 
solution in the qemu source code, so there is no existing solution to extend.

3. Why RXE (SoftRoCE) can't be extended to perform this inter-VM
   communication via virtual NIC?

Eventually they want this to work on real hardware, and to be more or less 
transparent to the guest.  They will need to make it independent of the kernel 
hardware/driver in use.  That means their own
virt driver, then the virt driver will eventually hook into whatever hardware 
is present on the system, or failing that, fall back to soft RoCE or soft iWARP 
if that ever makes it in the kernel.

Hi Leon and Doug,
Your feedback is much appreciated!

As Doug mentioned, the RFC is a QEMU implementation of a pvrdma device,
so SoftRoCE can't help here (we are emulating a PCI device).

I just responded to the latest email, but as you understood from my question,
it was related to your KDBR module.

Regarding the new KDBR module (Kernel Data Bridge), as the name suggests is
a bridge between different VMs or between a VM and a hardware/software device
and does not replace it.

Leon, utilizing the Soft RoCE is definitely part of our roadmap from the start,
we find the project a must since most of our systems don't even have real
RDMA hardware, and the question is how do best integrate with it.

This is exactly the question, you chose as an implementation path to do
it with new module over char device. I'm not against your approach,
but would like to see the list with pros and cons for over possible
solutions if any. Does it make sense to do special ULP to share the data
between different drivers over shared memory?

Hi Leon,

Here are some thoughts regarding the Soft RoCE usage in our project.
We thought about using it as backend for QEMU pvrdma device
we didn't how it will support our requirements.

1. Does Soft RoCE support inter process (VM) fast path ? The KDBR
   removes the need for hw resources, emulated or not, concentrating
   on one copy from a VM to another.

2. We needed to support migration, meaning the PVRDMA device must preserve
   the RDMA resources between different hosts. Our solution includes a clear
   separation between the guest resources namespace and the actual hw/sw device.
   This is why the KDBR is intended to run outside the scope of the SoftRoCE
   so it can open/close hw connections independent from the VM.

3. Our intention is for KDBR to be used in other contexts as well when we need
   inter VM data exchange, e.g. backend for virtio devices. We didn't see how 
   kind of requirement can be implemented inside SoftRoce as we don't see any
   connection between them.

4. We don't want all the VM memory to be pinned since it disable 
   which in turn will make the pvrdma device useless.
   We weren't sure how nice would play Soft RoCE with memory pinning and we 
   more control on memory management. It may be a solvable issue, but combined
   with the others lead us to our decision to come up with our kernel bridge 
   device or not, we went for it since it was the easiest to implement for a 

Marcel & Yuval


Marcel & Yuval

Can you please help us to fill this knowledge gap?

[1] http://marc.info/?l=linux-rdma&m=149063626907175&w=2

To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to address@hidden
More majordomo info at  http://vger.kernel.org/majordomo-info.html

reply via email to

[Prev in Thread] Current Thread [Next in Thread]