[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
[Qemu-devel] [PATCH v7 23/23] docs: Update pvrdma device documentation
[Qemu-devel] [PATCH v7 23/23] docs: Update pvrdma device documentation
Mon, 17 Dec 2018 10:10:47 +0200
Interface with the device is changed with the addition of support for
Adjust documentation accordingly.
While there fix a minor mistake which may lead to think that there is a
relation between using RXE on host and the compatibility with bare-metal
Signed-off-by: Yuval Shaia <address@hidden>
Reviewed-by: Marcel Apfelbaum<address@hidden>
docs/pvrdma.txt | 126 ++++++++++++++++++++++++++++++++++++++++--------
1 file changed, 107 insertions(+), 19 deletions(-)
diff --git a/docs/pvrdma.txt b/docs/pvrdma.txt
index 5599318159..5175251b47 100644
@@ -9,8 +9,9 @@ It works with its Linux Kernel driver AS IS, no need for any
While it complies with the VMware device, it can also communicate with bare
-metal RDMA-enabled machines and does not require an RDMA HCA in the host, it
-can work with Soft-RoCE (rxe).
+metal RDMA-enabled machines as peers.
+It does not require an RDMA HCA in the host, it can work with Soft-RoCE (rxe).
It does not require the whole guest RAM to be pinned allowing memory
over-commit and, even if not implemented yet, migration support will be
@@ -78,29 +79,116 @@ the required RDMA libraries.
+3.1 VM Memory settings
Currently the device is working only with memory backed RAM
and it must be mark as "shared":
-m 1G \
-object memory-backend-ram,id=mb1,size=1G,share \
-numa node,memdev=mb1 \
-The pvrdma device is composed of two functions:
- - Function 0 is a vmxnet Ethernet Device which is redundant in Guest
- but is required to pass the ibdevice GID using its MAC.
- For an rxe backend using eth0 interface it will use its mac:
- -device vmxnet3,addr=<slot>.0,multifunction=on,mac=<eth0 MAC>
- For an SRIOV VF, we take the Ethernet Interface exposed by it:
- -device vmxnet3,multifunction=on,mac=<RoCE eth MAC>
- - Function 1 is the actual device:
- where the ibdevice can be rxe or RDMA VF (e.g. mlx5_4)
- Note: Pay special attention that the GID at backend-gid-idx matches vmxnet's
- The rules of conversion are part of the RoCE spec, but since manual conversion
- is not required, spotting problems is not hard:
- Example: GID: fe80:0000:0000:0000:7efe:90ff:fecb:743a
- MAC: 7c:fe:90:cb:74:3a
- Note the difference between the first byte of the MAC and the GID.
+3.2 MAD Multiplexer
+MAD Multiplexer is a service that exposes MAD-like interface for VMs in
+order to overcome the limitation where only single entity can register with
+MAD layer to send and receive RDMA-CM MAD packets.
+To build rdmacm-mux run
+# make rdmacm-mux
+The application accepts 3 command line arguments and exposes a UNIX socket
+to pass control and data to it.
+-d rdma-device-name Name of RDMA device to register with
+-s unix-socket-path Path to unix socket to listen (default
+-p rdma-device-port Port number of RDMA device to register with (default 1)
+The final UNIX socket file name is a concatenation of the 3 arguments so
+for example for device mlx5_0 on port 2 this /var/run/rdmacm-mux-mlx5_0-2
+will be created.
+pvrdma requires this service.
+Please refer to contrib/rdmacm-mux for more details.
+3.3 Service exposed by libvirt daemon
+The control over the RDMA device's GID table is done by updating the
+device's Ethernet function addresses.
+Usually the first GID entry is determined by the MAC address, the second by
+the first IPv6 address and the third by the IPv4 address. Other entries can
+be added by adding more IP addresses. The opposite is the same, i.e.
+whenever an address is removed, the corresponding GID entry is removed.
+The process is done by the network and RDMA stacks. Whenever an address is
+added the ib_core driver is notified and calls the device driver add_gid
+function which in turn update the device.
+To support this in pvrdma device the device hooks into the create_bind and
+destroy_bind HW commands triggered by pvrdma driver in guest.
+Whenever changed is made to the pvrdma port's GID table a special QMP
+messages is sent to be processed by libvirt to update the address of the
+backend Ethernet device.
+pvrdma requires that libvirt service will be up.
+3.4 PCI devices settings
+RoCE device exposes two functions - an Ethernet and RDMA.
+To support it, pvrdma device is composed of two PCI functions, an Ethernet
+device of type vmxnet3 on PCI slot 0 and a PVRDMA device on PCI slot 1. The
+Ethernet function can be used for other Ethernet purposes such as IP.
+3.5 Device parameters
+- netdev: Specifies the Ethernet device function name on the host for
+ example enp175s0f0. For Soft-RoCE device (rxe) this would be the Ethernet
+ device used to create it.
+- ibdev: The IB device name on host for example rxe0, mlx5_0 etc.
+- mad-chardev: The name of the MAD multiplexer char device.
+- ibport: In case of multi-port device (such as Mellanox's HCA) this
+ specify the port to use. If not set 1 will be used.
+- dev-caps-max-mr-size: The maximum size of MR.
+- dev-caps-max-qp: Maximum number of QPs.
+- dev-caps-max-sge: Maximum number of SGE elements in WR.
+- dev-caps-max-cq: Maximum number of CQs.
+- dev-caps-max-mr: Maximum number of MRs.
+- dev-caps-max-pd: Maximum number of PDs.
+- dev-caps-max-ah: Maximum number of AHs.
+- The first 3 parameters are mandatory settings, the rest have their
+- The last 8 parameters (the ones that prefixed by dev-caps) defines the top
+ limits but the final values is adjusted by the backend device limitations.
+- netdev can be extracted from ibdev's sysfs
+Define bridge device with vmxnet3 network backend:
+ <mac address='56:b4:44:e9:62:dc'/>
+ <source bridge='bridge1'/>
+ <model type='vmxnet3'/>
+ <address type='pci' domain='0x0000' bus='0x00' slot='0x10' function='0x0'
+Define pvrdma device:
+ <qemu:arg value='-object'/>
+ <qemu:arg value='memory-backend-ram,id=mb1,size=1G,share'/>
+ <qemu:arg value='-numa'/>
+ <qemu:arg value='node,memdev=mb1'/>
+ <qemu:arg value='-chardev'/>
+ <qemu:arg value='socket,path=/var/run/rdmacm-mux-rxe0-1,id=mads'/>
+ <qemu:arg value='-device'/>
- Re: [Qemu-devel] [PATCH v7 11/23] hw/pvrdma: Add support to allow guest to configure GID table, (continued)
- [Qemu-devel] [PATCH v7 13/23] hw/pvrdma: Make sure PCI function 0 is vmxnet3, Yuval Shaia, 2018/12/17
- [Qemu-devel] [PATCH v7 15/23] hw/pvrdma: Make device state depend on Ethernet function state, Yuval Shaia, 2018/12/17
- [Qemu-devel] [PATCH v7 14/23] hw/rdma: Initialize node_guid from vmxnet3 mac address, Yuval Shaia, 2018/12/17
- [Qemu-devel] [PATCH v7 16/23] hw/pvrdma: Fill all CQE fields, Yuval Shaia, 2018/12/17
- [Qemu-devel] [PATCH v7 17/23] hw/pvrdma: Fill error code in command's response, Yuval Shaia, 2018/12/17
- [Qemu-devel] [PATCH v7 18/23] hw/rdma: Remove unneeded code that handles more that one port, Yuval Shaia, 2018/12/17
- [Qemu-devel] [PATCH v7 19/23] vl: Introduce shutdown_notifiers, Yuval Shaia, 2018/12/17
- [Qemu-devel] [PATCH v7 20/23] hw/pvrdma: Clean device's resource when system is shutdown, Yuval Shaia, 2018/12/17
- [Qemu-devel] [PATCH v7 21/23] hw/rdma: Do not use bitmap_zero_extend to free bitmap, Yuval Shaia, 2018/12/17
- [Qemu-devel] [PATCH v7 23/23] docs: Update pvrdma device documentation,
Yuval Shaia <=
- [Qemu-devel] [PATCH v7 22/23] hw/rdma: Do not call rdma_backend_del_gid on an empty gid, Yuval Shaia, 2018/12/17