[Qemu-devel] [PATCH] docs/vhost-user: extend vhost-user to support the vhost-pci based inter-VM communication

[Wei Wang]

Signed-off-by: Wei Wang <address@hidden>
---
 docs/specs/vhost-user.txt | 127 +++++++++++++++++++++++++++++++++++++++-------
 1 file changed, 109 insertions(+), 18 deletions(-)

diff --git a/docs/specs/vhost-user.txt b/docs/specs/vhost-user.txt
index 7890d71..195fad6 100644
--- a/docs/specs/vhost-user.txt
+++ b/docs/specs/vhost-user.txt
@@ -17,31 +17,39 @@ The protocol defines 2 sides of the communication, master 
and slave. Master is
 the application that shares its virtqueues, in our case QEMU. Slave is the
 consumer of the virtqueues.
 
-In the current implementation QEMU is the Master, and the Slave is intended to
+In the traditional implementation QEMU is the Master, and the Slave is 
intended to
 be a software Ethernet switch running in user space, such as Snabbswitch.
 
 Master and slave can be either a client (i.e. connecting) or server (listening)
 in the socket communication.
 
+The current vhost-user protocol is extended to support the vhost-pci based 
inter-VM
+communication. In this case, Slave is a QEMU which runs a vhost-pci server, and
+Master is another QEMU which runs a vhost-pci client.
+
 Message Specification
 ---------------------
 
 Note that all numbers are in the machine native byte order. A vhost-user 
message
-consists of 3 header fields and a payload:
+consists of 4 header fields and a payload:
 
-------------------------------------
-| request | flags | size | payload |
-------------------------------------
+----------------------------------------------
+| request | flags | conn_id | size | payload |
+----------------------------------------------
 
  * Request: 32-bit type of the request
  * Flags: 32-bit bit field:
-   - Lower 2 bits are the version (currently 0x01)
-   - Bit 2 is the reply flag - needs to be sent on each reply from the slave
+   - Lower 2 bits are the version (currently 0x02)
+   - Bit 2 is the reply flag - needs to be sent on each reply
    - Bit 3 is the need_reply flag - see VHOST_USER_PROTOCOL_F_REPLY_ACK for
      details.
+ * Conn_id: 64-bit connection id to indentify a client socket connection. It is
+            introduced in version 0x02 to support the "1-server-N-client" model
+            and an asynchronous client read implementation. The connection id,
+            0xFFFFFFFFFFFFFFFF, is used by an anonymous client (e.g. a client 
who
+            has not got its connection id from the server in the initial talk)
  * Size - 32-bit size of the payload
 
-
 Depending on the request type, payload can be:
 
  * A single 64-bit integer
@@ -72,10 +80,12 @@ Depending on the request type, payload can be:
    Log: a 64-bit guest address for logging
 
  * Memory regions description
-   ---------------------------------------------------
-   | num regions | padding | region0 | ... | region7 |
-   ---------------------------------------------------
+   ----------------------------------------------------------
+   | uuid | num regions | padding | region0 | ... | region7 |
+   ----------------------------------------------------------
 
+   UUID: a 128-bit identification of the QEMU that the memory region
+         belongs to
    Num regions: a 32-bit number of regions
    Padding: 32-bit
 
@@ -97,6 +107,19 @@ Depending on the request type, payload can be:
    log offset: offset from start of supplied file descriptor
        where logging starts (i.e. where guest address 0 would be logged)
 
+* UUID description
+  --------
+  | uuid |
+  --------
+  uuid: 128 bit uuid
+
+* Rx queue gsi description
+   --------------------------------------
+   | num rx queues | gsi[num rx queues] |
+   --------------------------------------
+   num rx queues: 16 bit.
+   gsi[]: stores the GSI of each rx queue.
+
 In QEMU the vhost-user message is implemented with the following struct:
 
 typedef struct VhostUserMsg {
@@ -109,6 +132,8 @@ typedef struct VhostUserMsg {
         struct vhost_vring_addr addr;
         VhostUserMemory memory;
         VhostUserLog log;
+        uuit_t uuid;
+        RxQueueGSI rx_gsi;
     };
 } QEMU_PACKED VhostUserMsg;
 
@@ -119,17 +144,27 @@ The protocol for vhost-user is based on the existing 
implementation of vhost
 for the Linux Kernel. Most messages that can be sent via the Unix domain socket
 implementing vhost-user have an equivalent ioctl to the kernel implementation.
 
-The communication consists of master sending message requests and slave sending
-message replies. Most of the requests don't require replies. Here is a list of
-the ones that do:
+Traditionally, the communication consists of master sending message requests
+and slave sending message replies. Most of the requests don't require replies.
+Here is a list of the ones that do:
 
- * VHOST_GET_FEATURES
- * VHOST_GET_PROTOCOL_FEATURES
- * VHOST_GET_VRING_BASE
- * VHOST_SET_LOG_BASE (if VHOST_USER_PROTOCOL_F_LOG_SHMFD)
+ * VHOST_USER_GET_FEATURES
+ * VHOST_USER_GET_PROTOCOL_FEATURES
+ * VHOST_USER_GET_VRING_BASE
+ * VHOST_USER_SET_LOG_BASE (if VHOST_USER_PROTOCOL_F_LOG_SHMFD)
+ * VHOST_USER_GET_CONN_ID
+ * VHOST_USER_SET_PVI_SENDER_UUID
+ * VHOST_USER_SET_PEER_CONNECTION
 
 [ Also see the section on REPLY_ACK protocol extension. ]
 
+Currently, the communiaction also supports the Slave (server) sending messages 
or
+asynchronously replying to the Master (client). Here is a list of them:
+ * VHOST_USER_SET_RX_QUEUE_GSI
+ * VHOST_USER_SET_PVI_SENDER_UUID
+ * VHOST_USER_SET_PEER_CONNECTION (the serve may actively request to disconnect
+   with the client)
+
 There are several messages that the master sends with file descriptors passed
 in the ancillary data:
 
@@ -259,6 +294,7 @@ Protocol features
 #define VHOST_USER_PROTOCOL_F_LOG_SHMFD      1
 #define VHOST_USER_PROTOCOL_F_RARP           2
 #define VHOST_USER_PROTOCOL_F_REPLY_ACK      3
+#define VHOST_USER_PROTOCOL_F_VHOST_PCI      4
 
 Message types
 -------------
@@ -470,6 +506,61 @@ Message types
       The first 6 bytes of the payload contain the mac address of the guest to
       allow the vhost user backend to construct and broadcast the fake RARP.
 
+ * VHOST_USER_GET_CONN_ID
+      Id: 20
+      Equivalent ioctl: N/A
+      Master payload: u64
+
+      The client sends this message to the server to ask for its connection id.
+      The connection id is then put into the message header (the conn_id 
field),
+      so that the server can always know who it is talking with.
+
+ * VHOST_USER_SET_DEV_TYPE
+      Id: 21
+      Equivalent ioctl: N/A
+      Master payload: u64
+
+      The client sends the necessary info about the producer virtio device to
+      the server.
+      This request should be sent only when VHOST_USER_PROTOCOL_F_VHOST_PCI has
+      been negotiated.
+      #define VHOST_USER_DEV_TYPE_NET 1
+
+ * VHOST_USER_SET_PVI_SENDER_UUID
+      Id: 22
+      Equivalent ioctl: N/A
+      Master payload: uuid description
+
+      The producer sends its QEMU uuid to the consumer, the vhost-pci device, 
to
+      enable sending PV interrupts to the consumer, and vice versa.
+
+ * VHOST_USER_SET_RX_QUEUE_GSI
+      Id: 23
+      Equivalent ioctl: N/A
+      Master payload: rx queue gsi description
+
+      The producer device sends its rx queues' GSIs to its consumer device.
+      The consumer device also sends one of its rx queues' GSI to the producer
+      device after the queue's GSI is assigned.
+      This request should be sent only when VHOST_USER_PROTOCOL_F_VHOST_PCI has
+      been negotiated.
+
+ * VHOST_USER_SET_PEER_CONNECTION
+      Id: 24
+      Equivalent ioctl: N/A
+      Master payload: u64
+
+      The producer device requests to connect or disconnect with the consumer 
device.
+      The consumer device may request to disconnect with the producer device. 
This
+      request should be sent only when VHOST_USER_PROTOCOL_F_VHOST_PCI has been
+      negotiated.
+      Connection request: If the reply message indicates "success", the 
vhost-pci based
+      inter-VM communication channel has been established.
+      Disconnection request: If the reply message indicates "success", the 
vhost-pci based
+      inter-VM communication channel has been destroyed.
+      #define VHOST_USER_SET_PEER_CONNECTION_F_DISCONNECT 0
+      #define VHOST_USER_SET_PEER_CONNECTION_F_CONNECT 1
+
 VHOST_USER_PROTOCOL_F_REPLY_ACK:
 -------------------------------
 The original vhost-user specification only demands replies for certain
-- 
1.9.1