[Qemu-devel] [PATCH v8 4/6] docs: Add Documentation for Mediated devices

[Kirti Wankhede]

Add file Documentation/vfio-mediated-device.txt that include details of
mediated device framework.

Signed-off-by: Kirti Wankhede <address@hidden>
Signed-off-by: Neo Jia <address@hidden>
Change-Id: I137dd646442936090d92008b115908b7b2c7bc5d
---
 Documentation/vfio-mdev/vfio-mediated-device.txt | 219 +++++++++++++++++++++++
 1 file changed, 219 insertions(+)
 create mode 100644 Documentation/vfio-mdev/vfio-mediated-device.txt

diff --git a/Documentation/vfio-mdev/vfio-mediated-device.txt 
b/Documentation/vfio-mdev/vfio-mediated-device.txt
new file mode 100644
index 000000000000..c1eacb83807b
--- /dev/null
+++ b/Documentation/vfio-mdev/vfio-mediated-device.txt
@@ -0,0 +1,219 @@
+/*
+ * VFIO Mediated devices
+ *
+ * Copyright (c) 2016, NVIDIA CORPORATION. All rights reserved.
+ *     Author: Neo Jia <address@hidden>
+ *             Kirti Wankhede <address@hidden>
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+ */
+
+VFIO Mediated devices [1]
+-------------------------
+
+There are more and more use cases/demands to virtualize the DMA devices which
+don't have SR_IOV capability built-in. To do this, drivers of different
+devices had to develop their own management interface and set of APIs and then
+integrate it to user space software. We've identified common requirements and
+unified management interface for such devices to make user space software
+integration easier and simplify the device driver implementation to support 
such
+I/O virtualization solution.
+
+The VFIO driver framework provides unified APIs for direct device access from
+user space. It is an IOMMU/device agnostic framework for exposing direct device
+access to user space, in a secure, IOMMU protected environment. This framework
+is used for multiple devices like GPUs, network adapters and compute
+accelerators. With direct device access, virtual machines or user space
+applications have direct access of physical device. This framework is reused
+for mediated devices.
+
+Mediated core driver provides a common interface for mediated device management
+that can be used by drivers of different devices. This module provides a 
generic
+interface to create/destroy a mediated device, add/remove it to mediated bus
+driver and add/remove device to an IOMMU group. It also provides an interface 
to
+register bus driver, for example, Mediated VFIO mdev driver is designed for
+mediated devices and supports VFIO APIs. Mediated bus driver add/delete 
mediated
+device to VFIO Group.
+
+Below is the high Level block diagram, with NVIDIA, Intel and IBM devices
+as example, since these are the devices which are going to actively use
+this module as of now.
+
+     +---------------+
+     |               |
+     | +-----------+ |  mdev_register_driver() +--------------+
+     | |           | +<------------------------+              |
+     | |  mdev     | |                         |              |
+     | |  bus      | +------------------------>+ vfio_mdev.ko |<-> VFIO user
+     | |  driver   | |     probe()/remove()    |              |    APIs
+     | |           | |                         +--------------+
+     | +-----------+ |
+     |               |
+     |  MDEV CORE    |
+     |   MODULE      |
+     |   mdev.ko     |
+     | +-----------+ |  mdev_register_device() +--------------+
+     | |           | +<------------------------+              |
+     | |           | |                         |  nvidia.ko   |<-> physical
+     | |           | +------------------------>+              |    device
+     | |           | |        callbacks        +--------------+
+     | | Physical  | |
+     | |  device   | |  mdev_register_device() +--------------+
+     | | interface | |<------------------------+              |
+     | |           | |                         |  i915.ko     |<-> physical
+     | |           | +------------------------>+              |    device
+     | |           | |        callbacks        +--------------+
+     | |           | |
+     | |           | |  mdev_register_device() +--------------+
+     | |           | +<------------------------+              |
+     | |           | |                         | ccw_device.ko|<-> physical
+     | |           | +------------------------>+              |    device
+     | |           | |        callbacks        +--------------+
+     | +-----------+ |
+     +---------------+
+
+
+Registration Interfaces:
+------------------------
+
+Mediated core driver provides two types of registration interfaces:
+
+1. Registration interface for mediated bus driver:
+--------------------------------------------------
+     /*
+      * struct mdev_driver [2] - Mediated device's driver
+      * @name: driver name
+      * @probe: called when new device created
+      * @remove: called when device removed
+      * @driver: device driver structure
+      */
+     struct mdev_driver {
+            const char *name;
+            int  (*probe)  (struct device *dev);
+            void (*remove) (struct device *dev);
+            struct device_driver    driver;
+     };
+
+Mediated bus driver for mdev should use this interface to register and
+unregister with core driver respectively:
+
+extern int  mdev_register_driver(struct mdev_driver *drv, struct module 
*owner);
+extern void mdev_unregister_driver(struct mdev_driver *drv);
+
+Mediated bus driver is responsible to add/delete mediated devices to/from VFIO
+group when devices are bound and unbound to the driver.
+
+2. Physical device driver interface:
+------------------------------------
+This interface [3] provides a set of APIs to manage physical device related 
work
+in its driver. APIs are:
+
+* dev_attr_groups: attributes of parent device.
+* mdev_attr_groups: attributes of mediated device.
+* supported_type_groups: attributes to define supported types. It is mandatory
+                        to provide supported types.
+* create: to allocate basic resources in driver for a mediated device.
+* remove: to free resources in driver when mediated device is destroyed.
+* open: open callback of mediated device
+* release: close callback of mediated device
+* read : read emulation callback.
+* write: write emulation callback.
+* mmap: mmap emulation callback.
+* ioctl: ioctl callback.
+
+Drivers should use these interfaces to register and unregister device to mdev
+core driver respectively:
+
+extern int  mdev_register_device(struct device *dev,
+                                 const struct parent_ops *ops);
+extern void mdev_unregister_device(struct device *dev);
+
+Mediated device management interface via sysfs
+----------------------------------------------
+Management interface via sysfs allows user space software, like libvirt, to
+query and configure mediated device in a HW agnostic fashion. This management
+interface provide flexibility to underlying physical device's driver to support
+mediated device hotplug, multiple mediated devices per virtual machine, 
multiple
+mediated devices from different physical devices, etc.
+
+Under per-physical device sysfs:
+--------------------------------
+
+* mdev_supported_types:
+    List of current supported mediated device types and its details are added
+in this directory in following format:
+
+|- <parent phy device>
+|--- Vendor-specific-attributes [optional]
+|--- mdev_supported_types
+|     |--- <type id>
+|     |   |--- create
+|     |   |--- name
+|     |   |--- available_instances
+|     |   |--- description /class
+|     |   |--- [devices]
+|     |--- <type id>
+|     |   |--- create
+|     |   |--- name
+|     |   |--- available_instances
+|     |   |--- description /class
+|     |   |--- [devices]
+|     |--- <type id>
+|          |--- create
+|          |--- name
+|          |--- available_instances
+|          |--- description /class
+|          |--- [devices]
+
+[TBD : description or class is yet to be decided. This will change.]
+
+Under per mdev device:
+----------------------
+
+|- <parent phy device>
+|--- $MDEV_UUID
+         |--- remove
+         |--- {link to its type}
+         |--- vendor-specific-attributes [optional]
+
+* remove: (write only)
+       Write '1' to 'remove' file would destroy mdev device. Vendor driver can
+       fail remove() callback if that device is active and vendor driver
+       doesn't support hot-unplug.
+       Example:
+       # echo 1 > /sys/bus/mdev/devices/$mdev_UUID/remove
+
+
+Mediated device Hotplug:
+------------------------
+
+Mediated devices can be created and assigned during runtime. Procedure to
+hot-plug mediated device is same as hot-plug PCI device.
+
+Translation APIs for Mediated device
+------------------------------------
+
+Below APIs are provided for user pfn to host pfn translation in VFIO driver:
+
+extern long vfio_pin_pages(struct device *dev, unsigned long *user_pfn,
+                           long npage, int prot, unsigned long *phys_pfn);
+
+extern long vfio_unpin_pages(struct device *dev, unsigned long *pfn,
+                            long npage);
+
+These functions call back into the backend IOMMU module using two callbacks of
+struct vfio_iommu_driver_ops, pin_pages and unpin_pages [4]. Currently these 
are
+supported in TYPE1 IOMMU module. To enable the same for other IOMMU backend
+modules, such as PPC64 sPAPR module, they need to provide these two callback
+functions.
+
+References
+----------
+
+[1] See Documentation/vfio.txt for more information on VFIO.
+[2] struct mdev_driver in include/linux/mdev.h
+[3] struct parent_ops in include/linux/mdev.h
+[4] struct vfio_iommu_driver_ops in include/linux/vfio.h
+
-- 
2.7.0