Re: [PATCH v5 14/15] docs: Add documentation for SR-IOV and Virtualization Enhancements

[Lukasz Maniak]

On Tue, Mar 01, 2022 at 01:23:18PM +0100, Klaus Jensen wrote:
> On Feb 17 18:45, Lukasz Maniak wrote:
> > Signed-off-by: Lukasz Maniak <lukasz.maniak@linux.intel.com>
> 
> Please add a short commit description as well. Otherwise,

Klaus,

Sorry I forgot to add the description in v6 aka v7, been really busy
recently.
I am going to add the description for v8.

Regards,
Lukasz
> 
> Reviewed-by: Klaus Jensen <k.jensen@samsung.com>
> 
> > ---
> >  docs/system/devices/nvme.rst | 82 ++++++++++++++++++++++++++++++++++++
> >  1 file changed, 82 insertions(+)
> > 
> > diff --git a/docs/system/devices/nvme.rst b/docs/system/devices/nvme.rst
> > index b5acb2a9c19..aba253304e4 100644
> > --- a/docs/system/devices/nvme.rst
> > +++ b/docs/system/devices/nvme.rst
> > @@ -239,3 +239,85 @@ The virtual namespace device supports DIF- and 
> > DIX-based protection information
> >    to ``1`` to transfer protection information as the first eight bytes of
> >    metadata. Otherwise, the protection information is transferred as the 
> > last
> >    eight bytes.
> > +
> > +Virtualization Enhancements and SR-IOV (Experimental Support)
> > +-------------------------------------------------------------
> > +
> > +The ``nvme`` device supports Single Root I/O Virtualization and Sharing
> > +along with Virtualization Enhancements. The controller has to be linked to
> > +an NVM Subsystem device (``nvme-subsys``) for use with SR-IOV.
> > +
> > +A number of parameters are present (**please note, that they may be
> > +subject to change**):
> > +
> > +``sriov_max_vfs`` (default: ``0``)
> > +  Indicates the maximum number of PCIe virtual functions supported
> > +  by the controller. Specifying a non-zero value enables reporting of both
> > +  SR-IOV and ARI (Alternative Routing-ID Interpretation) capabilities
> > +  by the NVMe device. Virtual function controllers will not report SR-IOV.
> > +
> > +``sriov_vq_flexible``
> > +  Indicates the total number of flexible queue resources assignable to all
> > +  the secondary controllers. Implicitly sets the number of primary
> > +  controller's private resources to ``(max_ioqpairs - sriov_vq_flexible)``.
> > +
> > +``sriov_vi_flexible``
> > +  Indicates the total number of flexible interrupt resources assignable to
> > +  all the secondary controllers. Implicitly sets the number of primary
> > +  controller's private resources to ``(msix_qsize - sriov_vi_flexible)``.
> > +
> > +``sriov_max_vi_per_vf`` (default: ``0``)
> > +  Indicates the maximum number of virtual interrupt resources assignable
> > +  to a secondary controller. The default ``0`` resolves to
> > +  ``(sriov_vi_flexible / sriov_max_vfs)``
> > +
> > +``sriov_max_vq_per_vf`` (default: ``0``)
> > +  Indicates the maximum number of virtual queue resources assignable to
> > +  a secondary controller. The default ``0`` resolves to
> > +  ``(sriov_vq_flexible / sriov_max_vfs)``
> > +
> > +The simplest possible invocation enables the capability to set up one VF
> > +controller and assign an admin queue, an IO queue, and a MSI-X interrupt.
> > +
> > +.. code-block:: console
> > +
> > +   -device nvme-subsys,id=subsys0
> > +   -device nvme,serial=deadbeef,subsys=subsys0,sriov_max_vfs=1,
> > +    sriov_vq_flexible=2,sriov_vi_flexible=1
> > +
> > +The minimum steps required to configure a functional NVMe secondary
> > +controller are:
> > +
> > +  * unbind flexible resources from the primary controller
> > +
> > +.. code-block:: console
> > +
> > +   nvme virt-mgmt /dev/nvme0 -c 0 -r 1 -a 1 -n 0
> > +   nvme virt-mgmt /dev/nvme0 -c 0 -r 0 -a 1 -n 0
> > +
> > +  * perform a Function Level Reset on the primary controller to actually
> > +    release the resources
> > +
> > +.. code-block:: console
> > +
> > +   echo 1 > /sys/bus/pci/devices/0000:01:00.0/reset
> > +
> > +  * enable VF
> > +
> > +.. code-block:: console
> > +
> > +   echo 1 > /sys/bus/pci/devices/0000:01:00.0/sriov_numvfs
> > +
> > +  * assign the flexible resources to the VF and set it ONLINE
> > +
> > +.. code-block:: console
> > +
> > +   nvme virt-mgmt /dev/nvme0 -c 1 -r 1 -a 8 -n 1
> > +   nvme virt-mgmt /dev/nvme0 -c 1 -r 0 -a 8 -n 2
> > +   nvme virt-mgmt /dev/nvme0 -c 1 -r 0 -a 9 -n 0
> > +
> > +  * bind the NVMe driver to the VF
> > +
> > +.. code-block:: console
> > +
> > +   echo 0000:01:00.1 > /sys/bus/pci/drivers/nvme/bind
> > \ No newline at end of file
> > -- 
> > 2.25.1
> > 
> 
> -- 
> One of us - No more doubt, silence or taboo about mental illness.