Re: [Qemu-devel] [RFC 0/3] Extend vhost-user to support VFIO based accelerators

[Alexey Kardashevskiy]

On 02/01/18 16:49, Liang, Cunming wrote:
> 
> 
>> -----Original Message-----
>> From: Alexey Kardashevskiy [mailto:address@hidden
>> Sent: Tuesday, January 2, 2018 10:42 AM
>> To: Bie, Tiwei <address@hidden>; address@hidden; qemu-
>> address@hidden; address@hidden; address@hidden;
>> address@hidden; address@hidden
>> Cc: Tan, Jianfeng <address@hidden>; Liang, Cunming
>> <address@hidden>; Wang, Xiao W <address@hidden>; Wang,
>> Zhihong <address@hidden>; Daly, Dan <address@hidden>
>> Subject: Re: [Qemu-devel] [RFC 0/3] Extend vhost-user to support VFIO based
>> accelerators
>>
>> On 22/12/17 17:41, Tiwei Bie wrote:
>>> This RFC patch set does some small extensions to vhost-user protocol
>>> to support VFIO based accelerators, and makes it possible to get the
>>> similar performance of VFIO passthru while keeping the virtio device
>>> emulation in QEMU.
>>>
>>> When we have virtio ring compatible devices, it's possible to setup
>>> the device (DMA mapping, PCI config, etc) based on the existing info
>>> (memory-table, features, vring info, etc) which is available on the
>>> vhost-backend (e.g. DPDK vhost library). Then, we will be able to use
>>> such devices to accelerate the emulated device for the VM. And we call
>>> it vDPA: vhost DataPath Acceleration. The key difference between VFIO
>>> passthru and vDPA is that, in vDPA only the data path (e.g. ring,
>>> notify and queue interrupt) is pass-throughed, the device control path
>>> (e.g. PCI configuration space and MMIO regions) is still defined and
>>> emulated by QEMU.
>>>
>>> The benefits of keeping virtio device emulation in QEMU compared with
>>> virtio device VFIO passthru include (but not limit to):
>>>
>>> - consistent device interface from guest OS;
>>> - max flexibility on control path and hardware design;
>>> - leveraging the existing virtio live-migration framework;
>>>
>>> But the critical issue in vDPA is that the data path performance is
>>> relatively low and some host threads are needed for the data path,
>>> because some necessary mechanisms are missing to support:
>>>
>>> 1) guest driver notifies the device directly;
>>> 2) device interrupts the guest directly;
>>>
>>> So this patch set does some small extensions to vhost-user protocol to
>>> make both of them possible. It leverages the same mechanisms (e.g.
>>> EPT and Posted-Interrupt on Intel platform) as the VFIO passthru to
>>> achieve the data path pass through.
>>>
>>> A new protocol feature bit is added to negotiate the accelerator
>>> feature support. Two new slave message types are added to enable the
>>> notify and interrupt passthru for each queue. From the view of
>>> vhost-user protocol design, it's very flexible. The passthru can be
>>> enabled/disabled for each queue individually, and it's possible to
>>> accelerate each queue by different devices. More design and
>>> implementation details can be found from the last patch.
>>>
>>> There are some rough edges in this patch set (so this is a RFC patch
>>> set for now), but it's never too early to hear the thoughts from the
>>> community! So any comments and suggestions would be really appreciated!
>>
>> I am missing a lot of context here. Out of curiosity - how is this all 
>> supposed to
>> work? QEMU command line example would be useful, what will the guest see? A
>> virtio device (i.e. Redhat vendor ID) or an actual PCI device (since VFIO is
>> mentioned)? Thanks.
> 
> It's a normal virtio PCIe devices in the guest. Extensions on the host are 
> transparent to the guest.
> 
> In terms of the usage, there's a sample may help.
> http://dpdk.org/ml/archives/dev/2017-December/085044.html
> The sample takes virtio-net device in VM as data path accelerator of 
> virtio-net in nested VM.


Aaah, this is for nested VMs, the original description was not clear about
this. I get it now, thanks.


> When taking physical device on bare metal, it accelerates virtio-net in VM 
> equivalently.
> There's no additional params of QEMU command line needed for vhost-user.
> 
> One more context, including vDPA enabling in DPDK vhost-user library.
> http://dpdk.org/ml/archives/dev/2017-December/084792.html



-- 
Alexey