Re: [Qemu-devel] vhost-pci and virtio-vhost-user

[Stefan Hajnoczi]

On Fri, Jan 12, 2018 at 11:32:56AM +0800, Jason Wang wrote:
> 
> 
> On 2018年01月11日 23:23, Stefan Hajnoczi wrote:
> > On Thu, Jan 11, 2018 at 06:57:03PM +0800, Jason Wang wrote:
> > > 
> > > On 2018年01月11日 00:14, Stefan Hajnoczi wrote:
> > > > Hi Wei,
> > > > I wanted to summarize the differences between the vhost-pci and
> > > > virtio-vhost-user approaches because previous discussions may have been
> > > > confusing.
> > > > 
> > > > vhost-pci defines a new virtio device type for each vhost device type
> > > > (net, scsi, blk).  It therefore requires a virtio device driver for each
> > > > device type inside the slave VM.
> > > > 
> > > > Adding a new device type requires:
> > > > 1. Defining a new virtio device type in the VIRTIO specification.
> > > > 3. Implementing a new QEMU device model.
> > > > 2. Implementing a new virtio driver.
> > > > 
> > > > virtio-vhost-user is a single virtio device that acts as a vhost-user
> > > > protocol transport for any vhost device type.  It requires one virtio
> > > > driver inside the slave VM and device types are implemented using
> > > > existing vhost-user slave libraries (librte_vhost in DPDK and
> > > > libvhost-user in QEMU).
> > > > 
> > > > Adding a new device type to virtio-vhost-user involves:
> > > > 1. Adding any new vhost-user protocol messages to the QEMU
> > > >      virtio-vhost-user device model.
> > > > 2. Adding any new vhost-user protocol messages to the vhost-user slave
> > > >      library.
> > > > 3. Implementing the new device slave.
> > > > 
> > > > The simplest case is when no new vhost-user protocol messages are
> > > > required for the new device.  Then all that's needed for
> > > > virtio-vhost-user is a device slave implementation (#3).  That slave
> > > > implementation will also work with AF_UNIX because the vhost-user slave
> > > > library hides the transport (AF_UNIX vs virtio-vhost-user).  Even
> > > > better, if another person has already implemented that device slave to
> > > > use with AF_UNIX then no new code is needed for virtio-vhost-user
> > > > support at all!
> > > > 
> > > > If you compare this to vhost-pci, it would be necessary to design a new
> > > > virtio device, implement it in QEMU, and implement the virtio driver.
> > > > Much of virtio driver is more or less the same thing as the vhost-user
> > > > device slave but it cannot be reused because the vhost-user protocol
> > > > isn't being used by the virtio device.  The result is a lot of
> > > > duplication in DPDK and other codebases that implement vhost-user
> > > > slaves.
> > > > 
> > > > The way that vhost-pci is designed means that anyone wishing to support
> > > > a new device type has to become a virtio device designer.  They need to
> > > > map vhost-user protocol concepts to a new virtio device type.  This will
> > > > be time-consuming for everyone involved (e.g. the developer, the VIRTIO
> > > > community, etc).
> > > > 
> > > > The virtio-vhost-user approach stays at the vhost-user protocol level as
> > > > much as possible.  This way there are fewer concepts that need to be
> > > > mapped by people adding new device types.  As a result, it will allow
> > > > virtio-vhost-user to keep up with AF_UNIX vhost-user and grow because
> > > > it's easier to work with.
> > > > 
> > > > What do you think?
> > > > 
> > > > Stefan
> > > So a question is what's the motivation here?
> > > 
> > > Form what I'm understanding, vhost-pci tries to build a scalable V2V 
> > > private
> > > datapath. But according to what you describe here, virito-vhost-user tries
> > > to make it possible to implement the device inside another VM. I 
> > > understand
> > > the goal of vhost-pci could be done on top, but it looks to me it would 
> > > then
> > > rather similar to the design of Xen driver domain. So I can not figure out
> > > how it can be done in a high performance way.
> > vhost-pci and virtio-vhost-user both have the same goal.  They allow
> > a VM to implement a vhost device (net, scsi, blk, etc).
> 
> Looks not, if I read the code correctly, vhost-pci has a device
> implementation in qemu, and in slave VM it only have a vhost-pci-net driver.

You are right that the current "[PATCH v3 0/7] Vhost-pci for inter-VM
communication" does not reach this goal yet.  The patch series focusses
on a subset of vhost-user-net for poll mode drivers.

But the goal is to eventually let VMs implement any vhost device type.
Even if Wei, you, or I don't implement scsi, for example, someone else
should be able to do it based on vhost-pci or virtio-vhost-user.

Wei: Do you agree?

> > This allows
> > software defined network or storage appliances running inside a VM to
> > provide I/O services to other VMs.
> 
> Well, I think we can do it even with the existed virtio or whatever other
> emulated device which should not be bounded to any specific kind of device. 

Please explain the approach you have in mind.

> And what's more important, according to the kvm 2016 slides of vhost-pci,
> the motivation of vhost-pci is not building SDN but a chain of VNFs. So
> bypassing the central vswitch through a private VM2VM path does make sense.
> (Though whether or not vhost-pci is the best choice is still questionable).

This is probably my fault.  Maybe my networking terminology is wrong.  I
consider "virtual network functions" to be part of "software-defined
networking" use cases.  I'm not implying there must be a central virtual
switch.

To rephrase: vhost-pci enables exitless VM2VM communication.

> >   To the other VMs the devices look
> > like regular virtio devices.
> > 
> > I'm not sure I understand your reference to the Xen driver domain or
> > performance.
> 
> So what proposed here is basically memory sharing and event notification
> through eventfd, this model have been used by Xen for many years through
> grant table and event channel. Xen use this to move the backend
> implementation from dom0 to a driver domain which has direct access to some
> hardwares. Consider the case of network, it can then implement xen netback
> inside driver domain which can access hardware NIC directly.
>
> This makes sense for Xen and for performance since driver domain (backend)
> can access hardware directly and event was triggered through lower overhead
> hypercall (or it can do busypolling). But for virtio-vhost-user, unless you
> want SRIOV based solutions inside the slave VM, I believe we won't want to
> go back to Xen since the hardware virtualization can bring extra overheads.

Okay, this point is about the NFV use case.  I can't answer that because
I'm not familiar with it.

Even if the NFV use case is not ideal for VMs, there are many other use
cases for VMs implementing vhost devices.  In the cloud the VM is the
first-class object that users can manage.  They do not have the ability
to run vhost-user processes on the host.  Therefore I/O appliances need
to be able to run as VMs and vhost-pci (or virtio-vhost-user) solve that
problem.

> >   Both vhost-pci and virtio-vhost-user work using shared
> > memory access to the guest RAM of the other VM.  Therefore they can poll
> > virtqueues and avoid vmexit.  They do also support cross-VM interrupts,
> > thanks to QEMU setting up irqfd/ioeventfd appropriately on the host.
> > 
> > Stefan
> 
> So in conclusion, consider the complexity, I would suggest to figure out
> whether or not this (either vhost-pci or virito-vhost-user) is really
> required before moving ahead. E.g, for VM2VM direct network path, this looks
> simply an issue of network topology instead of the problem of device, so
> there's a lot of trick, for vhost-user one can easily image to write an
> application (or use testpmd) to build a zerocopied VM2VM datapath, isn't
> this not sufficient for the case?

See above, I described the general cloud I/O appliance use case.

Stefan

signature.asc
Description: PGP signature