Re: RFC: use VFIO over a UNIX domain socket to implement device offloading

[Stefan Hajnoczi]

On Mon, Apr 20, 2020 at 11:05:25AM +0000, Thanos Makatos wrote:
> > In order to interoperate we'll need to maintain a protocol
> > specification.  Mayb You and JJ could put that together and CC the vfio,
> > rust-vmm, and QEMU communities for discussion?
> > 
> > It should cover the UNIX domain socket connection semantics (does a
> > listen socket only accept 1 connection at a time?  What happens when the
> > client disconnects?  What happens when the server disconnects?), how
> > VFIO structs are exchanged, any vfio-over-socket specific protocol
> > messages, etc.  Basically everything needed to write an implementation
> > (although it's not necessary to copy the VFIO struct definitions from
> > the kernel headers into the spec or even document their semantics if
> > they are identical to kernel VFIO).
> > 
> > The next step beyond the LD_PRELOAD library is a native vfio-over-socket
> > client implementation in QEMU.  There is a prototype here:
> > https://github.com/elmarco/qemu/blob/wip/vfio-user/hw/vfio/libvfio-
> > user.c
> > 
> > If there are any volunteers for working on that then this would be a
> > good time to discuss it.
> 
> Hi,
> 
> I've just shared with you the Google doc we've working on with John where 
> we've
> been drafting the protocol specification, we think it's time for some first
> comments. Please feel free to comment/edit and suggest more people to be on 
> the
> reviewers list.
> 
> You can also find the Google doc here:
> 
> https://docs.google.com/document/d/1FspkL0hVEnZqHbdoqGLUpyC38rSk_7HhY471TsVwyK8/edit?usp=sharing
> 
> If a Google doc doesn't work for you we're open to suggestions.

I can't add comments to the document so I've inlined them here:

The spec assumes the reader is already familiar with VFIO and does not
explain concepts like the device lifecycle, regions, interrupts, etc.
We don't need to duplicate detailed VFIO information, but I think the
device model should be explained so that anyone can start from the
VFIO-user spec and begin working on an implementation.  Right now I
think they would have to do some serious investigation of VFIO first in
order to be able to write code.

"only the source header files are used"
I notice the current <linux/vfio.h> header is licensed "GPL-2.0 WITH
Linux-syscall-note".  I'm not a lawyer but I guess this means there are
some restrictions on using this header file.  The <linux/virtio*.h>
header files were explicitly licensed under the BSD license to make it
easy to use the non __KERNEL__ parts.

VFIO-user Command Types: please indicate for each request type whether
it is client->server, server->client, or both.  Also is it a "command"
or "request"?

vfio_user_req_type <-- is this an extension on top of <linux/vfio.h>?
Please make it clear what is part of the base <linux/vfio.h> protocol
and what is specific to vfio-user.

VFIO_USER_READ/WRITE serve completely different purposes depending on
whether they are sent client->server or server->client.  I suggest
defining separate request type constants instead of overloading them.

What is the difference between VFIO_USER_MAP_DMA and VFIO_USER_REG_MEM?
They both seem to be client->server messages for setting up memory but
I'm not sure why two request types are needed.

struct vfio_user_req->data.  Is this really a union so that every
message has the same size, regardless of how many parameters are passed
in the data field?

"a framebuffer where the guest does multiple stores to the virtual
device."  Do you mean in SMP guests?  Or even in a single CPU guest?

Also, is there any concurrency requirement on the client and server
side?  Can I implement a client/server that processes requests
sequentially and completes them before moving on to the next request or
would that deadlock for certain message types?

signature.asc
Description: PGP signature