Re: [RFC] Proposal of QEMU PCI Endpoint test environment

[Mattias Nissler]

On Thu, Oct 5, 2023 at 3:31 AM Shunsuke Mie <mie@igel.co.jp> wrote:
>
> Hi Jiri, Mattias and all.
>
> 2023年10月4日(水) 16:36 Mattias Nissler <mnissler@rivosinc.com>:
>>>
>>> hi shunsuke, all,
>>> what about vfio-user + qemu?
>
> Thank you for the suggestion.
>
>> FWIW, I have had some good success using VFIO-user to bridge software 
>> components to hardware designs. For the most part, I have been hooking up 
>> software endpoint models to hardware design components speaking the PCIe 
>> transaction layer protocol. The central piece you need is a way to translate 
>> between the VFIO-user protocol and PCIe transaction layer messages, 
>> basically converting ECAM accesses, memory accesses (DMA+MMIO), and 
>> interrupts between the two worlds. I have some code which implements the 
>> basics of that. It's certainly far from complete (TLP is a massive 
>> protocol), but it works well enough for me. I believe we should be able to 
>> open-source this if there's interest, let me know.
>
> It is what I want to do, but I'm not familiar with the vfio and vfio-user, 
> and I have a question. QEMU has a PCI TLP communication implementation for 
> Multi-process QEMU[1]. It is similar to your success.

I'm no qemu expert, but my understanding is that the plan is for the
existing multi-process QEMU implementation to eventually be
superseded/replaced by the VFIO-user based one (qemu folks, please
correct me if I'm wrong). From a functional perspective they are more
or less equivalent AFAICT.

> The multi-process qemu also communicates TLP over UDS. Could you let me know 
> your opinion about it?

Note that neither multi-process qemu nor VFIO-user actually pass
around TLPs, but rather have their own command language to encode
ECAM, MMIO, DMA, interrupts etc. However, translation from/to TLP is
possible and works well enough in my experience.

>
>> One thing to note is that there are currently some limits to bridging 
>> VFIO-user / TLP that I haven't figured out and/or will need further work: 
>> Advanced PCIe concepts like PASID, ATS/PRI, SR-IOV etc. may lack equivalents 
>> on the VFIO-user side that would have to be filled in. The folk behind 
>> libvfio-user[2] have been very approachable and open to improvements in my 
>> experience though.
>>
>> If I understand correctly, the specific goal here is testing PCIe endpoint 
>> designs against a Linux host. What you'd need for that is a PCI host 
>> controller for the Linux side to talk to and then hooking up endpoints on 
>> the transaction layer. QEMU can simulate host controllers that work with 
>> existing Linux drivers just fine. Then you can put a vfio-user-pci stub 
>> device (I don't think this has landed in qemu yet, but you can find the code 
>> at [1]) on the simulated PCI bus which will expose any software interactions 
>> with the endpoint as VFIO-user protocol messages over unix domain socket. 
>> The piece you need to bring is a VFIO-user server that handles these 
>> messages. Its task is basically translating between VFIO-user and TLP and 
>> then injecting TLP into your hardware design.
>
> Yes, If the pci host controller you said can be implemented, I can achieve my 
> goal.

I meant to say that the existing PCIe host controller implementations
in qemu can be used as is.

>
> To begin with, I'll investigate the vfio and libvfio-user.  Thanks!.
>
> [1] https://www.qemu.org/docs/master/system/multi-process.html
>
> Best,
> Shunsuke
>>
>>
>> [1] https://github.com/oracle/qemu/tree/vfio-user-p3.1 - I believe that's 
>> the latest version, Jagannathan Raman will know best
>> [2] https://github.com/nutanix/libvfio-user
>>
>