Re: [Qemu-arm] [Qemu-devel] [RFC v2 0/8] VIRTIO-IOMMU device

[Bharat Bhushan]

Hi Eric,

I started added replay in virtio-iommu and came across how MSI interrupts with 
work with VFIO. 
I understand that on intel this works differently but vsmmu will have same 
requirement. 
kvm-msi-irq-route are added using the msi-address to be translated by viommu 
and not the final translated address.
While currently the irqfd framework does not know about emulated iommus 
(virtio-iommu, vsmmuv3/vintel-iommu).
So in my view we have following options:
- Programming with translated address when setting up kvm-msi-irq-route
- Route the interrupts via QEMU, which is bad from performance
- vhost-virtio-iommu may solve the problem in long term

Is there any other better option I am missing?

Thanks
-Bharat

> -----Original Message-----
> From: Auger Eric [mailto:address@hidden
> Sent: Friday, June 09, 2017 5:24 PM
> To: Bharat Bhushan <address@hidden>;
> address@hidden; address@hidden;
> address@hidden; address@hidden; address@hidden;
> address@hidden; address@hidden
> Cc: address@hidden; address@hidden; address@hidden;
> address@hidden; address@hidden; address@hidden;
> address@hidden; address@hidden
> Subject: Re: [Qemu-devel] [RFC v2 0/8] VIRTIO-IOMMU device
> 
> Hi Bharat,
> 
> On 09/06/2017 13:30, Bharat Bhushan wrote:
> > Hi Eric,
> >
> >> -----Original Message-----
> >> From: Auger Eric [mailto:address@hidden
> >> Sent: Friday, June 09, 2017 12:14 PM
> >> To: Bharat Bhushan <address@hidden>;
> >> address@hidden; address@hidden;
> >> address@hidden; address@hidden; address@hidden;
> >> address@hidden; address@hidden
> >> Cc: address@hidden; address@hidden;
> address@hidden;
> >> address@hidden; address@hidden;
> >> address@hidden; address@hidden; address@hidden
> >> Subject: Re: [RFC v2 0/8] VIRTIO-IOMMU device
> >>
> >> Hi Bharat,
> >>
> >> On 09/06/2017 08:16, Bharat Bhushan wrote:
> >>> Hi Eric,
> >>>
> >>>> -----Original Message-----
> >>>> From: Eric Auger [mailto:address@hidden
> >>>> Sent: Wednesday, June 07, 2017 9:31 PM
> >>>> To: address@hidden; address@hidden;
> >>>> address@hidden; address@hidden;
> >> address@hidden;
> >>>> address@hidden; address@hidden; jean-
> >>>> address@hidden
> >>>> Cc: address@hidden; address@hidden;
> >> address@hidden;
> >>>> address@hidden; address@hidden;
> >>>> address@hidden; address@hidden; address@hidden; Bharat
> >> Bhushan
> >>>> <address@hidden>
> >>>> Subject: [RFC v2 0/8] VIRTIO-IOMMU device
> >>>>
> >>>> This series implements the virtio-iommu device. This is a proof of
> >>>> concept based on the virtio-iommu specification written by
> >>>> Jean-Philippe
> >> Brucker [1].
> >>>> This was tested with a guest using the virtio-iommu driver [2] and
> >>>> exposed with a virtio-net-pci using dma ops.
> >>>>
> >>>> The device gets instantiated using the "-device virtio-iommu-device"
> >>>> option. It currently works with ARM virt machine only as the
> >>>> machine must handle the dt binding between the virtio-mmio "iommu"
> >>>> node and the PCI host bridge node. ACPI booting is not yet supported.
> >>>>
> >>>> This should allow to start some benchmarking activities against
> >>>> pure emulated IOMMU (especially ARM SMMU).
> >>>
> >>> I am testing this on ARM64 and see below continuous error prints:
> >>>
> >>>   virtio_iommu_translate sid=8 is not known!!
> >>>   virtio_iommu_translate sid=8 is not known!!
> >>>   virtio_iommu_translate sid=8 is not known!!
> >>>   virtio_iommu_translate sid=8 is not known!!
> >>>   virtio_iommu_translate sid=8 is not known!!
> >>>   virtio_iommu_translate sid=8 is not known!!
> >>>   virtio_iommu_translate sid=8 is not known!!
> >>>   virtio_iommu_translate sid=8 is not known!!
> >>>   virtio_iommu_translate sid=8 is not known!!
> >>>   virtio_iommu_translate sid=8 is not known!!
> >>>
> >>>
> >>> Also in guest I do not see device-tree node with virtio-iommu.
> >> do you mean the virtio-mmio with #iommu-cells property?
> >>
> >> This one is created statically by virt machine. I would be surprised
> >> if it were not there. Are you using the virt = virt2.10 machine.
> >> Machines before do not support its instantiation.
> >>
> >> Please can you add a printf in hw/arm/virt.c create_virtio_mmio() at
> >> the moment when this node is created. Also you can add a printf in
> >> bind_virtio_iommu_device() to make sure the binding with the PCI host
> >> bridge is added on machine init done.
> >>
> >> Also worth to check, CONFIG_VIRTIO_IOMMU=y on guest side.
> >
> > It works on my side.
> Great.
> 
>  The driver config was disabled and also I was using guest kernel which was
> not have deferred-probing.
> Yes I did not mention in my cover letter the guest I have been using is based
> on Jean-Philippe's branch, featuring deferred IOMMU probing. I I have not
> tried yet with an upstream guest.
>  Now after fixing it works on my side
> > I placed some prints to see dma-map are mapping regions in virtio-iommu,
> it uses emulated iommu.
> >
> > I will continue to add VFIO support now on this and more testing !!
> 
> OK. I will do the VFIO integration first on the vsmmuv3 device as I already
> prepared the VFIO replay and hopefully we will sync ;-)
> 
> Thanks
> 
> Eric
> >
> > Thanks
> > -Bharat
> >
> >>
> >> Thanks
> >>
> >> Eric
> >>
> >>> I am using qemu-tree you mentioned below and iommu-driver patches
> >> published by Jean-P.
> >>> Qemu command line have additional ""-device virtio-iommu-device".
> >>> What
> >> I am missing ?
> >>
> >>
> >>>
> >>> Thanks
> >>> -Bharat
> >>>
> >>>>
> >>>> Best Regards
> >>>>
> >>>> Eric
> >>>>
> >>>> This series can be found at:
> >>>> https://github.com/eauger/qemu/tree/virtio-iommu-rfcv2
> >>>>
> >>>> References:
> >>>> [1] [RFC 0/3] virtio-iommu: a paravirtualized IOMMU, [2] [RFC PATCH
> >>>> linux]
> >>>> iommu: Add virtio-iommu driver [3] [RFC PATCH kvmtool 00/15] Add
> >>>> virtio- iommu
> >>>>
> >>>> History:
> >>>> v1 -> v2:
> >>>> - fix redifinition of viommu_as typedef
> >>>>
> >>>> Eric Auger (8):
> >>>>   update-linux-headers: import virtio_iommu.h
> >>>>   linux-headers: Update for virtio-iommu
> >>>>   virtio_iommu: add skeleton
> >>>>   virtio-iommu: Decode the command payload
> >>>>   virtio_iommu: Add the iommu regions
> >>>>   virtio-iommu: Implement the translation and commands
> >>>>   hw/arm/virt: Add 2.10 machine type
> >>>>   hw/arm/virt: Add virtio-iommu the virt board
> >>>>
> >>>>  hw/arm/virt.c                                 | 116 ++++-
> >>>>  hw/virtio/Makefile.objs                       |   1 +
> >>>>  hw/virtio/trace-events                        |  14 +
> >>>>  hw/virtio/virtio-iommu.c                      | 623
> >> ++++++++++++++++++++++++++
> >>>>  include/hw/arm/virt.h                         |   5 +
> >>>>  include/hw/virtio/virtio-iommu.h              |  60 +++
> >>>>  include/standard-headers/linux/virtio_ids.h   |   1 +
> >>>>  include/standard-headers/linux/virtio_iommu.h | 142 ++++++
> >>>>  linux-headers/linux/virtio_iommu.h            |   1 +
> >>>>  scripts/update-linux-headers.sh               |   3 +
> >>>>  10 files changed, 957 insertions(+), 9 deletions(-)  create mode
> >>>> 100644 hw/virtio/virtio-iommu.c  create mode 100644
> >>>> include/hw/virtio/virtio- iommu.h  create mode 100644
> >>>> include/standard- headers/linux/virtio_iommu.h  create mode 100644
> >>>> linux-headers/linux/virtio_iommu.h
> >>>>
> >>>> --
> >>>> 2.5.5
> >>>
> >