Re: [Qemu-devel] [RFC v5 0/8] ARM SMMUv3 Emulation Support

[Tomasz Nowicki]

Hi Eric,

Just letting you know that I am facing another issue with the following 
setup:
1. host (4.12 kernel & 64K page) and VM (4.12 kernel & 64K page)
2. QEMU + -netdev type=tap,ifname=tap,id=net0 -device 
virtio-net-pci,netdev=net0,iommu_platform,disable-modern=off,disable-legacy=on
2. On VM, I allocate some huge pages and run DPDK testpmd app:
# echo 4 > /sys/kernel/mm/hugepages/hugepages-524288kB/nr_hugepages
# ./dpdk/usertools/dpdk-devbind.py -b vfio-pci  0000:00:02.0
# ./dpdk/build/app/testpmd -l 0-13 -n 4 -w 0000:00:02.0 -- 
--disable-hw-vlan-filter --disable-rss -i
EAL: Detected 14 lcore(s)
EAL: Probing VFIO support...
EAL: VFIO support initialized
EAL: PCI device 0000:00:02.0 on NUMA socket -1
EAL:   probe driver: 1af4:1041 net_virtio
EAL:   using IOMMU type 1 (Type 1)
EAL: iommu_map_dma vaddr ffff20000000 size 80000000 iova 120000000
EAL: Can't write to PCI bar (0) : offset (12)
EAL: Can't read from PCI bar (0) : offset (12)
EAL: Can't read from PCI bar (0) : offset (12)
EAL: Can't write to PCI bar (0) : offset (12)
EAL: Can't read from PCI bar (0) : offset (12)
EAL: Can't write to PCI bar (0) : offset (12)
EAL: Can't read from PCI bar (0) : offset (0)
EAL: Can't write to PCI bar (0) : offset (4)
EAL: Can't write to PCI bar (0) : offset (14)
EAL: Can't write to PCI bar (0) : offset (e)
EAL: Can't read from PCI bar (0) : offset (c)
EAL: Requested device 0000:00:02.0 cannot be used
EAL: No probed ethernet devices
Interactive-mode selected
USER1: create a new mbuf pool <mbuf_pool_socket_0>: n=251456, size=2176, 
socket=0

When VM uses *4K pages* the same setup works fine. I will work on this 
but please let me know in case you already know what is going on.

Thanks,
Tomasz


On 09.07.2017 22:51, Eric Auger wrote:
> This series implements the emulation code for ARM SMMUv3.
> This is the continuation of Prem's work [1].
>
> This v5 mainly brings VFIO integration in DT mode. On guest kernel
> side, this requires a quirk [1] to force TLB invalidation on map.
>
> The following changes also are noticeable:
> - fix SMMU_CMDQ_CONS offset
> - adds dma-coherent dt property which fixes the unhandled command
>    opcode bug.
> - implements block PTE
>
> The smmu is instantiated when passing the smmu option to machvirt:
> "-M virt-2.10,smmu"
>
> As I haven't split the code yet so that it can be easily reviewable
> I don't expect deep reviews at this stage. Also the implementation may
> be largely sub-optimal.
>
> Tested Use Cases:
> - booted a guest in dt and acpi mode with an iommu_platform
>    virtio-net-pci device (using dma ops). Tested with the following
>    guest combinations: 4K page - 39 bit VA, 4K - 48b, 64K - 39b,
>    64K - 48b.
> - booted a guest (featuring [1]) with PCIe passthrough'ed PCIe devices:
>    - AMD Overdrive and igbvf passthrough (using gsi direct mapping)
>    - Cavium ThunderX and ixgbevf passthrough (using KVM MSI routing)
>
> Unfortunately I have not been able to run DPDK testpmd yet on guest side.
> The problem I see is the user space driver dma-maps a huge area
> and this causes plenty of CMDQ_OP_TLBI_NH_VA commands to be sent
> (tlbi-on-map) which are sent for each page whereas the dma-map covers a
> huge page. I will work on this issue for next version.
>
> Known limitations:
> - no VMSAv8-32 suport
> - no nested stage support (S1 + S2)
> - no support for HYP mappings
> - register fine emulation, commands, interrupts and errors were
>    not accurately tested. Handling is sufficient to run use cases
>    described hereafter though.
>
> Best Regards
>
> Eric
>
> This series can be found at:
> v5: https://github.com/eauger/qemu/tree/v2.9-SMMU-v5
> v4: https://github.com/eauger/qemu/tree/v2.9-SMMU-v4
>
> References:
> [1] [RFC 0/2] arm-smmu-v3 tlbi-on-map option
> [2] Prem's last iteration:
> - https://lists.gnu.org/archive/html/qemu-devel/2016-08/msg03531.html
>
> History:
> v4 -> v5:
> - initial_level now part of SMMUTransCfg
> - smmu_page_walk_64 takes into account the max input size
> - implement sys->iommu_ops.replay and sys->iommu_ops.notify_flag_changed
> - smmuv3_translate: bug fix: don't walk on bypass
> - smmu_update_qreg: fix PROD index update
> - I did not yet address Peter's comments as the code is not mature enough
>    to be split into sub patches.
>
> v3 -> v4 [Eric]:
> - page table walk rewritten to allow scan of the page table within a
>    range of IOVA. This prepares for VFIO integration and replay.
> - configuration parsing partially reworked.
> - do not advertise unsupported/untested features: S2, S1 + S2, HYP,
>    PRI, ATS, ..
> - added ACPI table generation
> - migrated to dynamic traces
> - mingw compilation fix
>
> v2 -> v3 [Eric]:
> - rebased on 2.9
> - mostly code and patch reorganization to ease the review process
> - optional patches removed. They may be handled separately. I am currently
>    working on ACPI enablement.
> - optional instantiation of the smmu in mach-virt
> - removed [2/9] (fdt functions) since not mandated
> - start splitting main patch into base and derived object
> - no new function feature added
>
> v1 -> v2 [Prem]:
> - Adopted review comments from Eric Auger
>          - Make SMMU_DPRINTF to internally call qemu_log
>              (since translation requests are too many, we need control
>               on the type of log we want)
>          - SMMUTransCfg modified to suite simplicity
>          - Change RegInfo to uint64 register array
>          - Code cleanup
>          - Test cleanups
> - Reshuffled patches
>
> v0 -> v1 [Prem]:
> - As per SMMUv3 spec 16.0 (only is_ste_consistant() is noticeable)
> - Reworked register access/update logic
> - Factored out translation code for
>          - single point bug fix
>          - sharing/removal in future
> - (optional) Unit tests added, with PCI test device
>          - S1 with 4k/64k, S1+S2 with 4k/64k
>          - (S1 or S2) only can be verified by Linux 4.7 driver
>          - (optional) Priliminary ACPI support
>
> v0 [Prem]:
> - Implements SMMUv3 spec 11.0
> - Supported for PCIe devices,
> - Command Queue and Event Queue supported
> - LPAE only, S1 is supported and Tested, S2 not tested
> - BE mode Translation not supported
> - IRQ support (legacy, no MSI)
> - Tested with DPDK and e1000
>
>
> Eric Auger (5):
>    hw/arm/smmu-common: smmu base class
>    hw/arm/virt: Add 2.10 machine type
>    hw/arm/virt: Add tlbi-on-map property to the smmuv3 node
>    target/arm/kvm: Translate the MSI doorbell in kvm_arch_fixup_msi_route
>    hw/arm/smmuv3: VFIO integration
>
> Prem Mallappa (3):
>    hw/arm/smmuv3: smmuv3 emulation model
>    hw/arm/virt: Add SMMUv3 to the virt board
>    hw/arm/virt-acpi-build: Add smmuv3 node in IORT table
>
>   default-configs/aarch64-softmmu.mak |    1 +
>   hw/arm/Makefile.objs                |    1 +
>   hw/arm/smmu-common.c                |  474 +++++++++++++
>   hw/arm/smmu-internal.h              |   89 +++
>   hw/arm/smmuv3-internal.h            |  651 ++++++++++++++++++
>   hw/arm/smmuv3.c                     | 1256 +++++++++++++++++++++++++++++++++++
>   hw/arm/trace-events                 |   54 ++
>   hw/arm/virt-acpi-build.c            |   56 +-
>   hw/arm/virt.c                       |  111 +++-
>   include/hw/acpi/acpi-defs.h         |   15 +
>   include/hw/arm/smmu-common.h        |  127 ++++
>   include/hw/arm/smmuv3.h             |   87 +++
>   include/hw/arm/virt.h               |    5 +
>   target/arm/kvm.c                    |   28 +
>   target/arm/trace-events             |    3 +
>   15 files changed, 2949 insertions(+), 9 deletions(-)
>   create mode 100644 hw/arm/smmu-common.c
>   create mode 100644 hw/arm/smmu-internal.h
>   create mode 100644 hw/arm/smmuv3-internal.h
>   create mode 100644 hw/arm/smmuv3.c
>   create mode 100644 include/hw/arm/smmu-common.h
>   create mode 100644 include/hw/arm/smmuv3.h