Re: [Qemu-devel] RFC: vfio API changes needed for powerpc

[Stuart Yoder]

On Tue, Apr 2, 2013 at 3:47 PM, Scott Wood <address@hidden> wrote:
> On 04/02/2013 03:38:42 PM, Stuart Yoder wrote:
>>
>> On Tue, Apr 2, 2013 at 2:39 PM, Scott Wood <address@hidden>
>> wrote:
>> > On 04/02/2013 12:32:00 PM, Yoder Stuart-B08248 wrote:
>> >>
>> >> Alex,
>> >>
>> >> We are in the process of implementing vfio-pci support for the
>> >> Freescale
>> >> IOMMU (PAMU).  It is an aperture/window-based IOMMU and is quite
>> >> different
>> >> than x86, and will involve creating a 'type 2' vfio implementation.
>> >>
>> >> For each device's DMA mappings, PAMU has an overall aperture and a
>> >> number
>> >> of windows.  All sizes and window counts must be power of 2.  To
>> >> illustrate,
>> >> below is a mapping for a 256MB guest, including guest memory (backed by
>> >> 64MB huge pages) and some windows for MSIs:
>> >>
>> >>     Total aperture: 512MB
>> >>     # of windows: 8
>> >>
>> >>     win gphys/
>> >>     #   iova        phys          size
>> >>     --- ----        ----          ----
>> >>     0   0x00000000  0xX_XX000000  64MB
>> >>     1   0x04000000  0xX_XX000000  64MB
>> >>     2   0x08000000  0xX_XX000000  64MB
>> >>     3   0x0C000000  0xX_XX000000  64MB
>> >>     4   0x10000000  0xf_fe044000  4KB    // msi bank 1
>> >>     5   0x14000000  0xf_fe045000  4KB    // msi bank 2
>> >>     6   0x18000000  0xf_fe046000  4KB    // msi bank 3
>> >>     7            -             -  disabled
>> >>
>> >> There are a couple of updates needed to the vfio user->kernel interface
>> >> that we would like your feedback on.
>> >>
>> >> 1.  IOMMU geometry
>> >>
>> >>    The kernel IOMMU driver now has an interface (see domain_set_attr,
>> >>    domain_get_attr) that lets us set the domain geometry using
>> >>    "attributes".
>> >>
>> >>    We want to expose that to user space, so envision needing a couple
>> >>    of new ioctls to do this:
>> >>         VFIO_IOMMU_SET_ATTR
>> >>         VFIO_IOMMU_GET_ATTR
>> >
>> >
>> > Note that this means attributes need to be updated for user-API
>> > appropriateness, such as using fixed-size types.
>> >
>> >
>> >> 2.   MSI window mappings
>> >>
>> >>    The more problematic question is how to deal with MSIs.  We need to
>> >>    create mappings for up to 3 MSI banks that a device may need to
>> >> target
>> >>    to generate interrupts.  The Linux MSI driver can allocate MSIs from
>> >>    the 3 banks any way it wants, and currently user space has no way of
>> >>    knowing which bank may be used for a given device.
>> >>
>> >>    There are 3 options we have discussed and would like your direction:
>> >>
>> >>    A.  Implicit mappings -- with this approach user space would not
>> >>        explicitly map MSIs.  User space would be required to set the
>> >>        geometry so that there are 3 unused windows (the last 3 windows)
>> >
>> >
>> > Where does userspace get the number "3" from?  E.g. on newer chips there
>> > are
>> > 4 MSI banks.  Maybe future chips have even more.
>>
>> Ok, then make the number 4.   The chance of more MSI banks in future chips
>> is nil,
>
>
> What makes you so sure?  Especially since you seem to be presenting this as
> not specifically an MPIC API.
>
>
>> and if it ever happened user space could adjust.
>
>
> What bit of API is going to tell it that it needs to adjust?

Haven't thought through that completely, but I guess we could add an API
to return the number of MSI banks for type 2 iommus.

>> Also, practically speaking since memory is typically allocate in powers of
>> 2 way you need to approximately double the window geometry anyway.
>
>
> Only if your existing mapping needs fit exactly in a power of two.
>
>
>> >>    B.  Explicit mapping using DMA map flags.  The idea is that a new
>> >>        flag to DMA map (VFIO_DMA_MAP_FLAG_MSI) would mean that
>> >>        a mapping is to be created for the supplied iova.  No vaddr
>> >>        is given though.  So in the above example there would be a
>> >>        a dma map at 0x10000000 for 24KB (and no vaddr).
>> >
>> >
>> > A single 24 KiB mapping wouldn't work (and why 24KB? What if only one
>> > MSI
>> > group is involved in this VFIO group?  What if four MSI groups are
>> > involved?).  You'd need to either have a naturally aligned, power-of-two
>> > sized mapping that covers exactly the pages you want to map and no more,
>> > or
>> > you'd need to create a separate mapping for each MSI bank, and due to
>> > PAMU
>> > subwindow alignment restrictions these mappings could not be contiguous
>> > in
>> > iova-space.
>>
>> You're right, a single 24KB mapping wouldn't work--  in the case of 3 MSI
>> banks
>> perhaps we could just do one 64MB*3 mapping to identify which windows
>> are used for MSIs.
>
>
> Where did the assumption of a 64MiB subwindow size come from?

The example I was using.   User space would need to create a
mapping for window_size * msi_bank_count.

Stuart