Re: [Qemu-devel] [RFC] Proposal: PCI/PCIe: inbound BAR0 emulation for PCI controller (Root Complex)

[Bhushan Bharat-R65777]

> -----Original Message-----
> From: Benjamin Herrenschmidt [mailto:address@hidden
> Sent: Friday, June 08, 2012 4:32 PM
> To: Bhushan Bharat-R65777
> Cc: address@hidden; Wood Scott-B07421; Yoder Stuart-B08248
> Subject: Re: [RFC] Proposal: PCI/PCIe: inbound BAR0 emulation for PCI 
> controller
> (Root Complex)
> 
> On Fri, 2012-06-08 at 09:03 +0000, Bhushan Bharat-R65777 wrote:
> > Hi All,
> >
> > When Freescale PCI controller configured in Root Complex mode then,
> > its configuration header (type 1) have one inbound BAR (BAR0, called
> > as CCSRBAR). And rest of BARs (inbound and outbound) are supported by
> > ATMU registers, which are outside the Type 1 configuration header.
> > This BAR0 of Type 1 configuration header always translate to CCSR
> > space and is of the size of CCSR. This BAR0 (inbound window) is
> > required for MSI interrupts support. With this window, the pci devices
> > can write to the MPIC MSI registers to generate MSI interrupt.
> 
> So you should start with giving an overview, nobody here knows what "ATMU" or
> "CCSR" are, those are device specific acronyms

ATMU - Address Translation and Mapping Unit :- This facilitate flexibility in 
defining the address maps for the external interfaces (PCI, Rapidio etc). 
This basically provides following features:
        1. Translating the local address to an external address space
        2. Translating external addresses to the local address space

CCSR - Configuration, Control, and Status Register (CCSR) Space. This is memory 
mapped system register space.

In PCI/PCIe context this can be visualized as:


---------        |----------------------------------|
Local    |       |PCI controller (Root Complex)|    |
Address  |       |   ______    _________________    | 
Space    |<----->|  |ATMU  |  |      Type 1    |    |<------PCI Address Space---
         |       |  |______|  | Config Header  |    |
         |       |            |________________|    |
---------        |----------------------------------|


> 
> As a matter of fact it looks like I misunderstood you on IRC :-) IE. I thought
> your problem was that BAR0 is 'special' as it represents the inbound memory
> window (as it does on some PPC 4xx for example and a whole other collection of
> embedded devices). You seem to indicate that it's in fact MMIO:

In some SOCs, BAR0 always maps to CCSR (MMIO) while in some new SOCs, the BAR0 
default maps to CCSR space while it can be programmed to map system memory in 
local address space.

> 
> > As far as I know, as of now no emulated PCI controller supports this
> > BAR0 in type 1 configuration header. But probably (I think so) that
> > supporting this is not of big concern, but the point is that this
> > window (BAR0) translate to mmio-regs (CCSR) and not to DDR memory.
> 
> So the 4xx case or the case where your BAR 0 actually represents system memory
> are something that is bothering me but not what you are trying to discuss 
> here.
> 
> IE. Normal BAR operations should work for an MMIO BAR, ie register it normally
> as a PCIe device and devices accessing those addresses will do the right 
> thing.
> 
> From there, AFAIK, the MSI code will simply do stl_le_phys, which I
> -believe- will hit a BAR that does MMIO decoding for those addresses, but I'll
> let people knowing qemu more in depth reply whether that's true or not.
> 
> There's very few devices with MSI support in hw/* so it's hard to test with
> anything other than virtio.

Probably testing with virtio should be ok as of now ( to put the framework in 
place).

Thanks
-Bharat
> 
> > So I have couple of concerns here:
> >
> > 1. Whenever PCI device does need DMA then these windows (inbound and
> > outbound ATMUs registers) need to used to translate pci address to
> > system physical address (Sometime we also call this as cpu address
> > space). This will probably be done by : [Qemu-devel] [PATCH 00/12]
> > IOMMU Infrastructure : patch-set ( I am trying to understand these
> > patches :-))
> 
> Yes, that's basically it. The patches allow you to add a set of routines that
> will be used for translating DMA accesses to system memory along with 
> map/unmap
> operations etc...
> 
> Beware that this only works with drivers that use the proper accessors.
> The patch series "fixes" some of them but not everything. I don't know off 
> hand
> (I don't have the code at hand right now) whether the MSI code goes through 
> that
> or not, if it does, you may need to be careful to
> -not- hit the translation layer and pass the accesses on.
> 
> > 2. Hook up this inbound BAR0 in the above patch-set to translate to
> > mmio-regs. As this would be controller specific, a callback will be
> > registered for translation. Now it will be upto the controller
> > specific code on how it translates. As this is needed only for MSI
> > interrupt, maybe, initially we do not do anything initially, till we
> > want MSI emulation in QEMU.
> 
> Well, virtio can make good use of MSI emulation ...
> 
> Cheers,
> Ben.
> 
> > Please provide your comment.
> >
> > Thanks
> > -Bharat
> >
> >
> 
>