Re: [Qemu-devel] [PULL 14/28] exec: make address spaces 64-bit wide

[Alex Williamson]

On Mon, 2014-01-20 at 19:04 +0200, Michael S. Tsirkin wrote:
> On Mon, Jan 20, 2014 at 09:45:25AM -0700, Alex Williamson wrote:
> > On Mon, 2014-01-20 at 11:20 -0500, Mike Day wrote:
> > > Do you know which device is writing to the BAR below? From the trace
> > > it appears it should be restoring the memory address to the BAR after
> > > writing all 1s to the BAR and reading back the contents. (the protocol
> > > for finding the length of the bar memory.)
> > 
> > The guest itself is writing the the BARs.  This is a standard sizing
> > operation by the guest.
> 
> Question is maybe device memory should be disabled?
> Does windows do this too (sizing when memory enabled)?

Per the spec I would have expected memory & I/O to be disabled on the
device during a sizing operation, but that's not the case here.  I
thought you were the one that said Linux doesn't do this because some
devices don't properly re-enable.  I'm not sure how it would change our
approach to this to know whether Windows behaves the same since sizing
while disabled is not an issue and we apparently need to support sizing
while enabled regardless.  Thanks,

Alex

> > > On Thu, Jan 9, 2014 at 12:24 PM, Alex Williamson
> > > <address@hidden> wrote:
> > > > On Wed, 2013-12-11 at 20:30 +0200, Michael S. Tsirkin wrote:
> > > >> From: Paolo Bonzini <address@hidden>
> > > > vfio: vfio_pci_read_config(0000:01:10.0, @0x10, len=0x4) febe0004
> > > > (save lower 32bits of BAR)
> > > > vfio: vfio_pci_write_config(0000:01:10.0, @0x10, 0xffffffff, len=0x4)
> > > > (write mask to BAR)
> > > 
> > > Here the device should restore the memory address (original contents)
> > > to the BAR.
> > 
> > Sorry if it's not clear, the trace here is what the vfio-pci driver
> > sees.  We're just observing the sizing operation of the guest, therefore
> > we see:
> > 
> > 1) orig = read()
> > 2) write(0xffffffff)
> > 3) size_mask = read()
> > 4) write(orig)
> > 
> > We're only at step 2)
> > 
> > > > vfio: region_del febe0000 - febe3fff
> > > > (memory region gets unmapped)
> > > > vfio: vfio_pci_read_config(0000:01:10.0, @0x10, len=0x4) ffffc004
> > > > (read size mask)
> > 
> > step 3)
> > 
> > > > vfio: vfio_pci_write_config(0000:01:10.0, @0x10, 0xfebe0004, len=0x4)
> > > > (restore BAR)
> > 
> > step 4)
> > 
> > > > vfio: region_add febe0000 - febe3fff [0x7fcf3654d000]
> > > > (memory region re-mapped)
> > > > vfio: vfio_pci_read_config(0000:01:10.0, @0x14, len=0x4) 0
> > > > (save upper 32bits of BAR)
> > > > vfio: vfio_pci_write_config(0000:01:10.0, @0x14, 0xffffffff, len=0x4)
> > > > (write mask to BAR)
> > > 
> > > and here ...
> > 
> > This is the same as above to the next BAR, which is the upper 32bits of
> > the 64bit BAR.
> > 
> > > > vfio: region_del febe0000 - febe3fff
> > > > (memory region gets unmapped)
> > > > vfio: region_add fffffffffebe0000 - fffffffffebe3fff [0x7fcf3654d000]
> > > > (memory region gets re-mapped with new address)
> > > > qemu-system-x86_64: vfio_dma_map(0x7fcf38861710, 0xfffffffffebe0000, 
> > > > 0x4000, 0x7fcf3654d000) = -14 (Bad address)
> > > > (iommu barfs because it can only handle 48bit physical addresses)
> > > 
> > > I looked around some but I couldn't find an obvious culprit. Could it
> > > be that the BAR is getting unmapped automatically due to
> > > x-intx-mmap-timeout-ms before the device has a chance to finish
> > > restoring the correct value to the BAR?
> > 
> > No, this is simply the guest sizing the BAR, this is not an internally
> > generated operation.  The INTx emulation isn't used here as KVM
> > acceleration is enabled.  That also only toggles the enable setting on
> > the mmap'd MemoryRegion, it doesn't change the address it's mapped to.
> > Thanks,
> > 
> > Alex