Re: [Qemu-devel] [GPU and VFIO] qemu hang at startup, VFIO_IOMMU_MAP_DMA is extremely slow

[Liu, Yi L]

> -----Original Message-----
> From: Qemu-devel [mailto:address@hidden
> On Behalf Of Bob Chen
> Sent: Tuesday, December 26, 2017 6:30 PM
> To: address@hidden
> Subject: [Qemu-devel] [GPU and VFIO] qemu hang at startup,
> VFIO_IOMMU_MAP_DMA is extremely slow
> 
> Hi,
> 
> I have a host server with multiple GPU cards, and was assigning them to qemu
> with VFIO.
> 
> I found that when setting up the last free GPU, the qemu process would hang

Are all the GPUs in the same iommu group?

> there and took almost 10 minutes before finishing startup. I made some dig by
> gdb, and found the slowest part occurred at the
> hw/vfio/common.c:vfio_dma_map function call.

This is to setup mapping and it takes time. This function would be called 
multiple
times and it will take some time. The slowest part, do you mean it takes
a long time for a single vfio_dma_map() calling or the whole passthru spends a 
lot
of time on creating mapping. If a single calling takes a lot of time, then it 
may be
a problem.

You may paste your Qemu command which might help. And the dmesg in host
would also help.

> 
> 
> static int vfio_dma_map(VFIOContainer *container, hwaddr iova, ram_addr_t
> size, void *vaddr, bool readonly) { ...
>     if (ioctl(container->fd, VFIO_IOMMU_MAP_DMA, &map) == 0 ||
>         (errno == EBUSY && vfio_dma_unmap(container, iova, size) == 0 &&
>          ioctl(container->fd, VFIO_IOMMU_MAP_DMA, &map) == 0)) {
>         return 0;
>     }
> ...
> }
> 
> 
> The hang was enable to reproduce on one of my hosts, I was setting up a 4GB
> memory VM, while the host still had 16GB free. GPU physical mem is 8G.

Does it happen when you only assign a single GPU?

> Also, this phenomenon was observed on other hosts occasionally, and the
> similarity is that they always happened on the last free GPU.
> 
> 
> Full stack trace file is attached. Looking forward for you help, thanks
> 
> 
> - Bob

Regards,
Yi L