Re: [Qemu-discuss] Error starting domain: vfio-pci

[Bandan Das]

Oh and one more thing! You already answered before but just wanted to confirm
that you don't have apparmor running, right ?

Bandan Das <address@hidden> writes:

> Gabriel Laupre <address@hidden> writes:
>
>>> Yes, indeed it is. What distro is this ? Do you have SELinux or any other
>> security feature enabled ? Can you please verify that the file has a
>> appropriate label if SELinux is enabled ? (ls -lZ /dev/vfio/vfio)
>> My distrib:
>> address@hidden ~]# cat /proc/version
>> Linux version 3.10.0-229.1.2.el7.x86_64 (address@hidden)
>> (gcc version 4.8.2 20140120 (Red Hat 4.8.2-16) (GCC) ) #1 SMP Fri Mar 27
>> 03:04:26 UTC 2015
>> address@hidden ~]# cat /etc/centos-release
>> CentOS Linux release 7.1.1503 (Core)
>>
>> address@hidden ~]# ls -lZ /dev/vfio/vfio
>> crw-rw-rw- root root ?                                /dev/vfio/vfio
>>
>> SELinux is disabled:
>> address@hidden ~]# getenforce
>> Disabled
>>
>> I guess no other security feature is enabled that I am aware of. I once had
>> a message saying that it can be one of the following issues (listing the
>> 5). So I guess it can be any combination of those issues, even something
>> completely different.
>
> Ugh, I am out of options! Can you please try a few more things: Can you try
> running qemu directly and see if you see the same behavior ? If you still
> haven't tried running as root, please try that too. Also, please check dmesg
> for any vfio related errors.
>
>> libvirtError: internal error: process exited while connecting to
>> monitor: 2015-05-19T21:46:21.935043Z qemu-kvm: -device
>> vfio-pci,host=04:10.4,id=hostdev0,bus=pci.0,addr=0x3: vfio: failed to
>> open /dev/vfio/vfio: Operation not permitted
> Well, this is the first error from vfio_connect_container() when it does:
>     fd = qemu_open("/dev/vfio/vfio", O_RDWR);
>     if (fd < 0) {
>         error_report("vfio: failed to open /dev/vfio/vfio: %m");
>         ret = -errno;
> ...
>
> The rest are followup errors printed from the other functions in the
> stack due to this error.
>
> Bandan
>
>> 2015-05-19T21:46:21.935091Z qemu-kvm: -device
>> vfio-pci,host=04:10.4,id=hostdev0,bus=pci.0,addr=0x3: vfio: failed to
>> setup container for group 24
>> 2015-05-19T21:46:21.935107Z qemu-kvm: -device
>> vfio-pci,host=04:10.4,id=hostdev0,bus=pci.0,addr=0x3: vfio: failed to
>> get group 24
>> 2015-05-19T21:46:21.935135Z qemu-kvm: -device
>> vfio-pci,host=04:10.4,id=hostdev0,bus=pci.0,addr=0x3: Device
>> initialization failed.
>> 2015-05-19T21:46:21.935157Z qemu-kvm: -device
>> vfio-pci,host=04:10.4,id=hostdev0,bus=pci.0,addr=0x3: Device
>> 'vfio-pci' could not be initialized
>>
>>
>>
>>
>> 2015-05-19 21:17 GMT-08:00 Bandan Das <address@hidden>:
>>
>>>
>>> > On May 20, 2015, at 12:29 AM, Gabriel Laupre <address@hidden> wrote:
>>> >
>>> > Thank Bandan,
>>> >
>>> >
>>> > > Assuming you are on an intel box, have you booted your kernel with
>>> intel_iommu=on ?
>>> > Yes, I have booted my kernel with the intel_iommu=on. (I don't remember
>>> how to check that now though ^^)
>>> >
>>> > > Please paste the output of dmesg | grep -e DMAR -e IOMMU ?
>>> > address@hidden ~]# dmesg | grep -e DMAR -e IOMMU
>>> > [    0.000000] ACPI: DMAR 00000000bf79e0c0 00118 (v01    AMI  OEMDMAR
>>> 00000001 MSFT 00000097)
>>> > [    0.000000] Intel-IOMMU: enabled
>>> > [    0.039149] dmar: IOMMU 0: reg_base_addr fbffe000 ver 1:0 cap
>>> c90780106f0462 ecap f020f6
>>> > [    0.550126] IOMMU 0 0xfbffe000: using Queued invalidation
>>> > [    0.550131] IOMMU: Setting RMRR:
>>> > [    0.550149] IOMMU: Setting identity map for device 0000:00:1a.0
>>> [0xbf7ec000 - 0xbf7fffff]
>>> > [    0.550184] IOMMU: Setting identity map for device 0000:00:1a.1
>>> [0xbf7ec000 - 0xbf7fffff]
>>> > [    0.550211] IOMMU: Setting identity map for device 0000:00:1a.2
>>> [0xbf7ec000 - 0xbf7fffff]
>>> > [    0.550241] IOMMU: Setting identity map for device 0000:00:1a.7
>>> [0xbf7ec000 - 0xbf7fffff]
>>> > [    0.550272] IOMMU: Setting identity map for device 0000:00:1d.0
>>> [0xbf7ec000 - 0xbf7fffff]
>>> > [    0.550302] IOMMU: Setting identity map for device 0000:00:1d.1
>>> [0xbf7ec000 - 0xbf7fffff]
>>> > [    0.550329] IOMMU: Setting identity map for device 0000:00:1d.2
>>> [0xbf7ec000 - 0xbf7fffff]
>>> > [    0.550358] IOMMU: Setting identity map for device 0000:00:1d.7
>>> [0xbf7ec000 - 0xbf7fffff]
>>> > [    0.550375] IOMMU: Setting identity map for device 0000:00:1a.0
>>> [0xec000 - 0xeffff]
>>> > [    0.550387] IOMMU: Setting identity map for device 0000:00:1a.1
>>> [0xec000 - 0xeffff]
>>> > [    0.550399] IOMMU: Setting identity map for device 0000:00:1a.2
>>> [0xec000 - 0xeffff]
>>> > [    0.550410] IOMMU: Setting identity map for device 0000:00:1a.7
>>> [0xec000 - 0xeffff]
>>> > [    0.550421] IOMMU: Setting identity map for device 0000:00:1d.0
>>> [0xec000 - 0xeffff]
>>> > [    0.550433] IOMMU: Setting identity map for device 0000:00:1d.1
>>> [0xec000 - 0xeffff]
>>> > [    0.550444] IOMMU: Setting identity map for device 0000:00:1d.2
>>> [0xec000 - 0xeffff]
>>> > [    0.550458] IOMMU: Setting identity map for device 0000:00:1d.7
>>> [0xec000 - 0xeffff]
>>> > [    0.550471] IOMMU: Prepare 0-16MiB unity mapping for LPC
>>> > [    0.550483] IOMMU: Setting identity map for device 0000:00:1f.0 [0x0
>>> - 0xffffff]
>>> >
>>>
>>> Yeah, this looks ok. Actually, taking a second look, I can’t think of
>>> anyway how this could be related to file permissions on /dev/vfio/vfio.
>>>
>>> > > Why does opening /dev/vfio/vfio fail ? Can you please confirm that you
>>> have read/write permissions as the user you are trying to run ?
>>> > address@hidden ~]# cd /dev/vfio/
>>> > address@hidden vfio]# ls -la | grep vfio
>>> > crw-rw-rw-  1 root root 10, 196 May 18 11:54 vfio
>>> > The right should be okay I guess.
>>> >
>>> Yes, indeed it is. What distro is this ? Do you have SELinux or any other
>>> security feature enabled ? Can you please verify that the file has a
>>> appropriate label if SELinux is enabled ? (ls -lZ /dev/vfio/vfio)
>>>
>>> Bandan
>>> >
>>> >
>>> >
>>> > 2015-05-19 18:54 GMT-08:00 Bandan Das <address@hidden>:
>>> >
>>> > Hello Gabriel,
>>> >
>>> > > On May 19, 2015, at 8:03 PM, Gabriel Laupre <address@hidden> wrote:
>>> > >
>>> > > Hello everyone,
>>> > >
>>> > > I am using a Centos 7.1 machine with the kernel 3.10.229. I want to
>>> use my host with SR-IOV to use a virtual function on my NIC as the vNIC in
>>> my new VM.
>>> > >
>>> > > I have an instance started with a old NIC using macvtap that I want to
>>> change. I am using the
>>> > > virsh edit instance-00000034
>>> > > command to edit the XML configuration to add the new device I want to
>>> attach.
>>> > …
>>> > Assuming you are on an intel box, have you booted your kernel with
>>> intel_iommu=on ?
>>> > Please paste the output of dmesg | grep -e DMAR -e IOMMU ?
>>> >
>>> > > When I try to reboot the VM I get this error:
>>> > > Error starting domain: internal error: process exited while connecting
>>> to monitor: 2015-05-19T21:46:21.935043Z qemu-kvm: -device
>>> vfio-pci,host=04:10.4,id=hostdev0,bus=pci.0,addr=0x3: vfio: failed to open
>>> /dev/vfio/vfio: Operation not permitted
>>> > Why does opening /dev/vfio/vfio fail ? Can you please confirm that you
>>> have read/write permissions as the user you are trying to run ?
>>> >
>>> > > 2015-05-19T21:46:21.935091Z qemu-kvm: -device
>>> vfio-pci,host=04:10.4,id=hostdev0,bus=pci.0,addr=0x3: vfio: failed to setup
>>> container for group 24
>>> > > 2015-05-19T21:46:21.935107Z qemu-kvm: -device
>>> vfio-pci,host=04:10.4,id=hostdev0,bus=pci.0,addr=0x3: vfio: failed to get
>>> group 24
>>> > > 2015-05-19T21:46:21.935135Z qemu-kvm: -device
>>> vfio-pci,host=04:10.4,id=hostdev0,bus=pci.0,addr=0x3: Device initialization
>>> failed.
>>> > > 2015-05-19T21:46:21.935157Z qemu-kvm: -device
>>> vfio-pci,host=04:10.4,id=hostdev0,bus=pci.0,addr=0x3: Device 'vfio-pci'
>>> could not be initialized
>>> > >
>>> > > total Trace here: http://sprunge.us/XZFB
>>> > >
>>> > > Any idea how to fix that?
>>> > >
>>> > > Thank you very much :)
>>> > >
>>> > > Gabriel
>>> > >
>>> >
>>> >
>>>
>>>