Re: [Qemu-discuss] Error starting domain: vfio-pci

[Gabriel Laupre]

> AppArmor is Debian based, but that doesn't stop you from running/installing it on other distros

> $ dmesg | grep apparmor

The service isn't started nor present in my machine. Thank you for the hint though.

I am still stuck with this error and it is not dependent of my Intel NIC, I used another one and I still got the same error message.

ps: tks to remind me to send to the mailing list:)

2015-05-21 12:32 GMT-07:00 Karl Apsite <address@hidden>:

AppArmor is Debian based, but that doesn't stop you from running/installing it
on other distros

$ dmesg | grep apparmor

On 05/20/2015 02:56 PM, Bandan Das wrote:
> Gabriel Laupre <address@hidden> writes:
>
>> Mmmhh,
>> My SELinux is disabled. Is Apparmor not only on debian/ubuntu and suse, am
>> I wrong?  I have no idea on that :)
>
> Yeah, me neither :) I am just trying to rule out all possibilities.
>
>> 2015-05-20 11:23 GMT-07:00 Bandan Das <address@hidden>:
>>
>>> Oh and one more thing! You already answered before but just wanted to
>>> confirm
>>> that you don't have apparmor running, right ?
>>>
>>> Bandan Das <address@hidden> writes:
>>>
>>>> Gabriel Laupre <address@hidden> writes:
>>>>
>>>>>> Yes, indeed it is. What distro is this ? Do you have SELinux or any
>>> other
>>>>> security feature enabled ? Can you please verify that the file has a
>>>>> appropriate label if SELinux is enabled ? (ls -lZ /dev/vfio/vfio)
>>>>> My distrib:
>>>>> address@hidden ~]# cat /proc/version
>>>>> Linux version 3.10.0-229.1.2.el7.x86_64 (
>>> address@hidden)
>>>>> (gcc version 4.8.2 20140120 (Red Hat 4.8.2-16) (GCC) ) #1 SMP Fri Mar 27
>>>>> 03:04:26 UTC 2015
>>>>> address@hidden ~]# cat /etc/centos-release
>>>>> CentOS Linux release 7.1.1503 (Core)
>>>>>
>>>>> address@hidden ~]# ls -lZ /dev/vfio/vfio
>>>>> crw-rw-rw- root root ?                                /dev/vfio/vfio
>>>>>
>>>>> SELinux is disabled:
>>>>> address@hidden ~]# getenforce
>>>>> Disabled
>>>>>
>>>>> I guess no other security feature is enabled that I am aware of. I once
>>> had
>>>>> a message saying that it can be one of the following issues (listing the
>>>>> 5). So I guess it can be any combination of those issues, even something
>>>>> completely different.
>>>>
>>>> Ugh, I am out of options! Can you please try a few more things: Can you
>>> try
>>>> running qemu directly and see if you see the same behavior ? If you still
>>>> haven't tried running as root, please try that too. Also, please check
>>> dmesg
>>>> for any vfio related errors.
>>>>
>>>>> libvirtError: internal error: process exited while connecting to
>>>>> monitor: 2015-05-19T21:46:21.935043Z qemu-kvm: -device
>>>>> vfio-pci,host=04:10.4,id=hostdev0,bus=pci.0,addr=0x3: vfio: failed to
>>>>> open /dev/vfio/vfio: Operation not permitted
>>>> Well, this is the first error from vfio_connect_container() when it does:
>>>>     fd = qemu_open("/dev/vfio/vfio", O_RDWR);
>>>>     if (fd < 0) {
>>>>         error_report("vfio: failed to open /dev/vfio/vfio: %m");
>>>>         ret = -errno;
>>>> ...
>>>>
>>>> The rest are followup errors printed from the other functions in the
>>>> stack due to this error.
>>>>
>>>> Bandan
>>>>
>>>>> 2015-05-19T21:46:21.935091Z qemu-kvm: -device
>>>>> vfio-pci,host=04:10.4,id=hostdev0,bus=pci.0,addr=0x3: vfio: failed to
>>>>> setup container for group 24
>>>>> 2015-05-19T21:46:21.935107Z qemu-kvm: -device
>>>>> vfio-pci,host=04:10.4,id=hostdev0,bus=pci.0,addr=0x3: vfio: failed to
>>>>> get group 24
>>>>> 2015-05-19T21:46:21.935135Z qemu-kvm: -device
>>>>> vfio-pci,host=04:10.4,id=hostdev0,bus=pci.0,addr=0x3: Device
>>>>> initialization failed.
>>>>> 2015-05-19T21:46:21.935157Z qemu-kvm: -device
>>>>> vfio-pci,host=04:10.4,id=hostdev0,bus=pci.0,addr=0x3: Device
>>>>> 'vfio-pci' could not be initialized
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> 2015-05-19 21:17 GMT-08:00 Bandan Das <address@hidden>:
>>>>>
>>>>>>
>>>>>>> On May 20, 2015, at 12:29 AM, Gabriel Laupre <address@hidden>
>>> wrote:
>>>>>>>
>>>>>>> Thank Bandan,
>>>>>>>
>>>>>>>
>>>>>>>> Assuming you are on an intel box, have you booted your kernel with
>>>>>> intel_iommu=on ?
>>>>>>> Yes, I have booted my kernel with the intel_iommu=on. (I don't
>>> remember
>>>>>> how to check that now though ^^)
>>>>>>>
>>>>>>>> Please paste the output of dmesg | grep -e DMAR -e IOMMU ?
>>>>>>> address@hidden ~]# dmesg | grep -e DMAR -e IOMMU
>>>>>>> [    0.000000] ACPI: DMAR 00000000bf79e0c0 00118 (v01    AMI  OEMDMAR
>>>>>> 00000001 MSFT 00000097)
>>>>>>> [    0.000000] Intel-IOMMU: enabled
>>>>>>> [    0.039149] dmar: IOMMU 0: reg_base_addr fbffe000 ver 1:0 cap
>>>>>> c90780106f0462 ecap f020f6
>>>>>>> [    0.550126] IOMMU 0 0xfbffe000: using Queued invalidation
>>>>>>> [    0.550131] IOMMU: Setting RMRR:
>>>>>>> [    0.550149] IOMMU: Setting identity map for device 0000:00:1a.0
>>>>>> [0xbf7ec000 - 0xbf7fffff]
>>>>>>> [    0.550184] IOMMU: Setting identity map for device 0000:00:1a.1
>>>>>> [0xbf7ec000 - 0xbf7fffff]
>>>>>>> [    0.550211] IOMMU: Setting identity map for device 0000:00:1a.2
>>>>>> [0xbf7ec000 - 0xbf7fffff]
>>>>>>> [    0.550241] IOMMU: Setting identity map for device 0000:00:1a.7
>>>>>> [0xbf7ec000 - 0xbf7fffff]
>>>>>>> [    0.550272] IOMMU: Setting identity map for device 0000:00:1d.0
>>>>>> [0xbf7ec000 - 0xbf7fffff]
>>>>>>> [    0.550302] IOMMU: Setting identity map for device 0000:00:1d.1
>>>>>> [0xbf7ec000 - 0xbf7fffff]
>>>>>>> [    0.550329] IOMMU: Setting identity map for device 0000:00:1d.2
>>>>>> [0xbf7ec000 - 0xbf7fffff]
>>>>>>> [    0.550358] IOMMU: Setting identity map for device 0000:00:1d.7
>>>>>> [0xbf7ec000 - 0xbf7fffff]
>>>>>>> [    0.550375] IOMMU: Setting identity map for device 0000:00:1a.0
>>>>>> [0xec000 - 0xeffff]
>>>>>>> [    0.550387] IOMMU: Setting identity map for device 0000:00:1a.1
>>>>>> [0xec000 - 0xeffff]
>>>>>>> [    0.550399] IOMMU: Setting identity map for device 0000:00:1a.2
>>>>>> [0xec000 - 0xeffff]
>>>>>>> [    0.550410] IOMMU: Setting identity map for device 0000:00:1a.7
>>>>>> [0xec000 - 0xeffff]
>>>>>>> [    0.550421] IOMMU: Setting identity map for device 0000:00:1d.0
>>>>>> [0xec000 - 0xeffff]
>>>>>>> [    0.550433] IOMMU: Setting identity map for device 0000:00:1d.1
>>>>>> [0xec000 - 0xeffff]
>>>>>>> [    0.550444] IOMMU: Setting identity map for device 0000:00:1d.2
>>>>>> [0xec000 - 0xeffff]
>>>>>>> [    0.550458] IOMMU: Setting identity map for device 0000:00:1d.7
>>>>>> [0xec000 - 0xeffff]
>>>>>>> [    0.550471] IOMMU: Prepare 0-16MiB unity mapping for LPC
>>>>>>> [    0.550483] IOMMU: Setting identity map for device 0000:00:1f.0
>>> [0x0
>>>>>> - 0xffffff]
>>>>>>>
>>>>>>
>>>>>> Yeah, this looks ok. Actually, taking a second look, I can’t think of
>>>>>> anyway how this could be related to file permissions on /dev/vfio/vfio.
>>>>>>
>>>>>>>> Why does opening /dev/vfio/vfio fail ? Can you please confirm that
>>> you
>>>>>> have read/write permissions as the user you are trying to run ?
>>>>>>> address@hidden ~]# cd /dev/vfio/
>>>>>>> address@hidden vfio]# ls -la | grep vfio
>>>>>>> crw-rw-rw-  1 root root 10, 196 May 18 11:54 vfio
>>>>>>> The right should be okay I guess.
>>>>>>>
>>>>>> Yes, indeed it is. What distro is this ? Do you have SELinux or any
>>> other
>>>>>> security feature enabled ? Can you please verify that the file has a
>>>>>> appropriate label if SELinux is enabled ? (ls -lZ /dev/vfio/vfio)
>>>>>>
>>>>>> Bandan
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> 2015-05-19 18:54 GMT-08:00 Bandan Das <address@hidden>:
>>>>>>>
>>>>>>> Hello Gabriel,
>>>>>>>
>>>>>>>> On May 19, 2015, at 8:03 PM, Gabriel Laupre <address@hidden>
>>> wrote:
>>>>>>>>
>>>>>>>> Hello everyone,
>>>>>>>>
>>>>>>>> I am using a Centos 7.1 machine with the kernel 3.10.229. I want to
>>>>>> use my host with SR-IOV to use a virtual function on my NIC as the
>>> vNIC in
>>>>>> my new VM.
>>>>>>>>
>>>>>>>> I have an instance started with a old NIC using macvtap that I
>>> want to
>>>>>> change. I am using the
>>>>>>>> virsh edit instance-00000034
>>>>>>>> command to edit the XML configuration to add the new device I want
>>> to
>>>>>> attach.
>>>>>>> …
>>>>>>> Assuming you are on an intel box, have you booted your kernel with
>>>>>> intel_iommu=on ?
>>>>>>> Please paste the output of dmesg | grep -e DMAR -e IOMMU ?
>>>>>>>
>>>>>>>> When I try to reboot the VM I get this error:
>>>>>>>> Error starting domain: internal error: process exited while
>>> connecting
>>>>>> to monitor: 2015-05-19T21:46:21.935043Z qemu-kvm: -device
>>>>>> vfio-pci,host=04:10.4,id=hostdev0,bus=pci.0,addr=0x3: vfio: failed to
>>> open
>>>>>> /dev/vfio/vfio: Operation not permitted
>>>>>>> Why does opening /dev/vfio/vfio fail ? Can you please confirm that
>>> you
>>>>>> have read/write permissions as the user you are trying to run ?
>>>>>>>
>>>>>>>> 2015-05-19T21:46:21.935091Z qemu-kvm: -device
>>>>>> vfio-pci,host=04:10.4,id=hostdev0,bus=pci.0,addr=0x3: vfio: failed to
>>> setup
>>>>>> container for group 24
>>>>>>>> 2015-05-19T21:46:21.935107Z qemu-kvm: -device
>>>>>> vfio-pci,host=04:10.4,id=hostdev0,bus=pci.0,addr=0x3: vfio: failed to
>>> get
>>>>>> group 24
>>>>>>>> 2015-05-19T21:46:21.935135Z qemu-kvm: -device
>>>>>> vfio-pci,host=04:10.4,id=hostdev0,bus=pci.0,addr=0x3: Device
>>> initialization
>>>>>> failed.
>>>>>>>> 2015-05-19T21:46:21.935157Z qemu-kvm: -device
>>>>>> vfio-pci,host=04:10.4,id=hostdev0,bus=pci.0,addr=0x3: Device 'vfio-pci'
>>>>>> could not be initialized
>>>>>>>>
>>>>>>>> total Trace here: http://sprunge.us/XZFB
>>>>>>>>
>>>>>>>> Any idea how to fix that?
>>>>>>>>
>>>>>>>> Thank you very much :)
>>>>>>>>
>>>>>>>> Gabriel
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>
>