Re: [Qemu-discuss] Error starting domain: vfio-pci

[Gabriel Laupre]

> Thanks for the update. Did you get a chance to run qemu standalone without libvirt?

How can I do that, what are the purpose to do so and why to try that? (thank you ^^)

2015-05-21 13:32 GMT-07:00 Bandan Das <address@hidden>:

Gabriel Laupre <address@hidden> writes:

>> AppArmor is Debian based, but that doesn't stop you from
> running/installing it on other distros
>
>> $ dmesg | grep apparmor
>
> The service isn't started nor present in my machine. Thank you for the hint
> though.
> I am still stuck with this error and it is not dependent of my Intel NIC, I
> used another one and I still got the same error message.
> ps: tks to remind me to send to the mailing list:)

Thanks for the update. Did you get a chance to run qemu standalone without
libvirt ?

> 2015-05-21 12:32 GMT-07:00 Karl Apsite <address@hidden>:
>
>> AppArmor is Debian based, but that doesn't stop you from
>> running/installing it
>> on other distros
>>
>> $ dmesg | grep apparmor
>>
>> On 05/20/2015 02:56 PM, Bandan Das wrote:
>> > Gabriel Laupre <address@hidden> writes:
>> >
>> >> Mmmhh,
>> >> My SELinux is disabled. Is Apparmor not only on debian/ubuntu and suse,
>> am
>> >> I wrong?  I have no idea on that :)
>> >
>> > Yeah, me neither :) I am just trying to rule out all possibilities.
>> >
>> >> 2015-05-20 11:23 GMT-07:00 Bandan Das <address@hidden>:
>> >>
>> >>> Oh and one more thing! You already answered before but just wanted to
>> >>> confirm
>> >>> that you don't have apparmor running, right ?
>> >>>
>> >>> Bandan Das <address@hidden> writes:
>> >>>
>> >>>> Gabriel Laupre <address@hidden> writes:
>> >>>>
>> >>>>>> Yes, indeed it is. What distro is this ? Do you have SELinux or any
>> >>> other
>> >>>>> security feature enabled ? Can you please verify that the file has a
>> >>>>> appropriate label if SELinux is enabled ? (ls -lZ /dev/vfio/vfio)
>> >>>>> My distrib:
>> >>>>> address@hidden ~]# cat /proc/version
>> >>>>> Linux version 3.10.0-229.1.2.el7.x86_64 (
>> >>> address@hidden)
>> >>>>> (gcc version 4.8.2 20140120 (Red Hat 4.8.2-16) (GCC) ) #1 SMP Fri
>> Mar 27
>> >>>>> 03:04:26 UTC 2015
>> >>>>> address@hidden ~]# cat /etc/centos-release
>> >>>>> CentOS Linux release 7.1.1503 (Core)
>> >>>>>
>> >>>>> address@hidden ~]# ls -lZ /dev/vfio/vfio
>> >>>>> crw-rw-rw- root root ?                                /dev/vfio/vfio
>> >>>>>
>> >>>>> SELinux is disabled:
>> >>>>> address@hidden ~]# getenforce
>> >>>>> Disabled
>> >>>>>
>> >>>>> I guess no other security feature is enabled that I am aware of. I
>> once
>> >>> had
>> >>>>> a message saying that it can be one of the following issues (listing
>> the
>> >>>>> 5). So I guess it can be any combination of those issues, even
>> something
>> >>>>> completely different.
>> >>>>
>> >>>> Ugh, I am out of options! Can you please try a few more things: Can
>> you
>> >>> try
>> >>>> running qemu directly and see if you see the same behavior ? If you
>> still
>> >>>> haven't tried running as root, please try that too. Also, please check
>> >>> dmesg
>> >>>> for any vfio related errors.
>> >>>>
>> >>>>> libvirtError: internal error: process exited while connecting to
>> >>>>> monitor: 2015-05-19T21:46:21.935043Z qemu-kvm: -device
>> >>>>> vfio-pci,host=04:10.4,id=hostdev0,bus=pci.0,addr=0x3: vfio: failed to
>> >>>>> open /dev/vfio/vfio: Operation not permitted
>> >>>> Well, this is the first error from vfio_connect_container() when it
>> does:
>> >>>>     fd = qemu_open("/dev/vfio/vfio", O_RDWR);
>> >>>>     if (fd < 0) {
>> >>>>         error_report("vfio: failed to open /dev/vfio/vfio: %m");
>> >>>>         ret = -errno;
>> >>>> ...
>> >>>>
>> >>>> The rest are followup errors printed from the other functions in the
>> >>>> stack due to this error.
>> >>>>
>> >>>> Bandan
>> >>>>
>> >>>>> 2015-05-19T21:46:21.935091Z qemu-kvm: -device
>> >>>>> vfio-pci,host=04:10.4,id=hostdev0,bus=pci.0,addr=0x3: vfio: failed to
>> >>>>> setup container for group 24
>> >>>>> 2015-05-19T21:46:21.935107Z qemu-kvm: -device
>> >>>>> vfio-pci,host=04:10.4,id=hostdev0,bus=pci.0,addr=0x3: vfio: failed to
>> >>>>> get group 24
>> >>>>> 2015-05-19T21:46:21.935135Z qemu-kvm: -device
>> >>>>> vfio-pci,host=04:10.4,id=hostdev0,bus=pci.0,addr=0x3: Device
>> >>>>> initialization failed.
>> >>>>> 2015-05-19T21:46:21.935157Z qemu-kvm: -device
>> >>>>> vfio-pci,host=04:10.4,id=hostdev0,bus=pci.0,addr=0x3: Device
>> >>>>> 'vfio-pci' could not be initialized
>> >>>>>
>> >>>>>
>> >>>>>
>> >>>>>
>> >>>>> 2015-05-19 21:17 GMT-08:00 Bandan Das <address@hidden>:
>> >>>>>
>> >>>>>>
>> >>>>>>> On May 20, 2015, at 12:29 AM, Gabriel Laupre <address@hidden>
>> >>> wrote:
>> >>>>>>>
>> >>>>>>> Thank Bandan,
>> >>>>>>>
>> >>>>>>>
>> >>>>>>>> Assuming you are on an intel box, have you booted your kernel with
>> >>>>>> intel_iommu=on ?
>> >>>>>>> Yes, I have booted my kernel with the intel_iommu=on. (I don't
>> >>> remember
>> >>>>>> how to check that now though ^^)
>> >>>>>>>
>> >>>>>>>> Please paste the output of dmesg | grep -e DMAR -e IOMMU ?
>> >>>>>>> address@hidden ~]# dmesg | grep -e DMAR -e IOMMU
>> >>>>>>> [    0.000000] ACPI: DMAR 00000000bf79e0c0 00118 (v01    AMI
>> OEMDMAR
>> >>>>>> 00000001 MSFT 00000097)
>> >>>>>>> [    0.000000] Intel-IOMMU: enabled
>> >>>>>>> [    0.039149] dmar: IOMMU 0: reg_base_addr fbffe000 ver 1:0 cap
>> >>>>>> c90780106f0462 ecap f020f6
>> >>>>>>> [    0.550126] IOMMU 0 0xfbffe000: using Queued invalidation
>> >>>>>>> [    0.550131] IOMMU: Setting RMRR:
>> >>>>>>> [    0.550149] IOMMU: Setting identity map for device 0000:00:1a.0
>> >>>>>> [0xbf7ec000 - 0xbf7fffff]
>> >>>>>>> [    0.550184] IOMMU: Setting identity map for device 0000:00:1a.1
>> >>>>>> [0xbf7ec000 - 0xbf7fffff]
>> >>>>>>> [    0.550211] IOMMU: Setting identity map for device 0000:00:1a.2
>> >>>>>> [0xbf7ec000 - 0xbf7fffff]
>> >>>>>>> [    0.550241] IOMMU: Setting identity map for device 0000:00:1a.7
>> >>>>>> [0xbf7ec000 - 0xbf7fffff]
>> >>>>>>> [    0.550272] IOMMU: Setting identity map for device 0000:00:1d.0
>> >>>>>> [0xbf7ec000 - 0xbf7fffff]
>> >>>>>>> [    0.550302] IOMMU: Setting identity map for device 0000:00:1d.1
>> >>>>>> [0xbf7ec000 - 0xbf7fffff]
>> >>>>>>> [    0.550329] IOMMU: Setting identity map for device 0000:00:1d.2
>> >>>>>> [0xbf7ec000 - 0xbf7fffff]
>> >>>>>>> [    0.550358] IOMMU: Setting identity map for device 0000:00:1d.7
>> >>>>>> [0xbf7ec000 - 0xbf7fffff]
>> >>>>>>> [    0.550375] IOMMU: Setting identity map for device 0000:00:1a.0
>> >>>>>> [0xec000 - 0xeffff]
>> >>>>>>> [    0.550387] IOMMU: Setting identity map for device 0000:00:1a.1
>> >>>>>> [0xec000 - 0xeffff]
>> >>>>>>> [    0.550399] IOMMU: Setting identity map for device 0000:00:1a.2
>> >>>>>> [0xec000 - 0xeffff]
>> >>>>>>> [    0.550410] IOMMU: Setting identity map for device 0000:00:1a.7
>> >>>>>> [0xec000 - 0xeffff]
>> >>>>>>> [    0.550421] IOMMU: Setting identity map for device 0000:00:1d.0
>> >>>>>> [0xec000 - 0xeffff]
>> >>>>>>> [    0.550433] IOMMU: Setting identity map for device 0000:00:1d.1
>> >>>>>> [0xec000 - 0xeffff]
>> >>>>>>> [    0.550444] IOMMU: Setting identity map for device 0000:00:1d.2
>> >>>>>> [0xec000 - 0xeffff]
>> >>>>>>> [    0.550458] IOMMU: Setting identity map for device 0000:00:1d.7
>> >>>>>> [0xec000 - 0xeffff]
>> >>>>>>> [    0.550471] IOMMU: Prepare 0-16MiB unity mapping for LPC
>> >>>>>>> [    0.550483] IOMMU: Setting identity map for device 0000:00:1f.0
>> >>> [0x0
>> >>>>>> - 0xffffff]
>> >>>>>>>
>> >>>>>>
>> >>>>>> Yeah, this looks ok. Actually, taking a second look, I can’t think
>> of
>> >>>>>> anyway how this could be related to file permissions on
>> /dev/vfio/vfio.
>> >>>>>>
>> >>>>>>>> Why does opening /dev/vfio/vfio fail ? Can you please confirm that
>> >>> you
>> >>>>>> have read/write permissions as the user you are trying to run ?
>> >>>>>>> address@hidden ~]# cd /dev/vfio/
>> >>>>>>> address@hidden vfio]# ls -la | grep vfio
>> >>>>>>> crw-rw-rw-  1 root root 10, 196 May 18 11:54 vfio
>> >>>>>>> The right should be okay I guess.
>> >>>>>>>
>> >>>>>> Yes, indeed it is. What distro is this ? Do you have SELinux or any
>> >>> other
>> >>>>>> security feature enabled ? Can you please verify that the file has a
>> >>>>>> appropriate label if SELinux is enabled ? (ls -lZ /dev/vfio/vfio)
>> >>>>>>
>> >>>>>> Bandan
>> >>>>>>>
>> >>>>>>>
>> >>>>>>>
>> >>>>>>> 2015-05-19 18:54 GMT-08:00 Bandan Das <address@hidden>:
>> >>>>>>>
>> >>>>>>> Hello Gabriel,
>> >>>>>>>
>> >>>>>>>> On May 19, 2015, at 8:03 PM, Gabriel Laupre <address@hidden>
>> >>> wrote:
>> >>>>>>>>
>> >>>>>>>> Hello everyone,
>> >>>>>>>>
>> >>>>>>>> I am using a Centos 7.1 machine with the kernel 3.10.229. I want
>> to
>> >>>>>> use my host with SR-IOV to use a virtual function on my NIC as the
>> >>> vNIC in
>> >>>>>> my new VM.
>> >>>>>>>>
>> >>>>>>>> I have an instance started with a old NIC using macvtap that I
>> >>> want to
>> >>>>>> change. I am using the
>> >>>>>>>> virsh edit instance-00000034
>> >>>>>>>> command to edit the XML configuration to add the new device I want
>> >>> to
>> >>>>>> attach.
>> >>>>>>> …
>> >>>>>>> Assuming you are on an intel box, have you booted your kernel with
>> >>>>>> intel_iommu=on ?
>> >>>>>>> Please paste the output of dmesg | grep -e DMAR -e IOMMU ?
>> >>>>>>>
>> >>>>>>>> When I try to reboot the VM I get this error:
>> >>>>>>>> Error starting domain: internal error: process exited while
>> >>> connecting
>> >>>>>> to monitor: 2015-05-19T21:46:21.935043Z qemu-kvm: -device
>> >>>>>> vfio-pci,host=04:10.4,id=hostdev0,bus=pci.0,addr=0x3: vfio: failed
>> to
>> >>> open
>> >>>>>> /dev/vfio/vfio: Operation not permitted
>> >>>>>>> Why does opening /dev/vfio/vfio fail ? Can you please confirm that
>> >>> you
>> >>>>>> have read/write permissions as the user you are trying to run ?
>> >>>>>>>
>> >>>>>>>> 2015-05-19T21:46:21.935091Z qemu-kvm: -device
>> >>>>>> vfio-pci,host=04:10.4,id=hostdev0,bus=pci.0,addr=0x3: vfio: failed
>> to
>> >>> setup
>> >>>>>> container for group 24
>> >>>>>>>> 2015-05-19T21:46:21.935107Z qemu-kvm: -device
>> >>>>>> vfio-pci,host=04:10.4,id=hostdev0,bus=pci.0,addr=0x3: vfio: failed
>> to
>> >>> get
>> >>>>>> group 24
>> >>>>>>>> 2015-05-19T21:46:21.935135Z qemu-kvm: -device
>> >>>>>> vfio-pci,host=04:10.4,id=hostdev0,bus=pci.0,addr=0x3: Device
>> >>> initialization
>> >>>>>> failed.
>> >>>>>>>> 2015-05-19T21:46:21.935157Z qemu-kvm: -device
>> >>>>>> vfio-pci,host=04:10.4,id=hostdev0,bus=pci.0,addr=0x3: Device
>> 'vfio-pci'
>> >>>>>> could not be initialized
>> >>>>>>>>
>> >>>>>>>> total Trace here: http://sprunge.us/XZFB
>> >>>>>>>>
>> >>>>>>>> Any idea how to fix that?
>> >>>>>>>>
>> >>>>>>>> Thank you very much :)
>> >>>>>>>>
>> >>>>>>>> Gabriel
>> >>>>>>>>
>> >>>>>>>
>> >>>>>>>
>> >>>>>>
>> >>>>>>
>> >>>
>> >
>>