Re: [Qemu-discuss] Error starting domain: vfio-pci

2015-05-22 13:14 GMT-07:00 Bandan Das <address@hidden>:

Gabriel Laupre <address@hidden> writes:

> Actually, I get the same error if I enter a PCI address completely random
> (for example 04.01.5, which doesn't exist in my system)

No, that has nothing to do with the pci address of the host device you are
trying to assign. It's telling you that that slot is already used up in
the guest.

Just, try using a different slot number.

> 2015-05-22 12:00 GMT-07:00 Bandan Das <address@hidden>:
>
>> Gabriel Laupre <address@hidden> writes:
>>
>> >>To rule out if it's a problem with your guest xml. Let me know if you are
>> > interested and I will write down the steps.
>> >
>> > I am way more than interested to try that :) Thanks a lot
>>
>> Ok, Based on your command line below, I am assuming that you have a
>> multifunction card. You might need to bind all functions to the
>> stub driver if they are in the same iommu group or else vfio will complain.
>>
>> virsh nodedev-list | grep 04_10
>>
>> The output would be something like
>> pci_0000_04_10_0
>> pci_0000_04_10_1 and so on.
>>
>> For each, assuming they are the same card, you have to bind them to the
>> stub driver:
>> virsh nodedev-detach pci_0000_04_10_0
>> virsh nodedev-detach pci_0000_04_10_1 ...
>>
>> Shut down all guests and make sure no qemu processes are running
>> and then run qemu:
>> /usr/libexec/qemu-kvm -device
>> vfio-pci,host=04:10.4,id=hostdev0,bus=pci.0,addr=0x3 -monitor stdio
>>
>> I think this should be enough to spit out the error, there's no need to
>> actually
>> boot a guest image.
>>
>> > 2015-05-22 10:46 GMT-07:00 Bandan Das <address@hidden>:
>> >
>> >> Gabriel Laupre <address@hidden> writes:
>> >>
>> >> >> Thanks for the update. Did you get a chance to run qemu standalone
>> >> > without libvirt?
>> >> > How can I do that, what are the purpose to do so and why to try that?
>> >> > (thank you ^^)
>> >> To rule out if it's a problem with your guest xml. Let me know if you
>> are
>> >> interested and I will write down the steps.
>> >>
>> >> > 2015-05-21 13:32 GMT-07:00 Bandan Das <address@hidden>:
>> >> >
>> >> >> Gabriel Laupre <address@hidden> writes:
>> >> >>
>> >> >> >> AppArmor is Debian based, but that doesn't stop you from
>> >> >> > running/installing it on other distros
>> >> >> >
>> >> >> >> $ dmesg | grep apparmor
>> >> >> >
>> >> >> > The service isn't started nor present in my machine. Thank you for
>> the
>> >> >> hint
>> >> >> > though.
>> >> >> > I am still stuck with this error and it is not dependent of my
>> Intel
>> >> >> NIC, I
>> >> >> > used another one and I still got the same error message.
>> >> >> > ps: tks to remind me to send to the mailing list:)
>> >> >>
>> >> >> Thanks for the update. Did you get a chance to run qemu standalone
>> >> without
>> >> >> libvirt ?
>> >> >>
>> >> >> > 2015-05-21 12:32 GMT-07:00 Karl Apsite <
>> address@hidden>:
>> >> >> >
>> >> >> >> AppArmor is Debian based, but that doesn't stop you from
>> >> >> >> running/installing it
>> >> >> >> on other distros
>> >> >> >>
>> >> >> >> $ dmesg | grep apparmor
>> >> >> >>
>> >> >> >> On 05/20/2015 02:56 PM, Bandan Das wrote:
>> >> >> >> > Gabriel Laupre <address@hidden> writes:
>> >> >> >> >
>> >> >> >> >> Mmmhh,
>> >> >> >> >> My SELinux is disabled. Is Apparmor not only on debian/ubuntu
>> and
>> >> >> suse,
>> >> >> >> am
>> >> >> >> >> I wrong? I have no idea on that :)
>> >> >> >> >
>> >> >> >> > Yeah, me neither :) I am just trying to rule out all
>> possibilities.
>> >> >> >> >
>> >> >> >> >> 2015-05-20 11:23 GMT-07:00 Bandan Das <address@hidden>:
>> >> >> >> >>
>> >> >> >> >>> Oh and one more thing! You already answered before but just
>> >> wanted
>> >> >> to
>> >> >> >> >>> confirm
>> >> >> >> >>> that you don't have apparmor running, right ?
>> >> >> >> >>>
>> >> >> >> >>> Bandan Das <address@hidden> writes:
>> >> >> >> >>>
>> >> >> >> >>>> Gabriel Laupre <address@hidden> writes:
>> >> >> >> >>>>
>> >> >> >> >>>>>> Yes, indeed it is. What distro is this ? Do you have
>> SELinux
>> >> or
>> >> >> any
>> >> >> >> >>> other
>> >> >> >> >>>>> security feature enabled ? Can you please verify that the
>> file
>> >> >> has a
>> >> >> >> >>>>> appropriate label if SELinux is enabled ? (ls -lZ
>> >> /dev/vfio/vfio)
>> >> >> >> >>>>> My distrib:
>> >> >> >> >>>>> address@hidden ~]# cat /proc/version
>> >> >> >> >>>>> Linux version 3.10.0-229.1.2.el7.x86_64 (
>> >> >> >> >>> address@hidden)
>> >> >> >> >>>>> (gcc version 4.8.2 20140120 (Red Hat 4.8.2-16) (GCC) ) #1
>> SMP
>> >> Fri
>> >> >> >> Mar 27
>> >> >> >> >>>>> 03:04:26 UTC 2015
>> >> >> >> >>>>> address@hidden ~]# cat /etc/centos-release
>> >> >> >> >>>>> CentOS Linux release 7.1.1503 (Core)
>> >> >> >> >>>>>
>> >> >> >> >>>>> address@hidden ~]# ls -lZ /dev/vfio/vfio
>> >> >> >> >>>>> crw-rw-rw- root root ?
>> >> >> /dev/vfio/vfio
>> >> >> >> >>>>>
>> >> >> >> >>>>> SELinux is disabled:
>> >> >> >> >>>>> address@hidden ~]# getenforce
>> >> >> >> >>>>> Disabled
>> >> >> >> >>>>>
>> >> >> >> >>>>> I guess no other security feature is enabled that I am aware
>> >> of. I
>> >> >> >> once
>> >> >> >> >>> had
>> >> >> >> >>>>> a message saying that it can be one of the following issues
>> >> >> (listing
>> >> >> >> the
>> >> >> >> >>>>> 5). So I guess it can be any combination of those issues,
>> even
>> >> >> >> something
>> >> >> >> >>>>> completely different.
>> >> >> >> >>>>
>> >> >> >> >>>> Ugh, I am out of options! Can you please try a few more
>> things:
>> >> Can
>> >> >> >> you
>> >> >> >> >>> try
>> >> >> >> >>>> running qemu directly and see if you see the same behavior ?
>> If
>> >> you
>> >> >> >> still
>> >> >> >> >>>> haven't tried running as root, please try that too. Also,
>> please
>> >> >> check
>> >> >> >> >>> dmesg
>> >> >> >> >>>> for any vfio related errors.
>> >> >> >> >>>>
>> >> >> >> >>>>> libvirtError: internal error: process exited while
>> connecting
>> >> to
>> >> >> >> >>>>> monitor: 2015-05-19T21:46:21.935043Z qemu-kvm: -device
>> >> >> >> >>>>> vfio-pci,host=04:10.4,id=hostdev0,bus=pci.0,addr=0x3: vfio:
>> >> >> failed to
>> >> >> >> >>>>> open /dev/vfio/vfio: Operation not permitted
>> >> >> >> >>>> Well, this is the first error from vfio_connect_container()
>> >> when it
>> >> >> >> does:
>> >> >> >> >>>> fd = qemu_open("/dev/vfio/vfio", O_RDWR);
>> >> >> >> >>>> if (fd < 0) {
>> >> >> >> >>>> error_report("vfio: failed to open /dev/vfio/vfio:
>> %m");
>> >> >> >> >>>> ret = -errno;
>> >> >> >> >>>> ...
>> >> >> >> >>>>
>> >> >> >> >>>> The rest are followup errors printed from the other
>> functions in
>> >> >> the
>> >> >> >> >>>> stack due to this error.
>> >> >> >> >>>>
>> >> >> >> >>>> Bandan
>> >> >> >> >>>>
>> >> >> >> >>>>> 2015-05-19T21:46:21.935091Z qemu-kvm: -device
>> >> >> >> >>>>> vfio-pci,host=04:10.4,id=hostdev0,bus=pci.0,addr=0x3: vfio:
>> >> >> failed to
>> >> >> >> >>>>> setup container for group 24
>> >> >> >> >>>>> 2015-05-19T21:46:21.935107Z qemu-kvm: -device
>> >> >> >> >>>>> vfio-pci,host=04:10.4,id=hostdev0,bus=pci.0,addr=0x3: vfio:
>> >> >> failed to
>> >> >> >> >>>>> get group 24
>> >> >> >> >>>>> 2015-05-19T21:46:21.935135Z qemu-kvm: -device
>> >> >> >> >>>>> vfio-pci,host=04:10.4,id=hostdev0,bus=pci.0,addr=0x3: Device
>> >> >> >> >>>>> initialization failed.
>> >> >> >> >>>>> 2015-05-19T21:46:21.935157Z qemu-kvm: -device
>> >> >> >> >>>>> vfio-pci,host=04:10.4,id=hostdev0,bus=pci.0,addr=0x3: Device
>> >> >> >> >>>>> 'vfio-pci' could not be initialized
>> >> >> >> >>>>>
>> >> >> >> >>>>>
>> >> >> >> >>>>>
>> >> >> >> >>>>>
>> >> >> >> >>>>> 2015-05-19 21:17 GMT-08:00 Bandan Das <address@hidden>:
>> >> >> >> >>>>>
>> >> >> >> >>>>>>
>> >> >> >> >>>>>>> On May 20, 2015, at 12:29 AM, Gabriel Laupre <
>> >> address@hidden
>> >> >> >
>> >> >> >> >>> wrote:
>> >> >> >> >>>>>>>
>> >> >> >> >>>>>>> Thank Bandan,
>> >> >> >> >>>>>>>
>> >> >> >> >>>>>>>
>> >> >> >> >>>>>>>> Assuming you are on an intel box, have you booted your
>> >> kernel
>> >> >> with
>> >> >> >> >>>>>> intel_iommu=on ?
>> >> >> >> >>>>>>> Yes, I have booted my kernel with the intel_iommu=on. (I
>> >> don't
>> >> >> >> >>> remember
>> >> >> >> >>>>>> how to check that now though ^^)
>> >> >> >> >>>>>>>
>> >> >> >> >>>>>>>> Please paste the output of dmesg | grep -e DMAR -e IOMMU
>> ?
>> >> >> >> >>>>>>> address@hidden ~]# dmesg | grep -e DMAR -e IOMMU
>> >> >> >> >>>>>>> [ 0.000000] ACPI: DMAR 00000000bf79e0c0 00118 (v01
>> AMI
>> >> >> >> OEMDMAR
>> >> >> >> >>>>>> 00000001 MSFT 00000097)
>> >> >> >> >>>>>>> [ 0.000000] Intel-IOMMU: enabled
>> >> >> >> >>>>>>> [ 0.039149] dmar: IOMMU 0: reg_base_addr fbffe000 ver
>> 1:0
>> >> cap
>> >> >> >> >>>>>> c90780106f0462 ecap f020f6
>> >> >> >> >>>>>>> [ 0.550126] IOMMU 0 0xfbffe000: using Queued
>> invalidation
>> >> >> >> >>>>>>> [ 0.550131] IOMMU: Setting RMRR:
>> >> >> >> >>>>>>> [ 0.550149] IOMMU: Setting identity map for device
>> >> >> 0000:00:1a.0
>> >> >> >> >>>>>> [0xbf7ec000 - 0xbf7fffff]
>> >> >> >> >>>>>>> [ 0.550184] IOMMU: Setting identity map for device
>> >> >> 0000:00:1a.1
>> >> >> >> >>>>>> [0xbf7ec000 - 0xbf7fffff]
>> >> >> >> >>>>>>> [ 0.550211] IOMMU: Setting identity map for device
>> >> >> 0000:00:1a.2
>> >> >> >> >>>>>> [0xbf7ec000 - 0xbf7fffff]
>> >> >> >> >>>>>>> [ 0.550241] IOMMU: Setting identity map for device
>> >> >> 0000:00:1a.7
>> >> >> >> >>>>>> [0xbf7ec000 - 0xbf7fffff]
>> >> >> >> >>>>>>> [ 0.550272] IOMMU: Setting identity map for device
>> >> >> 0000:00:1d.0
>> >> >> >> >>>>>> [0xbf7ec000 - 0xbf7fffff]
>> >> >> >> >>>>>>> [ 0.550302] IOMMU: Setting identity map for device
>> >> >> 0000:00:1d.1
>> >> >> >> >>>>>> [0xbf7ec000 - 0xbf7fffff]
>> >> >> >> >>>>>>> [ 0.550329] IOMMU: Setting identity map for device
>> >> >> 0000:00:1d.2
>> >> >> >> >>>>>> [0xbf7ec000 - 0xbf7fffff]
>> >> >> >> >>>>>>> [ 0.550358] IOMMU: Setting identity map for device
>> >> >> 0000:00:1d.7
>> >> >> >> >>>>>> [0xbf7ec000 - 0xbf7fffff]
>> >> >> >> >>>>>>> [ 0.550375] IOMMU: Setting identity map for device
>> >> >> 0000:00:1a.0
>> >> >> >> >>>>>> [0xec000 - 0xeffff]
>> >> >> >> >>>>>>> [ 0.550387] IOMMU: Setting identity map for device
>> >> >> 0000:00:1a.1
>> >> >> >> >>>>>> [0xec000 - 0xeffff]
>> >> >> >> >>>>>>> [ 0.550399] IOMMU: Setting identity map for device
>> >> >> 0000:00:1a.2
>> >> >> >> >>>>>> [0xec000 - 0xeffff]
>> >> >> >> >>>>>>> [ 0.550410] IOMMU: Setting identity map for device
>> >> >> 0000:00:1a.7
>> >> >> >> >>>>>> [0xec000 - 0xeffff]
>> >> >> >> >>>>>>> [ 0.550421] IOMMU: Setting identity map for device
>> >> >> 0000:00:1d.0
>> >> >> >> >>>>>> [0xec000 - 0xeffff]
>> >> >> >> >>>>>>> [ 0.550433] IOMMU: Setting identity map for device
>> >> >> 0000:00:1d.1
>> >> >> >> >>>>>> [0xec000 - 0xeffff]
>> >> >> >> >>>>>>> [ 0.550444] IOMMU: Setting identity map for device
>> >> >> 0000:00:1d.2
>> >> >> >> >>>>>> [0xec000 - 0xeffff]
>> >> >> >> >>>>>>> [ 0.550458] IOMMU: Setting identity map for device
>> >> >> 0000:00:1d.7
>> >> >> >> >>>>>> [0xec000 - 0xeffff]
>> >> >> >> >>>>>>> [ 0.550471] IOMMU: Prepare 0-16MiB unity mapping for
>> LPC
>> >> >> >> >>>>>>> [ 0.550483] IOMMU: Setting identity map for device
>> >> >> 0000:00:1f.0
>> >> >> >> >>> [0x0
>> >> >> >> >>>>>> - 0xffffff]
>> >> >> >> >>>>>>>
>> >> >> >> >>>>>>
>> >> >> >> >>>>>> Yeah, this looks ok. Actually, taking a second look, I
>> can’t
>> >> >> think
>> >> >> >> of
>> >> >> >> >>>>>> anyway how this could be related to file permissions on
>> >> >> >> /dev/vfio/vfio.
>> >> >> >> >>>>>>
>> >> >> >> >>>>>>>> Why does opening /dev/vfio/vfio fail ? Can you please
>> >> confirm
>> >> >> that
>> >> >> >> >>> you
>> >> >> >> >>>>>> have read/write permissions as the user you are trying to
>> run
>> >> ?
>> >> >> >> >>>>>>> address@hidden ~]# cd /dev/vfio/
>> >> >> >> >>>>>>> address@hidden vfio]# ls -la | grep vfio
>> >> >> >> >>>>>>> crw-rw-rw- 1 root root 10, 196 May 18 11:54 vfio
>> >> >> >> >>>>>>> The right should be okay I guess.
>> >> >> >> >>>>>>>
>> >> >> >> >>>>>> Yes, indeed it is. What distro is this ? Do you have
>> SELinux
>> >> or
>> >> >> any
>> >> >> >> >>> other
>> >> >> >> >>>>>> security feature enabled ? Can you please verify that the
>> file
>> >> >> has a
>> >> >> >> >>>>>> appropriate label if SELinux is enabled ? (ls -lZ
>> >> /dev/vfio/vfio)
>> >> >> >> >>>>>>
>> >> >> >> >>>>>> Bandan
>> >> >> >> >>>>>>>
>> >> >> >> >>>>>>>
>> >> >> >> >>>>>>>
>> >> >> >> >>>>>>> 2015-05-19 18:54 GMT-08:00 Bandan Das <address@hidden>:
>> >> >> >> >>>>>>>
>> >> >> >> >>>>>>> Hello Gabriel,
>> >> >> >> >>>>>>>
>> >> >> >> >>>>>>>> On May 19, 2015, at 8:03 PM, Gabriel Laupre <
>> >> address@hidden
>> >> >> >
>> >> >> >> >>> wrote:
>> >> >> >> >>>>>>>>
>> >> >> >> >>>>>>>> Hello everyone,
>> >> >> >> >>>>>>>>
>> >> >> >> >>>>>>>> I am using a Centos 7.1 machine with the kernel
>> 3.10.229. I
>> >> >> want
>> >> >> >> to
>> >> >> >> >>>>>> use my host with SR-IOV to use a virtual function on my
>> NIC as
>> >> >> the
>> >> >> >> >>> vNIC in
>> >> >> >> >>>>>> my new VM.
>> >> >> >> >>>>>>>>
>> >> >> >> >>>>>>>> I have an instance started with a old NIC using macvtap
>> >> that I
>> >> >> >> >>> want to
>> >> >> >> >>>>>> change. I am using the
>> >> >> >> >>>>>>>> virsh edit instance-00000034
>> >> >> >> >>>>>>>> command to edit the XML configuration to add the new
>> device
>> >> I
>> >> >> want
>> >> >> >> >>> to
>> >> >> >> >>>>>> attach.
>> >> >> >> >>>>>>> …
>> >> >> >> >>>>>>> Assuming you are on an intel box, have you booted your
>> kernel
>> >> >> with
>> >> >> >> >>>>>> intel_iommu=on ?
>> >> >> >> >>>>>>> Please paste the output of dmesg | grep -e DMAR -e IOMMU ?
>> >> >> >> >>>>>>>
>> >> >> >> >>>>>>>> When I try to reboot the VM I get this error:
>> >> >> >> >>>>>>>> Error starting domain: internal error: process exited
>> while
>> >> >> >> >>> connecting
>> >> >> >> >>>>>> to monitor: 2015-05-19T21:46:21.935043Z qemu-kvm: -device
>> >> >> >> >>>>>> vfio-pci,host=04:10.4,id=hostdev0,bus=pci.0,addr=0x3: vfio:
>> >> >> failed
>> >> >> >> to
>> >> >> >> >>> open
>> >> >> >> >>>>>> /dev/vfio/vfio: Operation not permitted
>> >> >> >> >>>>>>> Why does opening /dev/vfio/vfio fail ? Can you please
>> confirm
>> >> >> that
>> >> >> >> >>> you
>> >> >> >> >>>>>> have read/write permissions as the user you are trying to
>> run
>> >> ?
>> >> >> >> >>>>>>>
>> >> >> >> >>>>>>>> 2015-05-19T21:46:21.935091Z qemu-kvm: -device
>> >> >> >> >>>>>> vfio-pci,host=04:10.4,id=hostdev0,bus=pci.0,addr=0x3: vfio:
>> >> >> failed
>> >> >> >> to
>> >> >> >> >>> setup
>> >> >> >> >>>>>> container for group 24
>> >> >> >> >>>>>>>> 2015-05-19T21:46:21.935107Z qemu-kvm: -device
>> >> >> >> >>>>>> vfio-pci,host=04:10.4,id=hostdev0,bus=pci.0,addr=0x3: vfio:
>> >> >> failed
>> >> >> >> to
>> >> >> >> >>> get
>> >> >> >> >>>>>> group 24
>> >> >> >> >>>>>>>> 2015-05-19T21:46:21.935135Z qemu-kvm: -device
>> >> >> >> >>>>>> vfio-pci,host=04:10.4,id=hostdev0,bus=pci.0,addr=0x3:
>> Device
>> >> >> >> >>> initialization
>> >> >> >> >>>>>> failed.
>> >> >> >> >>>>>>>> 2015-05-19T21:46:21.935157Z qemu-kvm: -device
>> >> >> >> >>>>>> vfio-pci,host=04:10.4,id=hostdev0,bus=pci.0,addr=0x3:
>> Device
>> >> >> >> 'vfio-pci'
>> >> >> >> >>>>>> could not be initialized
>> >> >> >> >>>>>>>>
>> >> >> >> >>>>>>>> total Trace here: http://sprunge.us/XZFB
>> >> >> >> >>>>>>>>
>> >> >> >> >>>>>>>> Any idea how to fix that?
>> >> >> >> >>>>>>>>
>> >> >> >> >>>>>>>> Thank you very much :)
>> >> >> >> >>>>>>>>
>> >> >> >> >>>>>>>> Gabriel
>> >> >> >> >>>>>>>>
>> >> >> >> >>>>>>>
>> >> >> >> >>>>>>>
>> >> >> >> >>>>>>
>> >> >> >> >>>>>>
>> >> >> >> >>>
>> >> >> >> >
>> >> >> >>
>> >> >>
>> >>
>>

From:	Gabriel Laupre
Subject:	Re: [Qemu-discuss] Error starting domain: vfio-pci
Date:	Fri, 22 May 2015 13:24:55 -0700