[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
[Qemu-devel] [Bug 1719196] Re: [arm64 ocata] newly created instances are
From: |
Sean Feole |
Subject: |
[Qemu-devel] [Bug 1719196] Re: [arm64 ocata] newly created instances are unable to raise network interfaces |
Date: |
Thu, 05 Oct 2017 01:46:24 -0000 |
I was able to gather libvirt XMLs from both Newton and Ocata Instances,
see attached
address@hidden:~$ diff xmlocata xmlnewton
1,2c1,2
<
< main type='kvm' id='1'>
---
> address@hidden:/var/lib/nova/instances/358596e4-135d-461d-a514-84116440014c$
> sudo virsh dumpxml instance-00000001
> <domain type='kvm' id='1'>
4c4
< <uuid>7c0dcd78-d6b4-4575-a882-ee5d29c64fe0</uuid>
---
> <uuid>358596e4-135d-461d-a514-84116440014c</uuid>
7,9c7,9
< <nova:package version="15.0.6"/>
< <nova:name>sfeole14</nova:name>
< <nova:creationTime>2017-10-05 00:58:15</nova:creationTime>
---
> <nova:package version="14.0.8"/>
> <nova:name>sfeole-newton</nova:name>
> <nova:creationTime>2017-10-05 01:40:39</nova:creationTime>
18,19c18,19
< <nova:user uuid="d9f92a61c37948d9a29b8cc37e1bca05">admin</nova:user>
< <nova:project
uuid="701441267bd148d3842f1696530b1c92">admin</nova:project>
---
> <nova:user uuid="12d13712253141ab845b89406507cd6c">admin</nova:user>
> <nova:project
> uuid="8b8fcaf183954f45b3eb15b27c52ec94">admin</nova:project>
21c21
< <nova:root type="image" uuid="f329117f-5da2-4d61-8341-89a969bf00e7"/>
---
> <nova:root type="image" uuid="fcdf7c26-4238-4594-b8ee-fd59f601fdcb"/>
34c34
< <type arch='aarch64' machine='virt-2.8'>hvm</type>
---
> <type arch='aarch64' machine='virt'>hvm</type>
58c58
< <source
file='/var/lib/nova/instances/7c0dcd78-d6b4-4575-a882-ee5d29c64fe0/disk'/>
---
> <source
> file='/var/lib/nova/instances/358596e4-135d-461d-a514-84116440014c/disk'/>
61c61
< <source
file='/var/lib/nova/instances/_base/035458feead7f83be80ed020442ebf815a4067ea'/>
---
> <source
> file='/var/lib/nova/instances/_base/ab7429e8558ea27fb54f70eb92fbd0c1c07dd595'/>
70c70
< <source
file='/var/lib/nova/instances/7c0dcd78-d6b4-4575-a882-ee5d29c64fe0/disk.eph0'/>
---
> <source
> file='/var/lib/nova/instances/358596e4-135d-461d-a514-84116440014c/disk.eph0'/>
82a83,93
> <controller type='pci' index='1' model='dmi-to-pci-bridge'>
> <model name='i82801b11-bridge'/>
> <alias name='pci.1'/>
> <address type='pci' domain='0x0000' bus='0x00' slot='0x01'
> function='0x0'/>
> </controller>
> <controller type='pci' index='2' model='pci-bridge'>
> <model name='pci-bridge'/>
> <target chassisNr='2'/>
> <alias name='pci.2'/>
> <address type='pci' domain='0x0000' bus='0x01' slot='0x01'
> function='0x0'/>
> </controller>
84,86c95,97
< <mac address='fa:16:3e:a6:4b:d4'/>
< <source bridge='qbra7012530-32'/>
< <target dev='tapa7012530-32'/>
---
> <mac address='fa:16:3e:8e:fc:48'/>
> <source bridge='qbrb5d335fc-46'/>
> <target dev='tapb5d335fc-46'/>
92c103
< <source
path='/var/lib/nova/instances/7c0dcd78-d6b4-4575-a882-ee5d29c64fe0/console.log'/>
---
> <source
> path='/var/lib/nova/instances/358596e4-135d-461d-a514-84116440014c/console.log'/>
95a107,111
> <serial type='pty'>
> <source path='/dev/pts/3'/>
> <target port='1'/>
> <alias name='serial1'/>
> </serial>
97c113
< <source
path='/var/lib/nova/instances/7c0dcd78-d6b4-4575-a882-ee5d29c64fe0/console.log'/>
---
> <source
> path='/var/lib/nova/instances/358596e4-135d-461d-a514-84116440014c/console.log'/>
108,113c124,125
< <label>libvirt-7c0dcd78-d6b4-4575-a882-ee5d29c64fe0</label>
< <imagelabel>libvirt-7c0dcd78-d6b4-4575-a882-ee5d29c64fe0</imagelabel>
< </seclabel>
< <seclabel type='dynamic' model='dac' relabel='yes'>
< <label>+64055:+117</label>
< <imagelabel>+64055:+117</imagelabel>
---
> <label>libvirt-358596e4-135d-461d-a514-84116440014c</label>
> <imagelabel>libvirt-358596e4-135d-461d-a514-84116440014c</imagelabel>
address@hidden:~$
** Attachment added: "xmlnotes.txt"
https://bugs.launchpad.net/libvirt/+bug/1719196/+attachment/4962430/+files/xmlnotes.txt
--
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.
https://bugs.launchpad.net/bugs/1719196
Title:
[arm64 ocata] newly created instances are unable to raise network
interfaces
Status in libvirt:
New
Status in QEMU:
New
Bug description:
arm64 Ocata ,
I'm testing to see I can get Ocata running on arm64 and using the
openstack-base bundle to deploy it. I have added the bundle to the
log file attached to this bug.
When I create a new instance via nova, the VM comes up and runs,
however fails to raise its eth0 interface. This occurs on both
internal and external networks.
address@hidden:~$ nova list
+--------------------------------------+---------+--------+------------+-------------+--------------------+
| ID | Name | Status | Task State |
Power State | Networks |
+--------------------------------------+---------+--------+------------+-------------+--------------------+
| dcaf6d51-f81e-4cbd-ac77-0c5d21bde57c | sfeole1 | ACTIVE | - |
Running | internal=10.5.5.3 |
| aa0b8aee-5650-41f4-8fa0-aeccdc763425 | sfeole2 | ACTIVE | - |
Running | internal=10.5.5.13 |
+--------------------------------------+---------+--------+------------+-------------+--------------------+
address@hidden:~$ nova show aa0b8aee-5650-41f4-8fa0-aeccdc763425
+--------------------------------------+----------------------------------------------------------+
| Property | Value
|
+--------------------------------------+----------------------------------------------------------+
| OS-DCF:diskConfig | MANUAL
|
| OS-EXT-AZ:availability_zone | nova
|
| OS-EXT-SRV-ATTR:host | awrep3
|
| OS-EXT-SRV-ATTR:hypervisor_hostname | awrep3.maas
|
| OS-EXT-SRV-ATTR:instance_name | instance-00000003
|
| OS-EXT-STS:power_state | 1
|
| OS-EXT-STS:task_state | -
|
| OS-EXT-STS:vm_state | active
|
| OS-SRV-USG:launched_at | 2017-09-24T14:23:08.000000
|
| OS-SRV-USG:terminated_at | -
|
| accessIPv4 |
|
| accessIPv6 |
|
| config_drive |
|
| created | 2017-09-24T14:22:41Z
|
| flavor | m1.small
(717660ae-0440-4b19-a762-ffeb32a0575c) |
| hostId |
5612a00671c47255d2ebd6737a64ec9bd3a5866d1233ecf3e988b025 |
| id | aa0b8aee-5650-41f4-8fa0-aeccdc763425
|
| image | zestynosplash
(e88fd1bd-f040-44d8-9e7c-c462ccf4b945) |
| internal network | 10.5.5.13
|
| key_name | mykey
|
| metadata | {}
|
| name | sfeole2
|
| os-extended-volumes:volumes_attached | []
|
| progress | 0
|
| security_groups | default
|
| status | ACTIVE
|
| tenant_id | 9f7a21c1ad264fec81abc09f3960ad1d
|
| updated | 2017-09-24T14:23:09Z
|
| user_id | e6bb6f5178a248c1b5ae66ed388f9040
|
+--------------------------------------+----------------------------------------------------------+
As seen above the instances boot an run. Full Console output is
attached to this bug.
[ OK ] Started Initial cloud-init job (pre-networking).
[ OK ] Reached target Network (Pre).
[ OK ] Started AppArmor initialization.
Starting Raise network interfaces...
[FAILED] Failed to start Raise network interfaces.
See 'systemctl status networking.service' for details.
Starting Initial cloud-init job (metadata service crawler)...
[ OK ] Reached target Network.
[ 315.051902] cloud-init[881]: Cloud-init v. 0.7.9 running 'init' at Fri, 22
Sep 2017 18:29:15 +0000. Up 314.70 seconds.
[ 315.057291] cloud-init[881]: ci-info: +++++++++++++++++++++++++++Net
device info+++++++++++++++++++++++++++
[ 315.060338] cloud-init[881]: ci-info:
+--------+------+-----------+-----------+-------+-------------------+
[ 315.063308] cloud-init[881]: ci-info: | Device | Up | Address |
Mask | Scope | Hw-Address |
[ 315.066304] cloud-init[881]: ci-info:
+--------+------+-----------+-----------+-------+-------------------+
[ 315.069303] cloud-init[881]: ci-info: | eth0: | True | . | .
| . | fa:16:3e:39:4c:48 |
[ 315.072308] cloud-init[881]: ci-info: | eth0: | True | . | .
| d | fa:16:3e:39:4c:48 |
[ 315.075260] cloud-init[881]: ci-info: | lo: | True | 127.0.0.1 |
255.0.0.0 | . | . |
[ 315.078258] cloud-init[881]: ci-info: | lo: | True | . | .
| d | . |
[ 315.081249] cloud-init[881]: ci-info:
+--------+------+-----------+-----------+-------+-------------------+
[ 315.084240] cloud-init[881]: 2017-09-22 18:29:15,393 -
url_helper.py[WARNING]: Calling
'http://169.254.169.254/2009-04-04/meta-data/instance-id' failed [0/120s]:
request error [HTTPConnectionPool(host='169.254.169.254', port=80): Max retries
exceeded with url: /2009-04-04/meta-data/instance-id (Caused by
NewConnectionError('<requests.packages.urllib3.connection.HTTPConnection object
at 0xffffb10794e0>: Failed to establish a new connection: [Errno 101] Network
is unreachable',))]
----------------
I have checked all services in neutron and made sure that they are
running and restarted in neutron-gateway
[ + ] neutron-dhcp-agent
[ + ] neutron-l3-agent
[ + ] neutron-lbaasv2-agent
[ + ] neutron-metadata-agent
[ + ] neutron-metering-agent
[ + ] neutron-openvswitch-agent
[ + ] neutron-ovs-cleanup
and have also restarted and checked the nova-compute logs for their
neutron counterparts.
[ + ] neutron-openvswitch-agent
[ + ] neutron-ovs-cleanup
There are some warnings/errors in the neutron-gateway logs which I
have attached the full tarball in a separate attachment to this bug:
2017-09-24 14:39:33.152 10322 INFO ryu.base.app_manager [-] instantiating app
ryu.controller.ofp_handler of OFPHandler
2017-09-24 14:39:33.153 10322 INFO ryu.base.app_manager [-] instantiating app
ryu.app.ofctl.service of OfctlService
2017-09-24 14:39:39.577 10322 ERROR neutron.agent.ovsdb.impl_vsctl
[req-2f084ae8-13dc-47dc-b351-24c8f3c57067 - - - - -] Unable to execute
['ovs-vsctl', '--timeout=10', '--oneline', '--format=json', '--',
'address@hidden', 'create', 'Manager', 'target="ptcp:6640:127.0.0.1"', '--',
'add', 'Open_vSwitch', '.', 'manager_options', '@manager']. Exception: Exit
code: 1; Stdin: ; Stdout: ; Stderr: ovs-vsctl: transaction error:
{"details":"Transaction causes multiple rows in \"Manager\" table to have
identical values (\"ptcp:6640:127.0.0.1\") for index on column \"target\".
First row, with UUID e02a5f7f-bfd2-4a1d-ae3c-0321db4bd3fb, existed in the
database before this transaction and was not modified by the transaction.
Second row, with UUID 6e9aba3a-471a-4976-bffd-b7131bbe5377, was inserted by
this transaction.","error":"constraint violation"}
These warnings/errors also occur on the nova-compute hosts in
/var/log/neutron/
2017-09-22 18:54:52.130 387556 INFO ryu.base.app_manager [-] instantiating
app ryu.app.ofctl.service of OfctlService
2017-09-22 18:54:56.124 387556 ERROR neutron.agent.ovsdb.impl_vsctl
[req-e291c2f9-a123-422c-be7c-fadaeb5decfa - - - - -] Unable to execute
['ovs-vsctl', '--timeout=10', '--oneline', '--format=json', '--',
'address@hidden', 'create', 'Manager', 'target="ptcp:6640:127.0.0.1"', '--',
'add', 'Open_vSwitch', '.', 'manager_options', '@manager']. Exception: Exit
code: 1; Stdin: ; Stdout: ; Stderr: ovs-vsctl: transaction error:
{"details":"Transaction causes multiple rows in \"Manager\" table to have
identical values (\"ptcp:6640:127.0.0.1\") for index on column \"target\".
First row, with UUID 9f27ddee-9881-4cbc-9777-2f42fee735e9, was inserted by this
transaction. Second row, with UUID ccf0e097-09d5-449c-b353-6b69781dc3f7,
existed in the database before this transaction and was not modified by the
transaction.","error":"constraint violation"}
I'm not sure if the above error could be pertaining to the failure, I
have also attached those logs to this bug as well...
.
(neutron) agent-list
+--------------------------------------+----------------------+--------+-------------------+-------+----------------+---------------------------+
| id | agent_type | host |
availability_zone | alive | admin_state_up | binary |
+--------------------------------------+----------------------+--------+-------------------+-------+----------------+---------------------------+
| 0cca03fb-abb2-4704-8b0b-e7d3e117d882 | DHCP agent | awrep1 | nova
| :-) | True | neutron-dhcp-agent |
| 14a5fd52-fbc3-450c-96d5-4e9a65776dad | L3 agent | awrep1 | nova
| :-) | True | neutron-l3-agent |
| 2ebc7238-5e61-41f8-bc60-df14ec6b226b | Loadbalancerv2 agent | awrep1 |
| :-) | True | neutron-lbaasv2-agent |
| 4f6275be-fc8b-4994-bdac-13a4b76f6a83 | Metering agent | awrep1 |
| :-) | True | neutron-metering-agent |
| 86ecc6b0-c100-4298-b861-40c17516cc08 | Open vSwitch agent | awrep1 |
| :-) | True | neutron-openvswitch-agent |
| 947ad3ab-650b-4b96-a520-00441ecb33e7 | Open vSwitch agent | awrep4 |
| :-) | True | neutron-openvswitch-agent |
| 996b0692-7d19-4641-bec3-e057f4a856f6 | Open vSwitch agent | awrep3 |
| :-) | True | neutron-openvswitch-agent |
| ab6b1065-0b98-4cf3-9f46-6bddba0c5e75 | Metadata agent | awrep1 |
| :-) | True | neutron-metadata-agent |
| fe24f622-b77c-4eed-ae22-18e4195cf763 | Open vSwitch agent | awrep2 |
| :-) | True | neutron-openvswitch-agent |
+--------------------------------------+----------------------+--------+-------------------+-------+----------------+---------------------------+
Should neutron-openvswitch be assigned to the 'nova' availability zone?
I have also attached the ovs-vsctl show from a nova-compute host and
the neutron-gateway host to ensure that the open v switch routes are
correct.
I can supply more logs if required.
To manage notifications about this bug go to:
https://bugs.launchpad.net/libvirt/+bug/1719196/+subscriptions
- [Qemu-devel] [Bug 1719196] Re: [arm64 ocata] newly created instances are unable to raise network interfaces, Sean Feole, 2017/10/02
- [Qemu-devel] [Bug 1719196] Re: [arm64 ocata] newly created instances are unable to raise network interfaces, Sean Feole, 2017/10/02
- [Qemu-devel] [Bug 1719196] Re: [arm64 ocata] newly created instances are unable to raise network interfaces, Sean Feole, 2017/10/03
- [Qemu-devel] [Bug 1719196] Re: [arm64 ocata] newly created instances are unable to raise network interfaces, Sean Feole, 2017/10/03
- [Qemu-devel] [Bug 1719196] Re: [arm64 ocata] newly created instances are unable to raise network interfaces, Sean Feole, 2017/10/03
- [Qemu-devel] [Bug 1719196] Re: [arm64 ocata] newly created instances are unable to raise network interfaces,
Sean Feole <=
- [Qemu-devel] [Bug 1719196] Re: [arm64 ocata] newly created instances are unable to raise network interfaces, dann frazier, 2017/10/05
- [Qemu-devel] [Bug 1719196] Re: [arm64 ocata] newly created instances are unable to raise network interfaces, ChristianEhrhardt, 2017/10/11
- [Qemu-devel] [Bug 1719196] Re: [arm64 ocata] newly created instances are unable to raise network interfaces, ChristianEhrhardt, 2017/10/11
- [Qemu-devel] [Bug 1719196] Re: [arm64 ocata] newly created instances are unable to raise network interfaces, Sean Feole, 2017/10/11
- [Qemu-devel] [Bug 1719196] Re: [arm64 ocata] newly created instances are unable to raise network interfaces, Andrew McLeod, 2017/10/17
- [Qemu-devel] [Bug 1719196] Re: [arm64 ocata] newly created instances are unable to raise network interfaces, ChristianEhrhardt, 2017/10/18
- [Qemu-devel] [Bug 1719196] Re: [arm64 ocata] newly created instances are unable to raise network interfaces, ChristianEhrhardt, 2017/10/18
- [Qemu-devel] [Bug 1719196] Re: [arm64 ocata] newly created instances are unable to raise network interfaces, Sean Feole, 2017/10/18
- [Qemu-devel] [Bug 1719196] Re: [arm64 ocata] newly created instances are unable to raise network interfaces, Ryan Beisner, 2017/10/18
- [Qemu-devel] [Bug 1719196] Re: [arm64 ocata] newly created instances are unable to raise network interfaces, Corey Bryant, 2017/10/18