Solved: VIO Installation issue - failure to start neutron

ganapathibhat · ‎07-09-2015

I am trying to install VMware Integrated Openstack (VIO) and I am getting following installation issues. I appreciate any insight into this. Thanks in advance

- With NSX Networking: Installation fails at 95% due to failure to start neutron..I tried couple of times it fails at same place…

Here is the log...

2015-06-29 21:11:23,601 p=504 u=jarvis | TASK: [config-controller | grant service role to glance user on service tenant] ***

2015-06-29 21:11:24,786 p=504 u=jarvis | ok: [10.2.18.207]

2015-06-29 21:11:24,787 p=504 u=jarvis | TASK: [config-controller | disable glance services start on boot] *************

2015-06-29 21:11:24,938 p=504 u=jarvis | ok: [10.2.18.207] => (item=glance-api)

2015-06-29 21:11:24,947 p=504 u=jarvis | ok: [10.2.18.208] => (item=glance-api)

2015-06-29 21:11:25,060 p=504 u=jarvis | ok: [10.2.18.207] => (item=glance-registry)

2015-06-29 21:11:25,067 p=504 u=jarvis | ok: [10.2.18.208] => (item=glance-registry)

2015-06-29 21:11:25,077 p=504 u=jarvis | TASK: [config-controller | write the fernet key file] *************************

2015-06-29 21:11:26,027 p=504 u=jarvis | ok: [10.2.18.207]

2015-06-29 21:11:26,028 p=504 u=jarvis | ok: [10.2.18.208]

2015-06-29 21:11:26,041 p=504 u=jarvis | TASK: [config-controller | update neutron server] *****************************

2015-06-29 21:11:31,096 p=504 u=jarvis | changed: [10.2.18.207]

2015-06-29 21:11:31,123 p=504 u=jarvis | changed: [10.2.18.208]

2015-06-29 21:11:31,136 p=504 u=jarvis | TASK: [config-controller | update neutron configuration] **********************

2015-06-29 21:11:34,334 p=504 u=jarvis | changed: [10.2.18.207]

2015-06-29 21:11:34,338 p=504 u=jarvis | changed: [10.2.18.208]

2015-06-29 21:11:34,350 p=504 u=jarvis | TASK: [config-controller | initialize neutron database] ***********************

2015-06-29 21:11:35,950 p=504 u=jarvis | changed: [10.2.18.207]

2015-06-29 21:11:35,951 p=504 u=jarvis | TASK: [config-controller | stop neutron on all controllers] *******************

2015-06-29 21:11:36,118 p=504 u=jarvis | ok: [10.2.18.208]

2015-06-29 21:11:36,128 p=504 u=jarvis | changed: [10.2.18.207]

2015-06-29 21:11:36,138 p=504 u=jarvis | TASK: [config-controller | start neutron on first controller] *****************

2015-06-29 21:11:36,386 p=504 u=jarvis | changed: [10.2.18.207]

2015-06-29 21:11:36,387 p=504 u=jarvis | TASK: [config-controller | wait for neutron to start on first controller for NSX] ***

2015-06-29 21:26:36,877 p=504 u=jarvis | failed: [10.2.18.207] => {"elapsed": 900, "failed": true}

2015-06-29 21:26:36,877 p=504 u=jarvis | msg: Timeout when waiting for 127.0.0.1:9696

2015-06-29 21:26:36,878 p=504 u=jarvis | FATAL: all hosts have already failed -- aborting

viouser@localhost:/var/log$ sudo cat /var/log/jarvis/ansible.log

ganapathibhat · ‎07-31-2015

This got resolved, after adding additional compute resource to the Management cluster. I already 4/6 Core 4 servers, added one more and installation went fine.

Thanks all for taking time and helping on this...Much appreciated..

View solution in original post

ricardokatz · ‎07-09-2015

Probably you forgot to provision at least one controller on NSX installation (or it's not acessible from your Neutron/Controller server):

KarolSte · ‎07-09-2015

Can you run viogetlogs and provide the output package here? or just the neutron log from VIO-controller-0 (/var/log/neutron/neutron-server.log)

Thanks,

Karol

ganapathibhat · ‎07-09-2015

Thanks ricardokatzricardokatz.... I have only 1 NSX controller which is up and running.... and it is accessible from the VIO management server (could ping)...VIO management server can access NSX manager node as well...

Hope it does't look for 3 NSX controller as recommended in the best practice doc...

ricardokatz · ‎07-09-2015

No, it doesn't (I'm running with this setup).

So get the logs from neutron-server.log (as KarolSte‌ suggested) so we can see why Neutron is not starting

admin · ‎07-09-2015

Hey Guys,

If you are using VIO with NSX just ping your VMware account team (or NSX account team). They can get you help from NSX experts on configuring them with VIO. Or just ping me with a private message and I will be happy to connect you with right NSX resources.

This will greatly expedite your setup process.

arvind

vChrisSt · ‎07-09-2015

Hi,

I had the same issue for a while and spend a lot of time to fix that issue.

What is in the Neutron Logfile?

What User do you use? Do you use an LDAP User or a local Account?

Regards

Chris

ganapathibhat · ‎07-15-2015

@ KarolSte, viogetlog.tar file attached. However I did't find /var/log/neutron/neutron-server.log file in either of the controller.

2015-07-15 00:09:37,259 p=504 u=jarvis | TASK: [config-controller | initialize neutron database] ***********************

2015-07-15 00:09:40,269 p=504 u=jarvis | changed: [10.2.18.207]

2015-07-15 00:09:40,270 p=504 u=jarvis | TASK: [config-controller | stop neutron on all controllers] *******************

2015-07-15 00:09:40,566 p=504 u=jarvis | changed: [10.2.18.207]

2015-07-15 00:09:40,574 p=504 u=jarvis | changed: [10.2.18.208]

2015-07-15 00:09:40,584 p=504 u=jarvis | TASK: [config-controller | start neutron on first controller] *****************

2015-07-15 00:09:40,858 p=504 u=jarvis | changed: [10.2.18.207]

2015-07-15 00:09:40,859 p=504 u=jarvis | TASK: [config-controller | wait for neutron to start on first controller for NSX] ***

2015-07-15 00:24:41,321 p=504 u=jarvis | failed: [10.2.18.207] => {"elapsed": 900, "failed": true}

2015-07-15 00:24:41,321 p=504 u=jarvis | msg: Timeout when waiting for 127.0.0.1:9696

2015-07-15 00:24:41,322 p=504 u=jarvis | FATAL: all hosts have already failed -- aborting

viouser@localhost:~$

viouser@localhost:~$ ssh viouser@10.2.18.207

Welcome to Ubuntu 14.04.2 LTS (GNU/Linux 3.13.0-53-generic x86_64)

* Documentation: https://help.ubuntu.com/

Last login: Wed Jul 15 00:09:40 2015 from 10.2.18.192

viouser@controller01:~$ cat /var/log/neutron/neutron-server.log

cat: /var/log/neutron/neutron-server.log: Permission denied

viouser@controller01:~$ sudo cat /var/log/neutron/neutron-server.log

cat: /var/log/neutron/neutron-server.log: No such file or directory

viouser@controller01:~$ ls /var/log/neutron/neutron-server.log

ls: cannot access /var/log/neutron/neutron-server.log: Permission denied

viouser@controller01:~$ ifconfig

eth0 Link encap:Ethernet HWaddr 00:50:56:9a:cf:01

inet addr:10.2.18.207 Bcast:10.2.18.255 Mask:255.255.255.0

inet6 addr: fe80::250:56ff:fe9a:cf01/64 Scope:Link

UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1

RX packets:2941261 errors:0 dropped:11 overruns:0 frame:0

TX packets:1859699 errors:0 dropped:0 overruns:0 carrier:0

collisions:0 txqueuelen:1000

RX bytes:1863095802 (1.8 GB) TX bytes:206987906 (206.9 MB)

viouser@controller01:~$ sudo ls /var/log/neutron/neutron-server.log

ls: cannot access /var/log/neutron/neutron-server.log: No such file or directory

viouser@controller01:~$ exit

logout

Connection to 10.2.18.207 closed.

viouser@localhost:~$ ssh viouser@10.2.18.208

The authenticity of host '10.2.18.208 (10.2.18.208)' can't be established.

ECDSA key fingerprint is 27:ea:cc:24:c1:5d:9b:25:e9:59:5f:b9:df:7c:98:62.

Are you sure you want to continue connecting (yes/no)? yes

Warning: Permanently added '10.2.18.208' (ECDSA) to the list of known hosts.

Welcome to Ubuntu 14.04.2 LTS (GNU/Linux 3.13.0-53-generic x86_64)

* Documentation: https://help.ubuntu.com/

Last login: Wed Jul 15 00:09:40 2015 from 10.2.18.192

viouser@controller02:~$ sudo ls /var/log/neutron/neutron-server.log

ls: cannot access /var/log/neutron/neutron-server.log: No such file or directory

viouser@controller02:~$ ifconfig

eth0 Link encap:Ethernet HWaddr 00:50:56:9a:9b:39

inet addr:10.2.18.208 Bcast:10.2.18.255 Mask:255.255.255.0

ganapathibhat · ‎07-15-2015

vChrisSt,

I am using the local accountan dI have attached log file to other reply.

KarolSte · ‎07-15-2015

The actual name of the neutron log is "server.log".

From the log:

2015-07-16 03:50:26.051 16743 DEBUG neutron.plugins.vmware.vshield.edge_utils [req-3f6fc6a6-959a-458e-8625-5119f25f7f9e None] Failed to deploy Edge for router backup-b12089cd-2cf8 edge_deploy_result_sync /usr/lib/python2.7/dist-packages/neutron/plugins/vmware/vshield/edge_utils.py:1509
2015-07-16 03:50:26.061 16743 ERROR neutron.plugins.vmware.vshield.edge_appliance_driver [req-3f6fc6a6-959a-458e-8625-5119f25f7f9e None] NSXv: deploy edge failed.
2015-07-16 03:50:26.061 16743 TRACE neutron.plugins.vmware.vshield.edge_appliance_driver Traceback (most recent call last):
2015-07-16 03:50:26.061 16743 TRACE neutron.plugins.vmware.vshield.edge_appliance_driver   File "/usr/lib/python2.7/dist-packages/neutron/plugins/vmware/vshield/edge_appliance_driver.py", line 504, in deploy_edge
2015-07-16 03:50:26.061 16743 TRACE neutron.plugins.vmware.vshield.edge_appliance_driver     async=False)[0]
2015-07-16 03:50:26.061 16743 TRACE neutron.plugins.vmware.vshield.edge_appliance_driver   File "/usr/lib/python2.7/dist-packages/neutron/plugins/vmware/vshield/vcns.py", line 126, in deploy_edge
2015-07-16 03:50:26.061 16743 TRACE neutron.plugins.vmware.vshield.edge_appliance_driver     return self.do_request(HTTP_POST, uri, request, decode=False)
2015-07-16 03:50:26.061 16743 TRACE neutron.plugins.vmware.vshield.edge_appliance_driver   File "/usr/lib/python2.7/dist-packages/neutron/plugins/vmware/vshield/vcns.py", line 111, in do_request
2015-07-16 03:50:26.061 16743 TRACE neutron.plugins.vmware.vshield.edge_appliance_driver     headers, encodeParams)
2015-07-16 03:50:26.061 16743 TRACE neutron.plugins.vmware.vshield.edge_appliance_driver   File "/usr/lib/python2.7/dist-packages/neutron/plugins/vmware/vshield/vcns.py", line 72, in retry_wrapper
2015-07-16 03:50:26.061 16743 TRACE neutron.plugins.vmware.vshield.edge_appliance_driver     return f(*args, **kwargs)
2015-07-16 03:50:26.061 16743 TRACE neutron.plugins.vmware.vshield.edge_appliance_driver   File "/usr/lib/python2.7/dist-packages/neutron/plugins/vmware/vshield/vcns.py", line 97, in _client_request
2015-07-16 03:50:26.061 16743 TRACE neutron.plugins.vmware.vshield.edge_appliance_driver     return client(method, uri, params, headers, encodeParams)
2015-07-16 03:50:26.061 16743 TRACE neutron.plugins.vmware.vshield.edge_appliance_driver   File "/usr/lib/python2.7/dist-packages/neutron/plugins/vmware/vshield/common/VcnsApiClient.py", line 111, in request
2015-07-16 03:50:26.061 16743 TRACE neutron.plugins.vmware.vshield.edge_appliance_driver     raise cls(uri=uri, status=status, header=header, response=response)
2015-07-16 03:50:26.061 16743 TRACE neutron.plugins.vmware.vshield.edge_appliance_driver RequestBad: Request https://10.2.18.5/api/4.0/edges is Bad, response {"details":"Failed while updating the edgeAssist instance for edge edge-3535.","errorCode":10025,"rootCauseString":"all controllers are inactive","moduleName":"vShield Edge"}

It failed to create edge instances, and the error that neutron got from NSX Manager is that all Controllers are inactive. Two things to check:

1) Check if you controller is active

2) Check if you can manually create NSX edges in the cluster that you selected for Edge cluster.

ganapathibhat · ‎07-16-2015

Thanks Karol..

1) Check if you controller is active

>>>> The controller is up and running.

2) Check if you can manually create NSX edges in the cluster that you selected for Edge cluster.

>>>> I could manually configure edge in the given edge cluster.

Does NSX license and versions of the SW matter here ?

- I have expired (evaluation ) license for NSX. It should not matter here as I could create NSX edge manually, bust just wanted to share that info.

- Also I am using following versions: 6.1.3 NSX, vCenter 5.5 and VIO 1.0.0.0

ganapathibhat · ‎07-16-2015

Thanks Arvind...I appreciate the help...I am reachable on 408 307 3548....

ganapathibhat · ‎07-20-2015

KarolSte and team....Anything else you can think could be an issue? I appreciate your help..

ganapathibhat · ‎07-31-2015

This got resolved, after adding additional compute resource to the Management cluster. I already 4/6 Core 4 servers, added one more and installation went fine.

Thanks all for taking time and helping on this...Much appreciated..

All

VIO Installation issue - failure to start neutron