cant access VIO management server

[Steave_ualberta]

I am new to VIO and am deploying it as a part of my master' project. I have  the VIO vApp running on Vcenter. After logging in to management server I am able to ping Vcenter and Vsphere(esxi) . I can see my openstack extensions in Vcenter. I can successfully ping the Vcenter and Vsphere from my PC but I am not able to ping the management server and hence cant open the webpage. Like wise I cant ping the management server from esxi or vcenter ?

Can anyone provide some inputs on this please

[gjayavelu]

icmp echo is disabled on OMS, so you won't get response to ping. Try ssh to the management server.

[Steave_ualberta]

I am able to SSH to the server. But, without my PC able to ping the server how do I open the webpage ? I need that so I can run the below resolution to fix the plugin issue

Resolution 6

For VMware Integrated OpenStack versions 1.0.2 and later, a new interface is available for managing vCenter Server registration if plug-in issues are encountered. Please use this procedure to remedy plug-in issues:

1. Navigate to https://vio_managment_server_ip:8443/register-plugin
   
   Where vio_managment_server_ip is the IP address of your  VMware Integrated OpenStack management server.
   
   
2. Log in with your vCenter Server credentials.
   Note: Under Status, there is a red icon that indicates that the management server connection to vCenter Server is not correctly configured.
3. Click on the Fix button.
4. In the Certificate popup, verify the certificate. If the certificate is valid, click OK.
5. Log out of this screen and log in to the vSphere Web Client.

[gjayavelu]

icmp and http are different. If you the http url you mentioned does not work, then it could be different issue.

Can you ssh to OMS and run

sudo service oms status

The response should be

"oms start/running, process xxxx"

[gjayavelu]

May I know the actual issue?

Are you not able to see the VIO web plugin on vcenter?

Did you try logout and login vsphere web client?

[Steave_ualberta]

viouser@localhost:~$ sudo service oms status

[sudo] password for viouser:

oms stop/waiting

[Steave_ualberta]

How can I start the oms ?

[Steave_ualberta]

Yup I logged out and logged in back but was not able to see the plugin. I found the resolution in Vmware docs , but couldn't access the webpage to resolve it ? I tried trouble shooting with ping and found that my PC wont communicate with the management server.

[Steave_ualberta]

root@localhost:~# service oms start

oms start/running, process 1560

I got it to start

[Steave_ualberta]

My OMS.log only shows the below error

root@localhost:/var/log/oms# grep failed oms.log

[2017-01-27T16:23:06.213+0000] INFO  localhost-startStop-1| org.hibernate.tool.hbm2ddl.TableMetadata: HHH000037: Columns: [error_message, vc_rp_id, rack, vc_datastores, memory, volumes, cpu_number, action_failed, version, moid, node_group_id, action, guest_host_name, id, vm_name, host_name, status, power_status_changed]

[Steave_ualberta]

This is my boot sequence log

root@localhost:/var/log# more bootsequence.log

Fri Jan 27 12:13:35 UTC 2017 : Running firstboot scripts

Running commands from firstboot script /opt/vmware/vio/sysctl/firstboot.d//00-common

Creating SSH2 RSA key; this may take some time ...

Creating SSH2 DSA key; this may take some time ...

Creating SSH2 ECDSA key; this may take some time ...

Creating SSH2 ED25519 key; this may take some time ...

ssh stop/waiting

ssh start/running, process 1469

Running commands from firstboot script /opt/vmware/vio/sysctl/firstboot.d//02-setup-user

passwd: password expiry information changed.

groupadd: group 'viogrp' already exists

Group viogrp exists!

Generating public/private rsa key pair.

Your identification has been saved in /home/viouser/.ssh/id_rsa.

Your public key has been saved in /home/viouser/.ssh/id_rsa.pub.

The key fingerprint is:

3b:b5:d4:50:ed:c3:89:fd:3d:20:96:05:20:c6:f5:ca root@localhost.localdom

The key's randomart image is:

+--[ RSA 2048]----+

|      .o.o..o.   |

|      ... .. ..  |

|          ..o= . |

|        . .*..*  |

|        SE+ o .o.|

|         + .   .o|

|        o .     .|

|         .       |

|                 |

+-----------------+

Running commands from firstboot script /opt/vmware/vio/sysctl/firstboot.d//20-generate-keys

1. Generating keystore for OMS server...

   Exporting self-signed certificate...

Certificate stored in file </opt/vmware/vio/etc/oms.crt>

   Generating the csr...

   Exporting pkcs12 store...

   Exporting private key...

MAC verified OK

2. Generating password guard keystore...

generated certificate and private key in /opt/vmware/vio/etc

Running commands from firstboot script /opt/vmware/vio/sysctl/firstboot.d//30-generate-vault-pass

Running commands from firstboot script /opt/vmware/vio/sysctl/firstboot.d//48-configure_rsyslog

Running commands from firstboot script /opt/vmware/vio/sysctl/firstboot.d//50-clean-logs

Fri Jan 27 12:13:45 UTC 2017 : Running everyboot scripts

Running commands from everyboot script /opt/vmware/vio/sysctl/everyboot.d//00-common

Running commands from everyboot script /opt/vmware/vio/sysctl/everyboot.d//55-set-vami-guestinfo

Running commands from everyboot script /opt/vmware/vio/sysctl/everyboot.d//70-configure_ntp

* Stopping NTP server ntpd

   ...done.

* Starting NTP server ntpd

   ...done.

Running commands from everyboot script /opt/vmware/vio/sysctl/everyboot.d//71-update-exec-context

Setting up execution env: export VC information to vc.properties

Running commands from everyboot script /opt/vmware/vio/sysctl/everyboot.d//75-configure-tc-oms

Running commands from everyboot script /opt/vmware/vio/sysctl/everyboot.d//76-configure-tc-osvmw

Running commands from everyboot script /opt/vmware/vio/sysctl/everyboot.d//100-boot-complete

Sending boot-complete signal to upstart

Fri Jan 27 12:25:45 UTC 2017 : Running everyboot scripts

Running commands from everyboot script /opt/vmware/vio/sysctl/everyboot.d//00-common

Running commands from everyboot script /opt/vmware/vio/sysctl/everyboot.d//55-set-vami-guestinfo

Running commands from everyboot script /opt/vmware/vio/sysctl/everyboot.d//70-configure_ntp

* Stopping NTP server ntpd

   ...done.

* Starting NTP server ntpd

   ...done.

Running commands from everyboot script /opt/vmware/vio/sysctl/everyboot.d//71-update-exec-context

Setting up execution env: export VC information to vc.properties

Running commands from everyboot script /opt/vmware/vio/sysctl/everyboot.d//75-configure-tc-oms

Running commands from everyboot script /opt/vmware/vio/sysctl/everyboot.d//76-configure-tc-osvmw

Running commands from everyboot script /opt/vmware/vio/sysctl/everyboot.d//100-boot-complete

Sending boot-complete signal to upstart

Fri Jan 27 12:48:07 UTC 2017 : Running everyboot scripts

Running commands from everyboot script /opt/vmware/vio/sysctl/everyboot.d//00-common

Running commands from everyboot script /opt/vmware/vio/sysctl/everyboot.d//55-set-vami-guestinfo

Running commands from everyboot script /opt/vmware/vio/sysctl/everyboot.d//70-configure_ntp

* Stopping NTP server ntpd

   ...done.

* Starting NTP server ntpd

   ...done.

Running commands from everyboot script /opt/vmware/vio/sysctl/everyboot.d//71-update-exec-context

Setting up execution env: export VC information to vc.properties

Running commands from everyboot script /opt/vmware/vio/sysctl/everyboot.d//75-configure-tc-oms

Running commands from everyboot script /opt/vmware/vio/sysctl/everyboot.d//76-configure-tc-osvmw

Running commands from everyboot script /opt/vmware/vio/sysctl/everyboot.d//100-boot-complete

Sending boot-complete signal to upstart

Fri Jan 27 13:00:01 UTC 2017 : Running everyboot scripts

Running commands from everyboot script /opt/vmware/vio/sysctl/everyboot.d//00-common

Running commands from everyboot script /opt/vmware/vio/sysctl/everyboot.d//55-set-vami-guestinfo

Running commands from everyboot script /opt/vmware/vio/sysctl/everyboot.d//70-configure_ntp

* Stopping NTP server ntpd

   ...done.

* Starting NTP server ntpd

   ...done.

Running commands from everyboot script /opt/vmware/vio/sysctl/everyboot.d//71-update-exec-context

Setting up execution env: export VC information to vc.properties

Running commands from everyboot script /opt/vmware/vio/sysctl/everyboot.d//75-configure-tc-oms

Running commands from everyboot script /opt/vmware/vio/sysctl/everyboot.d//76-configure-tc-osvmw

Running commands from everyboot script /opt/vmware/vio/sysctl/everyboot.d//100-boot-complete

Sending boot-complete signal to upstart

Fri Jan 27 13:28:29 UTC 2017 : Running everyboot scripts

Running commands from everyboot script /opt/vmware/vio/sysctl/everyboot.d//00-common

Running commands from everyboot script /opt/vmware/vio/sysctl/everyboot.d//55-set-vami-guestinfo

Running commands from everyboot script /opt/vmware/vio/sysctl/everyboot.d//70-configure_ntp

Unable to find the ovf environment.

Fri Jan 27 16:06:28 UTC 2017 : Running everyboot scripts

Running commands from everyboot script /opt/vmware/vio/sysctl/everyboot.d//00-common

Running commands from everyboot script /opt/vmware/vio/sysctl/everyboot.d//55-set-vami-guestinfo

Running commands from everyboot script /opt/vmware/vio/sysctl/everyboot.d//70-configure_ntp

Unable to find the ovf environment.

[gjayavelu]

Did you enter the password in ovf properties while deploying the OVF? Also, I'm not sure if the messages can cause issues to bring up OMS.

After oms service was restarted, please logout and login on vsphere client.

If none of the issues solved, can you please upload full oms.log file?

[Steave_ualberta]

I have attached the OMS log. Thank you for helping me with this issue. it really has me on edge. The main thing I am struggling with here is to get the plugin for VIO in vcenter

[Steave_ualberta]

I can also see the Vcenter connected to VIO please see the attached snap . I have also attached the omjs properties file

[gjayavelu]

I see ssl error from the log.

[2017-01-27T16:47:36.905+0000] INFO  localhost-startStop-1| com.vmware.aurora.vc.vcservice.VcService: VC login on behalf of {localhost-startStop-1:RegisterPlugin{1}} 112ms

[2017-01-27T16:47:36.936+0000] INFO  localhost-startStop-1| com.vmware.openstack.manager.PatchManager: Package: vio-osvmw-plugin Version: 3.0.0.4345506

[2017-01-27T16:47:36.937+0000] INFO  localhost-startStop-1| com.vmware.openstack.plugin.NgcRegistrar: Trying to update the old plugin...org.os.vmw.plugin

[2017-01-27T16:47:43.161+0000] INFO  localhost-startStop-1| com.vmware.openstack.plugin.NgcRegistrar: org.os.vmw.plugin plugin updation succeeded!

[2017-01-27T16:47:43.174+0000] INFO  localhost-startStop-1| com.vmware.openstack.plugin.NgcOsvmwRegistrar: OSVMW NGC plugin auto-installation succeeded!

[2017-01-27T16:47:43.177+0000] ERROR localhost-startStop-1| com.vmware.openstack.security.sso.utils.SecurityUtils: Authentication error :null

[2017-01-27T16:47:43.177+0000] ERROR localhost-startStop-1| com.vmware.openstack.manager.PluginRegisterManager: SSO server is misconfigured

[2017-01-27T16:47:43.181+0000] INFO  localhost-startStop-1| com.vmware.openstack.service.resmgmt.impl.ResourceInitializerService: resource initialized? true

[2017-01-27T16:47:43.182+0000] INFO  localhost-startStop-1| com.vmware.openstack.manager.PluginRegisterManager: Resource initialization completed

can you try the viocli deployment getlogs command and see if you hit error similar to https://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=21468...

If yes, then try following the instructions?

If no cert related issue, at least the following steps can be tried (which is subset of the instructions in kb article).

1. Stop oms services (do it as sudo)

service oms stop

service osvmw stop

2. backup /opt/vmware/vio/etc

3. Snapshot OMS

4. Modify /opt/vmware/vio/etc/omjs.properties and set "oms.extension.registered" to false to allow oms to register with VC with new certificate.

5. stop & start VIO vApp

6. Verify and remove snapshot

[gjayavelu]

Also, always logout, give it couple of minutes and login to vsphere client whenever you restart oms.

[Steave_ualberta]

Thanks i'll try it out, this is what the viocli log shows

root@localhost:/opt/vmware/vio/etc/test/201701281316/vio_mgmt/logs/viocli# more viocli.log

2017-01-28 13:16:34,681 viocli       DEBUG    ###### Starting execution of /usr/local/bin/viocli deployment getlogs ######

2017-01-28 13:16:34,681 jarvis.cache.file_system DEBUG    Checking cache store directory /var/lib/vio/jarvis/cache exists

2017-01-28 13:16:35,065 p=1328 u=root |  INFO     PLAY [127.0.0.1] **************************************************************

2017-01-28 13:16:35,066 p=1328 u=root |  INFO     GATHERING FACTS ***************************************************************

2017-01-28 13:16:35,963 p=1328 u=root |  INFO     ok: [127.0.0.1]

2017-01-28 13:16:35,964 p=1328 u=root |  INFO     TASK: [support-local | create the logs directory] *****************************

2017-01-28 13:16:36,088 p=1328 u=root |  INFO     changed: [127.0.0.1]

2017-01-28 13:16:36,089 p=1328 u=root |  INFO     TASK: [support-local | create the configure directory] ************************

2017-01-28 13:16:36,212 p=1328 u=root |  INFO     changed: [127.0.0.1]

2017-01-28 13:16:36,213 p=1328 u=root |  INFO     TASK: [support-local | check log and configuration files existed] *************

2017-01-28 13:16:36,335 p=1328 u=root |  INFO     ok: [127.0.0.1] => (item=({'dst': 'logs'}, '/var/log/oms'))

2017-01-28 13:16:36,451 p=1328 u=root |  INFO     ok: [127.0.0.1] => (item=({'dst': 'logs'}, '/var/log/jarvis'))

2017-01-28 13:16:36,566 p=1328 u=root |  INFO     ok: [127.0.0.1] => (item=({'dst': 'logs'}, '/var/log/viocli'))

2017-01-28 13:16:36,677 p=1328 u=root |  INFO     ok: [127.0.0.1] => (item=({'dst': 'logs'}, '/var/log/viopatch'))

2017-01-28 13:16:36,790 p=1328 u=root |  INFO     ok: [127.0.0.1] => (item=({'dst': 'logs'}, '/opt/vmware/var/log/firstboot'))

2017-01-28 13:16:36,902 p=1328 u=root |  INFO     ok: [127.0.0.1] => (item=({'dst': 'logs'}, '/opt/vmware/var/log/subsequentboot'))

2017-01-28 13:16:37,018 p=1328 u=root |  INFO     ok: [127.0.0.1] => (item=({'dst': 'logs'}, '/var/log/syslog'))

2017-01-28 13:16:37,133 p=1328 u=root |  INFO     ok: [127.0.0.1] => (item=({'dst': 'logs'}, '/var/log/dpkg.log'))

2017-01-28 13:16:37,254 p=1328 u=root |  INFO     ok: [127.0.0.1] => (item=({'dst': 'logs'}, '/var/log/bootsequence.log'))

2017-01-28 13:16:37,376 p=1328 u=root |  INFO     ok: [127.0.0.1] => (item=({'dst': 'logs'}, '/opt/vmware/dbg/cores'))

2017-01-28 13:16:37,488 p=1328 u=root |  INFO     ok: [127.0.0.1] => (item=({'dst': 'logs'}, '/var/log/osvmw'))

2017-01-28 13:16:37,601 p=1328 u=root |  INFO     ok: [127.0.0.1] => (item=({'dst': 'logs'}, '/var/log/viomon'))

2017-01-28 13:16:37,712 p=1328 u=root |  INFO     ok: [127.0.0.1] => (item=({'dst': 'logs'}, '/var/log/vmware/vapi'))

2017-01-28 13:16:37,824 p=1328 u=root |  INFO     ok: [127.0.0.1] => (item=({'dst': 'config'}, '/opt/vmware/vio/etc/vc.properties'))

2017-01-28 13:16:37,937 p=1328 u=root |  INFO     ok: [127.0.0.1] => (item=({'dst': 'config'}, '/opt/vmware/vio/etc/omjs.properties'))

2017-01-28 13:16:38,050 p=1328 u=root |  INFO     ok: [127.0.0.1] => (item=({'dst': 'config'}, '/etc/ntp.conf'))

2017-01-28 13:16:38,162 p=1328 u=root |  INFO     ok: [127.0.0.1] => (item=({'dst': 'config'}, '/etc/rsyslog.conf'))

2017-01-28 13:16:38,275 p=1328 u=root |  INFO     ok: [127.0.0.1] => (item=({'dst': 'config'}, '/etc/rsyslog.d'))

2017-01-28 13:16:38,388 p=1328 u=root |  INFO     ok: [127.0.0.1] => (item=({'dst': 'config'}, '/var/lib/vio/jarvis'))

2017-01-28 13:16:38,501 p=1328 u=root |  INFO     ok: [127.0.0.1] => (item=({'dst': 'config'}, '/etc/hosts'))

2017-01-28 13:16:38,502 p=1328 u=root |  INFO     TASK: [support-local | copying syslog] ****************************************

2017-01-28 13:16:38,944 p=1328 u=root |  INFO     changed: [127.0.0.1] => (item={'invocation': {'module_name': u'stat', 'module_args': u'path=/var/log/oms'}, 'item': [{

'dst': 'logs'}, '/var/log/oms'], u'stat': {u'islnk': False, u'uid': 998, u'exists': True, u'rgrp': True, u'xoth': False, u'rusr': True, u'woth': False, u'nlink': 2, u'i

ssock': False, u'mtime': 1485534182.4886754, u'xusr': True, u'atime': 1485589948.0653641, u'inode': 1264212, u'isgid': False, u'size': 4096, u'isdir': True, u'ctime': 1

485534182.4886754, u'isblk': False, u'wgrp': False, u'xgrp': True, u'isuid': False, u'dev': 2054, u'roth': False, u'isreg': False, u'isfifo': False, u'mode': u'0750', u

'pw_name': u'omsapp', u'gid': 4, u'ischr': False, u'wusr': True}, u'changed': False})

2017-01-28 13:16:39,076 p=1328 u=root |  INFO     changed: [127.0.0.1] => (item={'invocation': {'module_name': u'stat', 'module_args': u'path=/var/log/jarvis'}, 'item':

[{'dst': 'logs'}, '/var/log/jarvis'], u'stat': {u'islnk': False, u'uid': 999, u'exists': True, u'rgrp': True, u'xoth': False, u'rusr': True, u'woth': False, u'nlink':

2, u'issock': False, u'mtime': 1473068110.3191955, u'xusr': True, u'atime': 1473068121.2070587, u'inode': 1288140, u'isgid': False, u'size': 4096, u'isdir': True, u'cti

me': 1473068110.3311954, u'isblk': False, u'wgrp': False, u'xgrp': True, u'isuid': False, u'dev': 2054, u'roth': False, u'isreg': False, u'isfifo': False, u'mode': u'07

50', u'pw_name': u'jarvis', u'gid': 4, u'ischr': False, u'wusr': True}, u'changed': False})

[Steave_ualberta]

I tried it but still no luck , cant get the plugin . I have attached the new logs

[Steave_ualberta]

Also " Go to the vmware integrated openstack plugin and reconnect the OMS " . When I restarted the vAPP the OMS came up and the status was running . Do i have to do another step ? I cant find it in vcenter an option to connect OMS.  My implementation is as below

Vsphere version 6.0 has Vcenter(VCSA 6.0) deployed on it and VIO 3.0 on vcenter .

In the plugin manager on vsphere i see nothing . Please find all the snaps attached

[gjayavelu]

One last suggestion. Try unregistering the plugin as mentioned below and restart oms which will register again. 

Removing VIO from vCenter Sever manually (2146541) | VMware KB