VMware Cloud Community
raflax
Contributor
Contributor

Tried to upgrade VIO from 2.0 to 2.5, now connection to OMS fails

I had a working VIO version 2.0 and tried to upgrade it to 2.5.

The upgrade steps succeeded through the viocli dbverify step.

I went to provision the new VIO deployment and vCenter can no longer connect to the OMS instance.  I can ssh to it just fine, I've shutdown and restarted the vApp that contains OMS to no avail.  I tried to connect directly to the web interface at https://<OMS_server>:8443/VIO and logged in with the vCenter credentials as it suggests and I get the resulting http 500 error page from the servlet.

Anyone else get this and get past it?

Thanks.

Reply
0 Kudos
46 Replies
ZhangAdam
VMware Employee
VMware Employee

Can you tell me the steps you did for upgrade, and attach oms log?

Reply
0 Kudos
ZhangAdam
VMware Employee
VMware Employee

Try to clean browser caches, and restart vsphere-client service, see if it helps.

Reply
0 Kudos
raflax
Contributor
Contributor

Unfortunately, this did not help.  I've rebooted VCSA, as well as OMS.  I've tried re-connecting vCenter to OMS and the connection still fails.

Reply
0 Kudos
raflax
Contributor
Contributor

Here are the steps I followed:

1. I did not add more IP addresses since I had already done this step last time I upgraded (from 1.0 to 2.0) and I had previously removed the 1.0 instance

2. Added and Installed the upgrade bundle using viopatch utility on OMS - successfully ran with no errors

3. Ran viocli dbverify - successfully ran with no errors

4. Opened browser session to vCenter using the Web Client and tried to navigate to VIO and ran into this problem of being unable to connect.

I've attached two OMS log files, one from yesterday when I tried to perform the upgrade, and one from today which is the most recent.

Reply
0 Kudos
raflax
Contributor
Contributor

I also now have the added problem of cinder-api not starting.

I tried viocli services stop/start and now the restart fails at the step where it starts up cinder.  I ssh'd to one of the controller VMs and looked at the cinder-api logs and I'm seeing the following error message:

2016-06-08 13:45:15.721 24615 CRITICAL cinder [-] DBConnectionError: (OperationalError) (2013, 'Lost connection to MySQL server at \'reading initial communication packet\', system error: 0 "Internal error/check (Not system error)"') None None

Reply
0 Kudos
yjia
VMware Employee
VMware Employee

Can you please check if the oms can ping VC 10.127.11.22

I find the follow error message in the oms log

ERROR VcEventListener| com.vmware.openstack.utils.IpAddressUtil: Can't connect to VC from OMS server 10.127.11.22

java.net.NoRouteToHostException: No route to host

Reply
0 Kudos
raflax
Contributor
Contributor

Yes just tried it.. can ping the IP@ as well as the DNS name successfully.

Reply
0 Kudos
raflax
Contributor
Contributor

There was a point where I rebooted VCSA and that time coincides with the inability of OMS to communicate with vCenter.

Reply
0 Kudos
yjia
VMware Employee
VMware Employee

Thanks for your reply.

Now can you please try the follow and let me know the result.

1> login to management server

2> edit /opt/vmware/vio/etc/omjs.properties change oms.extension.registered =false

save and restart oms service by $> restart oms

3> wait the oms server come up

4> try to connect to management server from vSphere UI

if the above steps still don't works, then please run $>viocli deployment getlogs

on management server to collect all the logs which can help us to identify the issue.

Reply
0 Kudos
raflax
Contributor
Contributor

Ok I've just tried this and I am still unable to connect from the Web Client.  I get the same error message (Attached Image)

When I run viocli deployment getlogs I get the following:

root@nsx11-vio-manager:/var/log/oms# viocli deployment getlogs

changed: [10.127.11.48]

changed: [10.127.11.46]

changed: [10.127.11.47]

TASK: [support-common | gather common info] ***********************************

changed: [10.127.11.48]

changed: [10.127.11.47]

changed: [10.127.11.46]

TASK: [support-common | cleanup] **********************************************

changed: [10.127.11.48]

changed: [10.127.11.47]

changed: [10.127.11.46]

TASK: [support-common | deploy node specific support script] ******************

ok: [10.127.11.48]

ok: [10.127.11.46]

ok: [10.127.11.47]

TASK: [support-common | gather node specific support info] ********************

changed: [10.127.11.48]

changed: [10.127.11.47]

changed: [10.127.11.46]

TASK: [support-common | get support bundle] ***********************************

fatal: [10.127.11.46] => One or more undefined variables: 'support_log_role' is undefined

fatal: [10.127.11.48] => One or more undefined variables: 'support_log_role' is undefined

fatal: [10.127.11.47] => One or more undefined variables: 'support_log_role' is undefined

FATAL: all hosts have already failed -- aborting

support-db failed because the following nodes were unreachable: ['10.127.11.48', '10.127.11.47', '10.127.11.46'].

I can definitely ping each of those IP@'s and I've verified that mysql is running properly on each of these..

Reply
0 Kudos
ssurana
VMware Employee
VMware Employee

Can you tell if the upgrade patch installation went successful?

Can you provide the following:

1. log files under /var/log/viopatch directory

2. output of the following commands:

     a. "dpkg --list | grep vio-"

     b. "viopatch version"

     c. "viopatch list"

Thanks,

~ Sidharth

Reply
0 Kudos
raflax
Contributor
Contributor

I've attached the two log files from that directory.  One is the log for the 1.0 to 2.0 upgrade, and the other is the one for the upgrade from 2.0 to 2.5

root@nsx11-vio-manager:/var/tmp# dpkg --list | grep vio-

ii  vio-1.0-upgrade                            2.0.0.3037964                    all          Upgrades VIO to version 1.0.0 build 3037964

hi  vio-ansible                                2.5.0.3955000                    all          VIO ansible component

ii  vio-binaries                               1.0.0.2824223                    all          VIO packaged component binaries

hi  vio-cli                                    2.0.0.3037964                    all          VIO CLI component provides vio* cli commands

hi  vio-jarvis                                 2.5.0.3955000                    all          VIO jarvis component

hi  vio-licenses                               2.5.0.3955000                    all          Open source license files for VMware Integrated OpenStack

hi  vio-monitor                                2.5.0.3955000                    all          Monitor status of control plane of VMware Integrated OpenStack

hi  vio-oms                                    2.0.0.3037964                    all          VIO oms component

ii  vio-openstack-images                       1.0.0.2824223                    all          VIO packaged OpenStack images

hi  vio-osvmw-plugin                           2.5.0.3955000                    all          VIO openstack WebClient plugin component

hi  vio-plugin                                 2.5.0.3955000                    all          VIO manager WebClient plugin component

hi  vio-repository                             2.5.0.3955000                    all          VIO package repository component

hi  vio-sysctl                                 2.5.0.3955000                    all          VIO system control scripts component

ii  vio-upgrade-2.5                            2.5.0.3955000                    all          Upgrades VIO to version 2.5.0 build 3955000

hi  vio-vpostgres                              2.5.0.3955000                    all          VIO vpostgres configuration component

root@nsx11-vio-manager:/var/tmp# viopatch version

2.0.0.3037964

root@nsx11-vio-manager:/var/tmp# viopatch list

Name             Version        Type    Installed

---------------  -------------  ------  -----------

vio-upgrade-2.5  2.5.0.3955000  infra   Yes

vio-1.0-upgrade  2.0.0.3037964  infra   Yes

Reply
0 Kudos
raflax
Contributor
Contributor

Looking at the 2.0 to 2.5 upgrade log you will notice that it failed the first time through on the vio-repository module.  During the upgrade of that component the system ran out of disk space.  I fixed the disk space problem and re-ran the viopatch install step which then succeeded.

Reply
0 Kudos
ssurana
VMware Employee
VMware Employee

this is kind of bizarre.. from the ansible logs it suggests that it did install the new version of the vio-cli and vio-oms package but from the console output of "dpkg --list" suggests it didn't

Can you provide the /var/log/dpkg.log and /var/log/syslog file from your management server. After you copy them do the following:

We will manually re run the patch installation ansible for the upgrade patch.

For that run the following command on you system and provide us the console output

1. cd /opt/vmware/vio/patches/vio-upgrade-2.5/2.5.0.3955000/

2. ansible-playbook -i localhost, patch-install.yml

This will try to re-install the newer version of the components again.

3. Once the #2 command is done try running the "dpkg --list| grep vio-" to confirm if the vio-cli and vio-oms components got update to the 2.5 versions.

~ Sidharth

Reply
0 Kudos
raflax
Contributor
Contributor

I've attached tarballs of the dpkg and syslog log files..

Here's the output from the manual ansible-playbook run:

root@nsx11-vio-manager:/opt/vmware/vio/patches/vio-upgrade-2.5/2.5.0.3955000# ansible-playbook -i localhost, patch-install.yml

PLAY [localhost] **************************************************************

GATHERING FACTS ***************************************************************

ok: [localhost]

TASK: [record the current system version] *************************************

changed: [localhost]

TASK: [record the current system build] ***************************************

changed: [localhost]

TASK: [enable dpkg force overwrite option] ************************************

ok: [localhost]

TASK: [run apt-get update] ****************************************************

ok: [localhost]

TASK: [record the existing versions] ******************************************

changed: [localhost] => (item={'Version': '9.3.9.0', 'Package': 'vmware-postgres-osslibs-server'})

changed: [localhost] => (item={'Version': '2.5.0.3955000', 'Package': 'vio-monitor'})

changed: [localhost] => (item={'Version': '2.5.0.3955000', 'Package': 'vio-ansible'})

changed: [localhost] => (item={'Version': '9.3.9.0', 'Package': 'vmware-postgres-osslibs'})

changed: [localhost] => (item={'Version': '2.5.0.3955000', 'Package': 'vio-licenses'})

changed: [localhost] => (item={'Version': '9.3.9.0', 'Package': 'vmware-postgres-libs'})

changed: [localhost] => (item={'Version': '2.5.0.3955000', 'Package': 'vio-vpostgres'})

changed: [localhost] => (item={'Version': '2.5.0.3955000', 'Package': 'vio-repository'})

changed: [localhost] => (item={'Version': '1.7.0u101', 'Package': 'vmware-jre'})

changed: [localhost] => (item={'Version': '2.5.0.3955000', 'Package': 'vio-sysctl'})

changed: [localhost] => (item={'Version': '9.3.9.0', 'Package': 'vmware-postgres-server'})

changed: [localhost] => (item={'Version': '3.1.4.SR1', 'Package': 'pivotal-tc-server-standard'})

changed: [localhost] => (item={'Version': '2.5.0.3955000', 'Package': 'vio-plugin'})

changed: [localhost] => (item={'Version': '2.5.0.3955000', 'Package': 'vio-osvmw-plugin'})

changed: [localhost] => (item={'Version': '2.5.0.3955000', 'Package': 'vio-cli'})

changed: [localhost] => (item={'Version': '2.5.0.3955000', 'Package': 'vio-jarvis'})

changed: [localhost] => (item={'Version': '2.5.0.3955000', 'Package': 'vio-oms'})

changed: [localhost] => (item={'Version': '9.3.9.0', 'Package': 'vmware-postgres'})

TASK: [stat path={{ patch_path }}/preinstalled.yml] ***************************

ok: [localhost]

TASK: [template src=preinstalled.j2 dest={{ patch_path }}/preinstalled.yml] ***

skipping: [localhost]

TASK: [touch marker file patch-mgmt.start] ************************************

changed: [localhost]

TASK: [unhold the vio components] *********************************************

changed: [localhost] => (item={'Version': '9.3.9.0', 'Package': 'vmware-postgres-osslibs-server'})

changed: [localhost] => (item={'Version': '2.5.0.3955000', 'Package': 'vio-monitor'})

changed: [localhost] => (item={'Version': '2.5.0.3955000', 'Package': 'vio-ansible'})

changed: [localhost] => (item={'Version': '9.3.9.0', 'Package': 'vmware-postgres-osslibs'})

changed: [localhost] => (item={'Version': '2.5.0.3955000', 'Package': 'vio-licenses'})

changed: [localhost] => (item={'Version': '9.3.9.0', 'Package': 'vmware-postgres-libs'})

changed: [localhost] => (item={'Version': '2.5.0.3955000', 'Package': 'vio-vpostgres'})

changed: [localhost] => (item={'Version': '2.5.0.3955000', 'Package': 'vio-repository'})

changed: [localhost] => (item={'Version': '1.7.0u101', 'Package': 'vmware-jre'})

changed: [localhost] => (item={'Version': '2.5.0.3955000', 'Package': 'vio-sysctl'})

changed: [localhost] => (item={'Version': '9.3.9.0', 'Package': 'vmware-postgres-server'})

changed: [localhost] => (item={'Version': '3.1.4.SR1', 'Package': 'pivotal-tc-server-standard'})

changed: [localhost] => (item={'Version': '2.5.0.3955000', 'Package': 'vio-plugin'})

changed: [localhost] => (item={'Version': '2.5.0.3955000', 'Package': 'vio-osvmw-plugin'})

changed: [localhost] => (item={'Version': '2.5.0.3955000', 'Package': 'vio-cli'})

changed: [localhost] => (item={'Version': '2.5.0.3955000', 'Package': 'vio-jarvis'})

changed: [localhost] => (item={'Version': '2.5.0.3955000', 'Package': 'vio-oms'})

changed: [localhost] => (item={'Version': '9.3.9.0', 'Package': 'vmware-postgres'})

TASK: [debug msg="Moving services state to stopped"] **************************

ok: [localhost] => {

    "msg": "Moving services state to stopped"

}

TASK: [service oms] ***********************************************************

changed: [localhost]

TASK: [service osvmw] *********************************************************

changed: [localhost]

TASK: [service jarvis] ********************************************************

changed: [localhost]

TASK: [service vpostgres] *****************************************************

changed: [localhost]

TASK: [install new vio components] ********************************************

ok: [localhost] => (item={'Version': '9.3.9.0', 'Package': 'vmware-postgres-osslibs-server'})

ok: [localhost] => (item={'Version': '2.5.0.3955000', 'Package': 'vio-monitor'})

ok: [localhost] => (item={'Version': '2.5.0.3955000', 'Package': 'vio-ansible'})

ok: [localhost] => (item={'Version': '9.3.9.0', 'Package': 'vmware-postgres-osslibs'})

ok: [localhost] => (item={'Version': '2.5.0.3955000', 'Package': 'vio-licenses'})

ok: [localhost] => (item={'Version': '9.3.9.0', 'Package': 'vmware-postgres-libs'})

ok: [localhost] => (item={'Version': '2.5.0.3955000', 'Package': 'vio-vpostgres'})

ok: [localhost] => (item={'Version': '2.5.0.3955000', 'Package': 'vio-repository'})

ok: [localhost] => (item={'Version': '1.7.0u101', 'Package': 'vmware-jre'})

ok: [localhost] => (item={'Version': '2.5.0.3955000', 'Package': 'vio-sysctl'})

ok: [localhost] => (item={'Version': '9.3.9.0', 'Package': 'vmware-postgres-server'})

ok: [localhost] => (item={'Version': '3.1.4.SR1', 'Package': 'pivotal-tc-server-standard'})

ok: [localhost] => (item={'Version': '2.5.0.3955000', 'Package': 'vio-plugin'})

ok: [localhost] => (item={'Version': '2.5.0.3955000', 'Package': 'vio-osvmw-plugin'})

changed: [localhost] => (item={'Version': '2.5.0.3955000', 'Package': 'vio-cli'})

ok: [localhost] => (item={'Version': '2.5.0.3955000', 'Package': 'vio-jarvis'})

changed: [localhost] => (item={'Version': '2.5.0.3955000', 'Package': 'vio-oms'})

ok: [localhost] => (item={'Version': '9.3.9.0', 'Package': 'vmware-postgres'})

TASK: [stat path=/opt/vmware/vio/sysctl/everyboot.d/71-update-exec-context] ***

ok: [localhost]

TASK: [configure oms - run script 71-update-exec-context] *********************

changed: [localhost]

TASK: [stat path=/opt/vmware/vio/sysctl/everyboot.d/73-configure_vcsso] *******

ok: [localhost]

TASK: [configure oms - run script 73-configure_vcsso] *************************

skipping: [localhost]

TASK: [stat path=/opt/vmware/vio/sysctl/everyboot.d/75-configure-tc-oms] ******

ok: [localhost]

TASK: [configure oms - run script 75-configure-tc-oms] ************************

changed: [localhost]

TASK: [stat path=/opt/vmware/vio/sysctl/everyboot.d/76-configure-tc-osvmw] ****

ok: [localhost]

TASK: [configure osvmw - run script 76-configure-tc-osvmw] ********************

changed: [localhost]

TASK: [create /var/log/viomon if it does not exist] ***************************

ok: [localhost]

TASK: [debug msg="Moving services state to started"] **************************

ok: [localhost] => {

    "msg": "Moving services state to started"

}

TASK: [service oms] ***********************************************************

changed: [localhost]

TASK: [service osvmw] *********************************************************

ok: [localhost]

TASK: [service jarvis] ********************************************************

changed: [localhost]

TASK: [service vpostgres] *****************************************************

ok: [localhost]

TASK: [hold the vio components] ***********************************************

changed: [localhost] => (item={'Version': '9.3.9.0', 'Package': 'vmware-postgres-osslibs-server'})

changed: [localhost] => (item={'Version': '2.5.0.3955000', 'Package': 'vio-monitor'})

changed: [localhost] => (item={'Version': '2.5.0.3955000', 'Package': 'vio-ansible'})

changed: [localhost] => (item={'Version': '9.3.9.0', 'Package': 'vmware-postgres-osslibs'})

changed: [localhost] => (item={'Version': '2.5.0.3955000', 'Package': 'vio-licenses'})

changed: [localhost] => (item={'Version': '9.3.9.0', 'Package': 'vmware-postgres-libs'})

changed: [localhost] => (item={'Version': '2.5.0.3955000', 'Package': 'vio-vpostgres'})

changed: [localhost] => (item={'Version': '2.5.0.3955000', 'Package': 'vio-repository'})

changed: [localhost] => (item={'Version': '1.7.0u101', 'Package': 'vmware-jre'})

changed: [localhost] => (item={'Version': '2.5.0.3955000', 'Package': 'vio-sysctl'})

changed: [localhost] => (item={'Version': '9.3.9.0', 'Package': 'vmware-postgres-server'})

changed: [localhost] => (item={'Version': '3.1.4.SR1', 'Package': 'pivotal-tc-server-standard'})

changed: [localhost] => (item={'Version': '2.5.0.3955000', 'Package': 'vio-plugin'})

changed: [localhost] => (item={'Version': '2.5.0.3955000', 'Package': 'vio-osvmw-plugin'})

changed: [localhost] => (item={'Version': '2.5.0.3955000', 'Package': 'vio-cli'})

changed: [localhost] => (item={'Version': '2.5.0.3955000', 'Package': 'vio-jarvis'})

changed: [localhost] => (item={'Version': '2.5.0.3955000', 'Package': 'vio-oms'})

changed: [localhost] => (item={'Version': '9.3.9.0', 'Package': 'vmware-postgres'})

TASK: [re-generate package index for the vioapp] ******************************

changed: [localhost]

TASK: [re-generate package index for the viopatch] ****************************

changed: [localhost]

TASK: [run apt-get update] ****************************************************

ok: [localhost]

TASK: [lineinfile ] ***********************************************************

ok: [localhost]

TASK: [lineinfile ] ***********************************************************

ok: [localhost]

TASK: [file ] *****************************************************************

changed: [localhost]

PLAY RECAP ********************************************************************

localhost                  : ok=35   changed=19   unreachable=0    failed=0  

Here's the updated dpkg results:

ii  vio-1.0-upgrade                            2.0.0.3037964                    all          Upgrades VIO to version 1.0.0 build 3037964

hi  vio-ansible                                2.5.0.3955000                    all          VIO ansible component

ii  vio-binaries                               1.0.0.2824223                    all          VIO packaged component binaries

hi  vio-cli                                    2.0.0.3037964                    all          VIO CLI component provides vio* cli commands

hi  vio-jarvis                                 2.5.0.3955000                    all          VIO jarvis component

hi  vio-licenses                               2.5.0.3955000                    all          Open source license files for VMware Integrated OpenStack

hi  vio-monitor                                2.5.0.3955000                    all          Monitor status of control plane of VMware Integrated OpenStack

hi  vio-oms                                    2.5.0.3955000                    all          VIO oms component

ii  vio-openstack-images                       1.0.0.2824223                    all          VIO packaged OpenStack images

hi  vio-osvmw-plugin                           2.5.0.3955000                    all          VIO openstack WebClient plugin component

hi  vio-plugin                                 2.5.0.3955000                    all          VIO manager WebClient plugin component

hi  vio-repository                             2.5.0.3955000                    all          VIO package repository component

hi  vio-sysctl                                 2.5.0.3955000                    all          VIO system control scripts component

ii  vio-upgrade-2.5                            2.5.0.3955000                    all          Upgrades VIO to version 2.5.0 build 3955000

hi  vio-vpostgres                              2.5.0.3955000                    all          VIO vpostgres configuration component

Looks like vio-oms got updated this time, but not vio-cli

Reply
0 Kudos
raflax
Contributor
Contributor

Re-running the ansible-playbook which updated the vio-oms module did fix the Web Client UI.  So I can now connect to VIO once again.  Thank you!!

I'm still curious as to why the vio-cli won't update?

Reply
0 Kudos
ssurana
VMware Employee
VMware Employee

I am curious too!

Can you try run "apt-get install vio-cli=2.5.0.3955000"  and see what is going on there and share the output

Reply
0 Kudos
raflax
Contributor
Contributor

root@nsx11-vio-manager:/opt/vmware/vio/patches/downloads# apt-get install vio-cli=2.5.0.3955000

Reading package lists... Done

Building dependency tree      

Reading state information... Done

The following packages were automatically installed and are no longer required:

  libevent-2.0-5 libgssglue1 libnfsidmap2 libtirpc1 rpcbind

Use 'apt-get autoremove' to remove them.

The following extra packages will be installed:

  vio-cli

0 upgraded, 0 newly installed, 0 to remove and 168 not upgraded.

Reply
0 Kudos
ssurana
VMware Employee
VMware Employee

can you try running the command with

apt-get --reinstall install vio-cli=2.5.0.3955000

Reply
0 Kudos