VMware Cloud Community
niam
Enthusiast
Enthusiast

Upgrade from 5.8.0 to 5.8.1 failed!

Hello

I tried to upgrade vC Ops 5.8.0 to 5.8.1 in the admin web console. Everything went well. Upgrade process stucked with the status window "upgrade in progress - please wait". I was waiting for 4 hr....

Then I logged in the UI console and "vcops-admin update status" showed failed. The admin web console is not running anymore and status is stopped (vcops-admin status).

vcops140326.png

Any ideas?

Cheers

Nils

Reply
0 Kudos
16 Replies
gradinka
VMware Employee
VMware Employee

seems like something went wrong. I hope you do have backup/snapshots from before the upgrade!

few things to check:

- do you have enough freespace

- can you start the services manually

- what happens when you open AdminUI

- what's in "/var/log/vmware/updatecoordinator.log" and "/var/log/vmware/dbupgrade-ciq.log"

I'd suggest to open a support call with vmware for that issue ...

Reply
0 Kudos
niam
Enthusiast
Enthusiast

- do you have enough freespace

Yes, I checked that - enough space

- can you start the services manually

No, "Waiting for the service to come up" messages for about 2mins, service won't start

- what happens when you open AdminUI

Page is not available, wait until..... only one line

- what's in "/var/log/vmware/updatecoordinator.log" and "/var/log/vmware/dbupgrade-ciq.log"

Log is available.

I'd suggest to open a support call with vmware for that issue ...

Already done

Thanks

Reply
0 Kudos
gradinka
VMware Employee
VMware Employee

what is that "only one line"...

also, anything interesting in the logs? I know they're there, but what's inside them is important Smiley Happy

Reply
0 Kudos
niam
Enthusiast
Enthusiast

"The web server is not yet available. You will be redirected when service is available" is the message

updatecoordinator.log:

Wed Mar 26 09:36:15 CET 2014 [INFO] Installed product: VMware vCenter Operations Manager 5.8.0 Build 1448123

Wed Mar 26 09:36:15 CET 2014 [INFO] Upgrading to version: 5.8.1 build 1637685

Wed Mar 26 09:36:15 CET 2014 [INFO] Remove ssh timeout

Wed Mar 26 09:36:15 CET 2014 [INFO] Upgrade ssh ciphers

Wed Mar 26 09:36:28 CET 2014 [INFO] Determining the rpms to be installed

Wed Mar 26 09:36:29 CET 2014 [INFO] Test result for /data/upd-tmp/vcops-update.unpacked/VMware-vPostgres-9.0.13.0-1136341.x86_64.rpm:     package VMware-vPostgres-9.0.13.0-1136341.x86_64 is already installed

Wed Mar 26 09:36:29 CET 2014 [INFO] VMware-vPostgres-9.0.13.0-1136341.x86_64.rpm or a newer version of it is already installed and will be skipped

Wed Mar 26 09:36:29 CET 2014 [INFO] Test result for /data/upd-tmp/vcops-update.unpacked/VMware-vPostgres-client-9.0.13.0-1136341.x86_64.rpm:     package VMware-vPostgres-client-9.0.13.0-1136341.x86_64 is already installed

Wed Mar 26 09:36:29 CET 2014 [INFO] VMware-vPostgres-client-9.0.13.0-1136341.x86_64.rpm or a newer version of it is already installed and will be skipped

Wed Mar 26 09:36:29 CET 2014 [INFO] Test result for /data/upd-tmp/vcops-update.unpacked/apache2-mod_jk-1.2.26-1.30.110.x86_64.rpm:     package apache2-mod_jk-1.2.30-19.3.x86_64 (which is newer than apache2-mod_jk-1.2.26-1.30.110.x86_64) is already installed

Wed Mar 26 09:36:29 CET 2014 [INFO] apache2-mod_jk-1.2.26-1.30.110.x86_64.rpm or a newer version of it is already installed and will be skipped

Wed Mar 26 09:36:29 CET 2014 [INFO] Test result for /data/upd-tmp/vcops-update.unpacked/bind-libs-9.9.4P2-0.6.1.x86_64.rpm: error: Failed dependencies:

    libdns.so.99()(64bit) is needed by (installed) bind-utils-9.9.3P2-0.5.1.x86_64

Wed Mar 26 09:36:29 CET 2014 [INFO] bind-libs-9.9.4P2-0.6.1.x86_64.rpm has failed validation test. Will try to validate again...

Wed Mar 26 09:36:29 CET 2014 [INFO] Test result for /data/upd-tmp/vcops-update.unpacked/bind-utils-9.9.4P2-0.6.1.x86_64.rpm: error: Failed dependencies:

    libdns.so.100()(64bit) is needed by bind-utils-9.9.4P2-0.6.1.x86_64

Wed Mar 26 09:36:29 CET 2014 [INFO] bind-utils-9.9.4P2-0.6.1.x86_64.rpm has failed validation test. Will try to validate again...

Wed Mar 26 09:36:29 CET 2014 [INFO] Test result for /data/upd-tmp/vcops-update.unpacked/dhcpcd-3.2.3-44.30.1.x86_64.rpm:     package dhcpcd-3.2.3-44.30.1.x86_64 is already installed

Wed Mar 26 09:36:29 CET 2014 [INFO] dhcpcd-3.2.3-44.30.1.x86_64.rpm or a newer version of it is already installed and will be skipped

Wed Mar 26 09:36:29 CET 2014 [INFO] Test result for /data/upd-tmp/vcops-update.unpacked/eject-2.1.0-115.19.2.1.x86_64.rpm:     package eject-2.1.0-115.19.2.1.x86_64 is already installed

Wed Mar 26 09:36:29 CET 2014 [INFO] eject-2.1.0-115.19.2.1.x86_64.rpm or a newer version of it is already installed and will be skipped

Wed Mar 26 09:36:30 CET 2014 [INFO] Test result for /data/upd-tmp/vcops-update.unpacked/gettext-runtime-0.17-61.40.x86_64.rpm:     package gettext-runtime-0.17-61.40.x86_64 is already installed

Wed Mar 26 09:36:30 CET 2014 [INFO] gettext-runtime-0.17-61.40.x86_64.rpm or a newer version of it is already installed and will be skipped

Wed Mar 26 09:36:30 CET 2014 [INFO] Test result for /data/upd-tmp/vcops-update.unpacked/iproute2-2.6.29.1-6.22.48.x86_64.rpm:     package iproute2-2.6.29.1-6.22.48.x86_64 is already installed

Wed Mar 26 09:36:30 CET 2014 [INFO] iproute2-2.6.29.1-6.22.48.x86_64.rpm or a newer version of it is already installed and will be skipped

Wed Mar 26 09:36:30 CET 2014 [INFO] Test result for /data/upd-tmp/vcops-update.unpacked/iptables-1.4.6-2.11.4.x86_64.rpm:     package iptables-1.4.6-2.11.4.x86_64 is already installed

Wed Mar 26 09:36:30 CET 2014 [INFO] iptables-1.4.6-2.11.4.x86_64.rpm or a newer version of it is already installed and will be skipped

Wed Mar 26 09:36:30 CET 2014 [INFO] Test result for /data/upd-tmp/vcops-update.unpacked/jre-7u45-linux-x64.rpm:     package jre-1.7.0_45-fcs.x86_64 is already installed

Wed Mar 26 09:36:30 CET 2014 [INFO] jre-7u45-linux-x64.rpm or a newer version of it is already installed and will be skipped

Wed Mar 26 09:36:30 CET 2014 [INFO] Test result for /data/upd-tmp/vcops-update.unpacked/libapr-util1-1.3.4-12.22.21.2.x86_64.rpm:     package libapr-util1-1.3.4-12.22.21.2.x86_64 is already installed

Wed Mar 26 09:36:30 CET 2014 [INFO] libapr-util1-1.3.4-12.22.21.2.x86_64.rpm or a newer version of it is already installed and will be skipped

Wed Mar 26 09:36:30 CET 2014 [INFO] Test result for /data/upd-tmp/vcops-update.unpacked/libapr1-1.3.3-11.18.19.1.x86_64.rpm:     package libapr1-1.3.3-11.18.19.1.x86_64 is already installed

Wed Mar 26 09:36:30 CET 2014 [INFO] libapr1-1.3.3-11.18.19.1.x86_64.rpm or a newer version of it is already installed and will be skipped

Wed Mar 26 09:36:30 CET 2014 [INFO] Test result for /data/upd-tmp/vcops-update.unpacked/libopenssl0_9_8-0.9.8j-0.50.1.x86_64.rpm:     package libopenssl0_9_8-0.9.8j-0.50.1.x86_64 is already installed

Wed Mar 26 09:36:30 CET 2014 [INFO] libopenssl0_9_8-0.9.8j-0.50.1.x86_64.rpm or a newer version of it is already installed and will be skipped

Wed Mar 26 09:36:30 CET 2014 [INFO] Test result for /data/upd-tmp/vcops-update.unpacked/libopenssl0_9_8-32bit-0.9.8j-0.50.1.x86_64.rpm:     package libopenssl0_9_8-32bit-0.9.8j-0.50.1.x86_64 is already installed

Wed Mar 26 09:36:30 CET 2014 [INFO] libopenssl0_9_8-32bit-0.9.8j-0.50.1.x86_64.rpm or a newer version of it is already installed and will be skipped

Wed Mar 26 09:36:31 CET 2014 [INFO] Test result for /data/upd-tmp/vcops-update.unpacked/net-tools-1.60-725.32.2.1.x86_64.rpm:     package net-tools-1.60-725.32.2.1.x86_64 is already installed

Wed Mar 26 09:36:31 CET 2014 [INFO] net-tools-1.60-725.32.2.1.x86_64.rpm or a newer version of it is already installed and will be skipped

Wed Mar 26 09:36:31 CET 2014 [INFO] Test result for /data/upd-tmp/vcops-update.unpacked/ntp-4.2.4p8-1.20.1.x86_64.rpm:     package ntp-4.2.4p8-1.20.1.x86_64 is already installed

Wed Mar 26 09:36:31 CET 2014 [INFO] ntp-4.2.4p8-1.20.1.x86_64.rpm or a newer version of it is already installed and will be skipped

Wed Mar 26 09:36:31 CET 2014 [INFO] Test result for /data/upd-tmp/vcops-update.unpacked/openssh-5.1p1-41.57.1.x86_64.rpm:     package openssh-5.1p1-41.57.1.x86_64 is already installed

Wed Mar 26 09:36:31 CET 2014 [INFO] openssh-5.1p1-41.57.1.x86_64.rpm or a newer version of it is already installed and will be skipped

Wed Mar 26 09:36:31 CET 2014 [INFO] Test result for /data/upd-tmp/vcops-update.unpacked/openssl-0.9.8j-0.50.1.x86_64.rpm:     package openssl-0.9.8j-0.50.1.x86_64 is already installed

Wed Mar 26 09:36:31 CET 2014 [INFO] openssl-0.9.8j-0.50.1.x86_64.rpm or a newer version of it is already installed and will be skipped

Wed Mar 26 09:36:31 CET 2014 [INFO] Test result for /data/upd-tmp/vcops-update.unpacked/perl-Config-Text-1.73-1.x86_64.rpm:     package perl-Config-Text-1.73-1.x86_64 is already installed

Wed Mar 26 09:36:31 CET 2014 [INFO] perl-Config-Text-1.73-1.x86_64.rpm or a newer version of it is already installed and will be skipped

Wed Mar 26 09:36:31 CET 2014 [INFO] Test result for /data/upd-tmp/vcops-update.unpacked/sudo-1.7.6p2-0.2.12.1.x86_64.rpm:     package sudo-1.7.6p2-0.2.12.1.x86_64 is already installed

Wed Mar 26 09:36:31 CET 2014 [INFO] sudo-1.7.6p2-0.2.12.1.x86_64.rpm or a newer version of it is already installed and will be skipped

Wed Mar 26 09:36:31 CET 2014 [INFO] Test result for /data/upd-tmp/vcops-update.unpacked/syslog-ng-2.0.9-27.34.36.1.x86_64.rpm:     package syslog-ng-2.0.9-27.34.36.1.x86_64 is already installed

Wed Mar 26 09:36:31 CET 2014 [INFO] syslog-ng-2.0.9-27.34.36.1.x86_64.rpm or a newer version of it is already installed and will be skipped

Wed Mar 26 09:36:40 CET 2014 [INFO] DB rpms to install:

Wed Mar 26 09:36:40 CET 2014 [INFO] Admin rpms to install:  vmware-vcops-admin-cli-5.8.1-1637685.noarch.rpm vmware-vcops-admin-web-5.8.1-1637685.noarch.rpm

Wed Mar 26 09:36:40 CET 2014 [INFO] Non-admin rpms to install:  apache2-2.2.12-1.42.1.x86_64.rpm apache2-prefork-2.2.12-1.42.1.x86_64.rpm apache2-utils-2.2.12-1.42.1.x86_64.rpm krb5-1.6.3-133.49.58.1.x86_64.rpm libxml2-2.7.6-0.25.1.x86_64.rpm libxml2-python-2.7.6-0.25.1.x86_64.rpm sysstat-8.1.5-7.38.40.1.x86_64.rpm vfabric-tc-server-standard-2.9.4-RELEASE.noarch.rpm vmware-vcops-5.8.1-1637685.x86_64.rpm vmware-vcops-VCM-adapter-1.0-1472552.x86_64.rpm vmware-vcops-VINAdapter3-adapter-1.0-1484379.x86_64.rpm vmware-vcops-VMWVIwareAdapter3-adapter-2.0-1625602.x86_64.rpm bind-libs-9.9.4P2-0.6.1.x86_64.rpm bind-utils-9.9.4P2-0.6.1.x86_64.rpm

Wed Mar 26 09:36:40 CET 2014 [INFO] Copying directory /data/upd-tmp/vcops-update.unpacked to remote VA at /data/upd-tmp

Wed Mar 26 09:37:10 CET 2014 [INFO] Stopping service admin

Stopping vcopsAdminService...

Stopped vcopsAdminService.

Wed Mar 26 09:37:15 CET 2014 [INFO] Testing rpm(s)  vmware-vcops-admin-cli-5.8.1-1637685.noarch.rpm vmware-vcops-admin-web-5.8.1-1637685.noarch.rpm

/data/upd-tmp/vcops-update.unpacked /usr/lib/vmware-vcops/common/bin

/usr/lib/vmware-vcops/common/bin

Wed Mar 26 09:37:15 CET 2014 [INFO] Upgrading rpm(s)  vmware-vcops-admin-cli-5.8.1-1637685.noarch.rpm vmware-vcops-admin-web-5.8.1-1637685.noarch.rpm

/data/upd-tmp/vcops-update.unpacked /usr/lib/vmware-vcops/common/bin

Saving configuration files...

2 blocks

Backing up new unmodified configuration files...

Restoring configuration files...

/tmp/vcops-admin-web-cfg-save /usr/lib/tomcat-ciqadmin

/usr/lib/tomcat-ciqadmin

/usr/lib/vmware-vcops/common/bin

Wed Mar 26 09:37:19 CET 2014 [INFO] Upgrading rpm(s) on Analytics VM

Saving configuration files...

2 blocks

Backing up new unmodified configuration files...

Restoring configuration files...

/tmp/vcops-admin-web-cfg-save /usr/lib/tomcat-ciqadmin

/usr/lib/tomcat-ciqadmin

Wed Mar 26 09:37:27 CET 2014 [INFO] Stopping service web

Stopping vcopsService...

Waiting for vcopsService to exit...

Waiting for vcopsService to exit...

Waiting for vcopsService to exit...

Waiting for vcopsService to exit...

Stopped vcopsService.

Restarting VMware vPostgres: ok

Stopping vcopsEnterpriseService...

Stopped vcopsEnterpriseService.

Stopping VCOps:

Old Password: Your password has expired. Choose a new password.

/bin/su: incorrect password

Old Password: Your password has expired. Choose a new password.

/bin/su: incorrect password

Old Password: Your password has expired. Choose a new password.

/bin/su: incorrect password

failed.

Failed to disable vcops-watchdog for the vcops service.

Wed Mar 26 09:38:19 CET 2014 [ERROR] Command returned 1

Wed Mar 26 09:38:19 CET 2014 [ERROR] Stopping core services failed

Wed Mar 26 09:38:19 CET 2014 [INFO] Cleaning up

Wed Mar 26 09:38:28 CET 2014 [INFO] Starting service all

Starting vcopsAdminService...

Waiting for the service to come up...

Waiting for the service to come up...

Waiting for the service to come up...

Waiting for the service to come up...

Waiting for the service to come up...

Waiting for the service to come up...

Waiting for the service to come up...

Waiting for the service to come up...

Waiting for the service to come up...

Waiting for the service to come up...

Waiting for the service to come up...

Waiting for the service to come up...

Waiting for the service to come up...

Waiting for the service to come up...

Waiting for the service to come up...

Waiting for the service to come up...

Waiting for the service to come up...

Waiting for the service to come up...

Waiting for the service to come up...

Waiting for the service to come up...

Starting VCOps:

Old Password: Your password has expired. Choose a new password.

su: incorrect password

Old Password: Your password has expired. Choose a new password.

su: incorrect password

Old Password: Your password has expired. Choose a new password.

su: incorrect password

Old Password: Your password has expired. Choose a new password.

su: incorrect password

Old Password: Your password has expired. Choose a new password.

su: incorrect password

Old Password: Your password has expired. Choose a new password.

su: incorrect password

Old Password: Your password has expired. Choose a new password.

/bin/su: incorrect password

Old Password: Your password has expired. Choose a new password.

/bin/su: incorrect password

Old Password: Your password has expired. Choose a new password.

/bin/su: incorrect password

vcopsService is not running.

Starting vcopsService...

Waiting for the service to come up...

Waiting for the service to come up...

vcopsEnterpriseService is not running.

Starting vcopsEnterpriseService...

Waiting for the service to come up...

Waiting for the service to come up...

Wed Mar 26 09:47:44 CET 2014 [ERROR] Command returned 1

Wed Mar 26 09:47:44 CET 2014 [ERROR] Starting core services failed

Reply
0 Kudos
gradinka
VMware Employee
VMware Employee

OK

can you please change your root and admin passwords?

the admin one is expired obviously.

Reply
0 Kudos
niam
Enthusiast
Enthusiast

Like described in KB2013358?

Reply
0 Kudos
gradinka
VMware Employee
VMware Employee

yes, try with admin password first. the root one is most likely ok as you can login and there is no error (right?)

Reply
0 Kudos
niam
Enthusiast
Enthusiast

and then should I start the admin web service again and redo the upgrade?

Reply
0 Kudos
gradinka
VMware Employee
VMware Employee

try starting services. I think the upgrade went fine, but can't really tell.

the safest thing to do would be to

1) revert to the old system (backup snapshot or whatever you did)

2) update your passwords

3) upgrade again

Reply
0 Kudos
niam
Enthusiast
Enthusiast

can't reset admin pw:

#./resetadminpwd.sh vmWA4vcop12atBE54!

spawn su - root -c passwd.admin --stin

vmWA4vcop12atBE54!

Changing password for admin.

-bash: psql: command not found

The password was not successfully reset.

#

Reply
0 Kudos
gradinka
VMware Employee
VMware Employee

you have used the file from here "/usr/lib/vmware-vcops/user/conf/install" right?

just typing "psql" works or not?

Get the vmware support to login and fix it for you... Seems like some linux wizardry is required to fix your environment issue(s) and make it work...

Reply
0 Kudos
niam
Enthusiast
Enthusiast

I modified the script with the right path to psql - success.

Restarted the services

vCOps Admin Web Service still state STOPPING

Reply
0 Kudos
gradinka
VMware Employee
VMware Employee

maybe just restart the vApp now

Reply
0 Kudos
niam
Enthusiast
Enthusiast

I restarted the vApp with no luck. Admin page shows "The web server is not yet available. You will be redirected when service is available"

I'll restore vApp and will redo the upgrade with changed admin and root pw.

Thanks!

Reply
0 Kudos
gradinka
VMware Employee
VMware Employee

OK!

keep us updated

Reply
0 Kudos
pratjain
VMware Employee
VMware Employee

I would suggest you to file a SR with VMware Support for further investigation if your upgrade still fails post changing the passwords    

Regards, PJ If you find this or any other answer useful please mark the answer as correct or helpful.
Reply
0 Kudos