Hello
I tried to upgrade vC Ops 5.8.0 to 5.8.1 in the admin web console. Everything went well. Upgrade process stucked with the status window "upgrade in progress - please wait". I was waiting for 4 hr....
Then I logged in the UI console and "vcops-admin update status" showed failed. The admin web console is not running anymore and status is stopped (vcops-admin status).
Any ideas?
Cheers
Nils
seems like something went wrong. I hope you do have backup/snapshots from before the upgrade!
few things to check:
- do you have enough freespace
- can you start the services manually
- what happens when you open AdminUI
- what's in "/var/log/vmware/updatecoordinator.log" and "/var/log/vmware/dbupgrade-ciq.log"
I'd suggest to open a support call with vmware for that issue ...
- do you have enough freespace
Yes, I checked that - enough space
- can you start the services manually
No, "Waiting for the service to come up" messages for about 2mins, service won't start
- what happens when you open AdminUI
Page is not available, wait until..... only one line
- what's in "/var/log/vmware/updatecoordinator.log" and "/var/log/vmware/dbupgrade-ciq.log"
Log is available.
I'd suggest to open a support call with vmware for that issue ...
Already done
Thanks
what is that "only one line"...
also, anything interesting in the logs? I know they're there, but what's inside them is important
"The web server is not yet available. You will be redirected when service is available" is the message
updatecoordinator.log:
Wed Mar 26 09:36:15 CET 2014 [INFO] Installed product: VMware vCenter Operations Manager 5.8.0 Build 1448123
Wed Mar 26 09:36:15 CET 2014 [INFO] Upgrading to version: 5.8.1 build 1637685
Wed Mar 26 09:36:15 CET 2014 [INFO] Remove ssh timeout
Wed Mar 26 09:36:15 CET 2014 [INFO] Upgrade ssh ciphers
Wed Mar 26 09:36:28 CET 2014 [INFO] Determining the rpms to be installed
Wed Mar 26 09:36:29 CET 2014 [INFO] Test result for /data/upd-tmp/vcops-update.unpacked/VMware-vPostgres-9.0.13.0-1136341.x86_64.rpm: package VMware-vPostgres-9.0.13.0-1136341.x86_64 is already installed
Wed Mar 26 09:36:29 CET 2014 [INFO] VMware-vPostgres-9.0.13.0-1136341.x86_64.rpm or a newer version of it is already installed and will be skipped
Wed Mar 26 09:36:29 CET 2014 [INFO] Test result for /data/upd-tmp/vcops-update.unpacked/VMware-vPostgres-client-9.0.13.0-1136341.x86_64.rpm: package VMware-vPostgres-client-9.0.13.0-1136341.x86_64 is already installed
Wed Mar 26 09:36:29 CET 2014 [INFO] VMware-vPostgres-client-9.0.13.0-1136341.x86_64.rpm or a newer version of it is already installed and will be skipped
Wed Mar 26 09:36:29 CET 2014 [INFO] Test result for /data/upd-tmp/vcops-update.unpacked/apache2-mod_jk-1.2.26-1.30.110.x86_64.rpm: package apache2-mod_jk-1.2.30-19.3.x86_64 (which is newer than apache2-mod_jk-1.2.26-1.30.110.x86_64) is already installed
Wed Mar 26 09:36:29 CET 2014 [INFO] apache2-mod_jk-1.2.26-1.30.110.x86_64.rpm or a newer version of it is already installed and will be skipped
Wed Mar 26 09:36:29 CET 2014 [INFO] Test result for /data/upd-tmp/vcops-update.unpacked/bind-libs-9.9.4P2-0.6.1.x86_64.rpm: error: Failed dependencies:
libdns.so.99()(64bit) is needed by (installed) bind-utils-9.9.3P2-0.5.1.x86_64
Wed Mar 26 09:36:29 CET 2014 [INFO] bind-libs-9.9.4P2-0.6.1.x86_64.rpm has failed validation test. Will try to validate again...
Wed Mar 26 09:36:29 CET 2014 [INFO] Test result for /data/upd-tmp/vcops-update.unpacked/bind-utils-9.9.4P2-0.6.1.x86_64.rpm: error: Failed dependencies:
libdns.so.100()(64bit) is needed by bind-utils-9.9.4P2-0.6.1.x86_64
Wed Mar 26 09:36:29 CET 2014 [INFO] bind-utils-9.9.4P2-0.6.1.x86_64.rpm has failed validation test. Will try to validate again...
Wed Mar 26 09:36:29 CET 2014 [INFO] Test result for /data/upd-tmp/vcops-update.unpacked/dhcpcd-3.2.3-44.30.1.x86_64.rpm: package dhcpcd-3.2.3-44.30.1.x86_64 is already installed
Wed Mar 26 09:36:29 CET 2014 [INFO] dhcpcd-3.2.3-44.30.1.x86_64.rpm or a newer version of it is already installed and will be skipped
Wed Mar 26 09:36:29 CET 2014 [INFO] Test result for /data/upd-tmp/vcops-update.unpacked/eject-2.1.0-115.19.2.1.x86_64.rpm: package eject-2.1.0-115.19.2.1.x86_64 is already installed
Wed Mar 26 09:36:29 CET 2014 [INFO] eject-2.1.0-115.19.2.1.x86_64.rpm or a newer version of it is already installed and will be skipped
Wed Mar 26 09:36:30 CET 2014 [INFO] Test result for /data/upd-tmp/vcops-update.unpacked/gettext-runtime-0.17-61.40.x86_64.rpm: package gettext-runtime-0.17-61.40.x86_64 is already installed
Wed Mar 26 09:36:30 CET 2014 [INFO] gettext-runtime-0.17-61.40.x86_64.rpm or a newer version of it is already installed and will be skipped
Wed Mar 26 09:36:30 CET 2014 [INFO] Test result for /data/upd-tmp/vcops-update.unpacked/iproute2-2.6.29.1-6.22.48.x86_64.rpm: package iproute2-2.6.29.1-6.22.48.x86_64 is already installed
Wed Mar 26 09:36:30 CET 2014 [INFO] iproute2-2.6.29.1-6.22.48.x86_64.rpm or a newer version of it is already installed and will be skipped
Wed Mar 26 09:36:30 CET 2014 [INFO] Test result for /data/upd-tmp/vcops-update.unpacked/iptables-1.4.6-2.11.4.x86_64.rpm: package iptables-1.4.6-2.11.4.x86_64 is already installed
Wed Mar 26 09:36:30 CET 2014 [INFO] iptables-1.4.6-2.11.4.x86_64.rpm or a newer version of it is already installed and will be skipped
Wed Mar 26 09:36:30 CET 2014 [INFO] Test result for /data/upd-tmp/vcops-update.unpacked/jre-7u45-linux-x64.rpm: package jre-1.7.0_45-fcs.x86_64 is already installed
Wed Mar 26 09:36:30 CET 2014 [INFO] jre-7u45-linux-x64.rpm or a newer version of it is already installed and will be skipped
Wed Mar 26 09:36:30 CET 2014 [INFO] Test result for /data/upd-tmp/vcops-update.unpacked/libapr-util1-1.3.4-12.22.21.2.x86_64.rpm: package libapr-util1-1.3.4-12.22.21.2.x86_64 is already installed
Wed Mar 26 09:36:30 CET 2014 [INFO] libapr-util1-1.3.4-12.22.21.2.x86_64.rpm or a newer version of it is already installed and will be skipped
Wed Mar 26 09:36:30 CET 2014 [INFO] Test result for /data/upd-tmp/vcops-update.unpacked/libapr1-1.3.3-11.18.19.1.x86_64.rpm: package libapr1-1.3.3-11.18.19.1.x86_64 is already installed
Wed Mar 26 09:36:30 CET 2014 [INFO] libapr1-1.3.3-11.18.19.1.x86_64.rpm or a newer version of it is already installed and will be skipped
Wed Mar 26 09:36:30 CET 2014 [INFO] Test result for /data/upd-tmp/vcops-update.unpacked/libopenssl0_9_8-0.9.8j-0.50.1.x86_64.rpm: package libopenssl0_9_8-0.9.8j-0.50.1.x86_64 is already installed
Wed Mar 26 09:36:30 CET 2014 [INFO] libopenssl0_9_8-0.9.8j-0.50.1.x86_64.rpm or a newer version of it is already installed and will be skipped
Wed Mar 26 09:36:30 CET 2014 [INFO] Test result for /data/upd-tmp/vcops-update.unpacked/libopenssl0_9_8-32bit-0.9.8j-0.50.1.x86_64.rpm: package libopenssl0_9_8-32bit-0.9.8j-0.50.1.x86_64 is already installed
Wed Mar 26 09:36:30 CET 2014 [INFO] libopenssl0_9_8-32bit-0.9.8j-0.50.1.x86_64.rpm or a newer version of it is already installed and will be skipped
Wed Mar 26 09:36:31 CET 2014 [INFO] Test result for /data/upd-tmp/vcops-update.unpacked/net-tools-1.60-725.32.2.1.x86_64.rpm: package net-tools-1.60-725.32.2.1.x86_64 is already installed
Wed Mar 26 09:36:31 CET 2014 [INFO] net-tools-1.60-725.32.2.1.x86_64.rpm or a newer version of it is already installed and will be skipped
Wed Mar 26 09:36:31 CET 2014 [INFO] Test result for /data/upd-tmp/vcops-update.unpacked/ntp-4.2.4p8-1.20.1.x86_64.rpm: package ntp-4.2.4p8-1.20.1.x86_64 is already installed
Wed Mar 26 09:36:31 CET 2014 [INFO] ntp-4.2.4p8-1.20.1.x86_64.rpm or a newer version of it is already installed and will be skipped
Wed Mar 26 09:36:31 CET 2014 [INFO] Test result for /data/upd-tmp/vcops-update.unpacked/openssh-5.1p1-41.57.1.x86_64.rpm: package openssh-5.1p1-41.57.1.x86_64 is already installed
Wed Mar 26 09:36:31 CET 2014 [INFO] openssh-5.1p1-41.57.1.x86_64.rpm or a newer version of it is already installed and will be skipped
Wed Mar 26 09:36:31 CET 2014 [INFO] Test result for /data/upd-tmp/vcops-update.unpacked/openssl-0.9.8j-0.50.1.x86_64.rpm: package openssl-0.9.8j-0.50.1.x86_64 is already installed
Wed Mar 26 09:36:31 CET 2014 [INFO] openssl-0.9.8j-0.50.1.x86_64.rpm or a newer version of it is already installed and will be skipped
Wed Mar 26 09:36:31 CET 2014 [INFO] Test result for /data/upd-tmp/vcops-update.unpacked/perl-Config-Text-1.73-1.x86_64.rpm: package perl-Config-Text-1.73-1.x86_64 is already installed
Wed Mar 26 09:36:31 CET 2014 [INFO] perl-Config-Text-1.73-1.x86_64.rpm or a newer version of it is already installed and will be skipped
Wed Mar 26 09:36:31 CET 2014 [INFO] Test result for /data/upd-tmp/vcops-update.unpacked/sudo-1.7.6p2-0.2.12.1.x86_64.rpm: package sudo-1.7.6p2-0.2.12.1.x86_64 is already installed
Wed Mar 26 09:36:31 CET 2014 [INFO] sudo-1.7.6p2-0.2.12.1.x86_64.rpm or a newer version of it is already installed and will be skipped
Wed Mar 26 09:36:31 CET 2014 [INFO] Test result for /data/upd-tmp/vcops-update.unpacked/syslog-ng-2.0.9-27.34.36.1.x86_64.rpm: package syslog-ng-2.0.9-27.34.36.1.x86_64 is already installed
Wed Mar 26 09:36:31 CET 2014 [INFO] syslog-ng-2.0.9-27.34.36.1.x86_64.rpm or a newer version of it is already installed and will be skipped
Wed Mar 26 09:36:40 CET 2014 [INFO] DB rpms to install:
Wed Mar 26 09:36:40 CET 2014 [INFO] Admin rpms to install: vmware-vcops-admin-cli-5.8.1-1637685.noarch.rpm vmware-vcops-admin-web-5.8.1-1637685.noarch.rpm
Wed Mar 26 09:36:40 CET 2014 [INFO] Non-admin rpms to install: apache2-2.2.12-1.42.1.x86_64.rpm apache2-prefork-2.2.12-1.42.1.x86_64.rpm apache2-utils-2.2.12-1.42.1.x86_64.rpm krb5-1.6.3-133.49.58.1.x86_64.rpm libxml2-2.7.6-0.25.1.x86_64.rpm libxml2-python-2.7.6-0.25.1.x86_64.rpm sysstat-8.1.5-7.38.40.1.x86_64.rpm vfabric-tc-server-standard-2.9.4-RELEASE.noarch.rpm vmware-vcops-5.8.1-1637685.x86_64.rpm vmware-vcops-VCM-adapter-1.0-1472552.x86_64.rpm vmware-vcops-VINAdapter3-adapter-1.0-1484379.x86_64.rpm vmware-vcops-VMWVIwareAdapter3-adapter-2.0-1625602.x86_64.rpm bind-libs-9.9.4P2-0.6.1.x86_64.rpm bind-utils-9.9.4P2-0.6.1.x86_64.rpm
Wed Mar 26 09:36:40 CET 2014 [INFO] Copying directory /data/upd-tmp/vcops-update.unpacked to remote VA at /data/upd-tmp
Wed Mar 26 09:37:10 CET 2014 [INFO] Stopping service admin
Stopping vcopsAdminService...
Stopped vcopsAdminService.
Wed Mar 26 09:37:15 CET 2014 [INFO] Testing rpm(s) vmware-vcops-admin-cli-5.8.1-1637685.noarch.rpm vmware-vcops-admin-web-5.8.1-1637685.noarch.rpm
/data/upd-tmp/vcops-update.unpacked /usr/lib/vmware-vcops/common/bin
/usr/lib/vmware-vcops/common/bin
Wed Mar 26 09:37:15 CET 2014 [INFO] Upgrading rpm(s) vmware-vcops-admin-cli-5.8.1-1637685.noarch.rpm vmware-vcops-admin-web-5.8.1-1637685.noarch.rpm
/data/upd-tmp/vcops-update.unpacked /usr/lib/vmware-vcops/common/bin
Saving configuration files...
2 blocks
Backing up new unmodified configuration files...
Restoring configuration files...
/tmp/vcops-admin-web-cfg-save /usr/lib/tomcat-ciqadmin
/usr/lib/tomcat-ciqadmin
/usr/lib/vmware-vcops/common/bin
Wed Mar 26 09:37:19 CET 2014 [INFO] Upgrading rpm(s) on Analytics VM
Saving configuration files...
2 blocks
Backing up new unmodified configuration files...
Restoring configuration files...
/tmp/vcops-admin-web-cfg-save /usr/lib/tomcat-ciqadmin
/usr/lib/tomcat-ciqadmin
Wed Mar 26 09:37:27 CET 2014 [INFO] Stopping service web
Stopping vcopsService...
Waiting for vcopsService to exit...
Waiting for vcopsService to exit...
Waiting for vcopsService to exit...
Waiting for vcopsService to exit...
Stopped vcopsService.
Restarting VMware vPostgres: ok
Stopping vcopsEnterpriseService...
Stopped vcopsEnterpriseService.
Stopping VCOps:
Old Password: Your password has expired. Choose a new password.
/bin/su: incorrect password
Old Password: Your password has expired. Choose a new password.
/bin/su: incorrect password
Old Password: Your password has expired. Choose a new password.
/bin/su: incorrect password
failed.
Failed to disable vcops-watchdog for the vcops service.
Wed Mar 26 09:38:19 CET 2014 [ERROR] Command returned 1
Wed Mar 26 09:38:19 CET 2014 [ERROR] Stopping core services failed
Wed Mar 26 09:38:19 CET 2014 [INFO] Cleaning up
Wed Mar 26 09:38:28 CET 2014 [INFO] Starting service all
Starting vcopsAdminService...
Waiting for the service to come up...
Waiting for the service to come up...
Waiting for the service to come up...
Waiting for the service to come up...
Waiting for the service to come up...
Waiting for the service to come up...
Waiting for the service to come up...
Waiting for the service to come up...
Waiting for the service to come up...
Waiting for the service to come up...
Waiting for the service to come up...
Waiting for the service to come up...
Waiting for the service to come up...
Waiting for the service to come up...
Waiting for the service to come up...
Waiting for the service to come up...
Waiting for the service to come up...
Waiting for the service to come up...
Waiting for the service to come up...
Waiting for the service to come up...
Starting VCOps:
Old Password: Your password has expired. Choose a new password.
su: incorrect password
Old Password: Your password has expired. Choose a new password.
su: incorrect password
Old Password: Your password has expired. Choose a new password.
su: incorrect password
Old Password: Your password has expired. Choose a new password.
su: incorrect password
Old Password: Your password has expired. Choose a new password.
su: incorrect password
Old Password: Your password has expired. Choose a new password.
su: incorrect password
Old Password: Your password has expired. Choose a new password.
/bin/su: incorrect password
Old Password: Your password has expired. Choose a new password.
/bin/su: incorrect password
Old Password: Your password has expired. Choose a new password.
/bin/su: incorrect password
vcopsService is not running.
Starting vcopsService...
Waiting for the service to come up...
Waiting for the service to come up...
vcopsEnterpriseService is not running.
Starting vcopsEnterpriseService...
Waiting for the service to come up...
Waiting for the service to come up...
Wed Mar 26 09:47:44 CET 2014 [ERROR] Command returned 1
Wed Mar 26 09:47:44 CET 2014 [ERROR] Starting core services failed
OK
can you please change your root and admin passwords?
the admin one is expired obviously.
Like described in KB2013358?
yes, try with admin password first. the root one is most likely ok as you can login and there is no error (right?)
and then should I start the admin web service again and redo the upgrade?
try starting services. I think the upgrade went fine, but can't really tell.
the safest thing to do would be to
1) revert to the old system (backup snapshot or whatever you did)
2) update your passwords
3) upgrade again
can't reset admin pw:
#./resetadminpwd.sh vmWA4vcop12atBE54!
spawn su - root -c passwd.admin --stin
vmWA4vcop12atBE54!
Changing password for admin.
-bash: psql: command not found
The password was not successfully reset.
#
you have used the file from here "/usr/lib/vmware-vcops/user/conf/install" right?
just typing "psql" works or not?
Get the vmware support to login and fix it for you... Seems like some linux wizardry is required to fix your environment issue(s) and make it work...
I modified the script with the right path to psql - success.
Restarted the services
vCOps Admin Web Service still state STOPPING
maybe just restart the vApp now
I restarted the vApp with no luck. Admin page shows "The web server is not yet available. You will be redirected when service is available"
I'll restore vApp and will redo the upgrade with changed admin and root pw.
Thanks!
OK!
keep us updated
I would suggest you to file a SR with VMware Support for further investigation if your upgrade still fails post changing the passwords