Lost_Steak
Enthusiast
Enthusiast

Problem Upgrading from vRealize Operations 6.0.2 to 6.1

Hi,

I'm having trouble upgrading vROps from version 6.0.2 to 6.1. The 6.1 OS upgrade works fine but the 6.1 appliance upgrade fails at "Steps Completed" "4 of 9 - Preapply Validated"

I was unable to go from 6.0.1 VA to 6.1. However upgrading from 6.0.1 to 6.0.2 was successful (both OS and VA)

I can also confirm that there were no projects in place when doing the upgrade (from the release notes)

Upgrade does not complete

When upgrading an earlier vRealize Operations Manager deployment to this release, the cluster might become unresponsive with the System Status page continuously displaying Waiting for Analytics. The problem occurs if any of your version 6.0.x capacity projects have expired by having an implementation date that is now in the past.

Workaround: Before starting the upgrade, open the World > Projects page in version 6.0.x. When you display the page, vRealize Operations Manager updates the internal state of expired projects, and the unresponsive upgrade issue does not occur. If you have already started an upgrade and encountered this issue, contact VMware Technical Support for assistance.

Going through some log files in localhost:/var/log/vmware/vcops # I see

vcops_upgrade_20151020-142712.log

2015-10-20 14:36:01,477 - b2b - ERROR - updateCoordinator - programExit - Run post upgrade product scripts failed

vcops_upgrade_20151020-142712.log.1

2015-10-20 14:34:51,818 - INFO - No need to update service names on non-windows

2015-10-20 14:34:51,819 - INFO - directory "/usr/lib/vmware-vcops/suiteapi/extensions/suiteapi-telemetry-extension" does not exist.

2015-10-20 14:34:51,819 - ERROR - Completed stats post upgrade, exit code: 4"

2015-10-20 14:34:51,849 - b2b - DEBUG - updateCoordinator - runScript - exit code: 4

2015-10-20 14:34:51,849 - b2b - ERROR - updateCoordinator - runScript - Script command: "/usr/lib/vmware-vcopssuite/python/bin/python /storage/db/pakRepoLocal/vRealizeOperationsManagerEnterprise-6103038037/extracted/upgrade_scripts/VA/post/stats-post-upgrade.py" failed with exit code: 4

2015-10-20 14:34:51,850 - b2b - ERROR - updateCoordinator - run_upgrade_scripts - The post upgrade product script: "/storage/db/pakRepoLocal/vRealizeOperationsManagerEnterprise-6103038037/extracted/upgrade_scripts/VA/post/stats-post-upgrade.py" failed with exit code: 4

dbupgrade_20151020-143045.log

2015-10-20 14:34:28,490 - INFO - runScript:82 - stderr:

2015-10-20 14:34:28,490 - INFO - runScript:83 - exit code: 0

2015-10-20 14:34:28,490 - INFO - runScript:74 - Running command: /usr/java/default/bin/java -cp "/usr/lib/vmware-vcops/common/lib/persistence-1.0-SNAPSHOT.jar:/opt/vmware/vpostgres/current/lib/postgresql-9.3.jdbc4.jar:/usr/lib/vmware-vco

ps/common/lib/spring-aop-3.2.9.RELEASE.jar:/usr/lib/vmware-vcops/common/lib/spring-beans-3.2.9.RELEASE.jar:/usr/lib/vmware-vcops/common/lib/spring-context-3.2.9.RELEASE.jar:/usr/lib/vmware-vcops/common/lib/spring-core-3.2.9.RELEASE.jar:/

usr/lib/vmware-vcops/common/lib/*" -Dlog4j.configuration=file:/usr/lib/vmware-vcops/user/conf/persistence/log4j-sql-dbupgrade.properties com.vmware.statsplatform.persistence.util.sql.SqlDbUpgrade

2015-10-20 14:34:45,257 - INFO - runScript:81 - stdout:

2015-10-20 14:34:45,258 - INFO - runScript:82 - stderr: SLF4J: Class path contains multiple SLF4J bindings.

SLF4J: Found binding in [jar:file:/usr/lib/vmware-vcops/common/lib/slf4j-jdk14-1.7.2.jar!/org/slf4j/impl/StaticLoggerBinder.class]

SLF4J: Found binding in [jar:file:/usr/lib/vmware-vcops/common/lib/slf4j-log4j12-1.7.2.jar!/org/slf4j/impl/StaticLoggerBinder.class]

SLF4J: Found binding in [jar:file:/usr/lib/vmware-vcops/common/lib/vcops-suiteapi-client-1.0-all.jar!/org/slf4j/impl/StaticLoggerBinder.class]

SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.

SLF4J: Actual binding is of type [org.slf4j.impl.JDK14LoggerFactory]

Oct 20, 2015 2:34:28 PM org.springframework.context.support.AbstractApplicationContext prepareRefresh

INFO: Refreshing org.springframework.context.support.ClassPathXmlApplicationContext@8bd1b6a: startup date [Tue Oct 20 14:34:28 AEDT 2015]; root of context hierarchy

Oct 20, 2015 2:34:28 PM org.springframework.beans.factory.xml.XmlBeanDefinitionReader loadBeanDefinitions

INFO: Loading XML bean definitions from URL [file:/usr/lib/vmware-vcops/user/conf/persistence/spring-sql-dbupgrade.xml]

Oct 20, 2015 2:34:29 PM org.springframework.core.io.support.PropertiesLoaderSupport loadProperties

INFO: Loading properties file from URL [file:/usr/lib/vmware-vcops/user/conf/persistence/persistence.properties]

Oct 20, 2015 2:34:29 PM com.vmware.statsplatform.persistence.SpringPropertiesUtil resolvePlaceholder

INFO: postgres is enabled for central db, override db.driver to be just local until code cleanup

Oct 20, 2015 2:34:29 PM com.vmware.statsplatform.persistence.SpringPropertiesUtil resolvePlaceholder

INFO: return local driver:/data/vcops/xdb/vcops.bootstrap

Oct 20, 2015 2:34:29 PM org.springframework.beans.factory.support.DefaultListableBeanFactory preInstantiateSingletons

INFO: Pre-instantiating singletons in org.springframework.beans.factory.support.DefaultListableBeanFactory@4c12331b: defining beans [propertiesHolder,postgresDataSource,org.springframework.aop.config.internalAutoProxyCreator,org.springfr

amework.transaction.annotation.AnnotationTransactionAttributeSource#0,org.springframework.transaction.interceptor.TransactionInterceptor#0,org.springframework.transaction.config.internalTransactionAdvisor,txnManager,upgradeActionExecutor

Manager]; root of factory hierarchy

Oct 20, 2015 2:34:45 PM org.springframework.context.support.AbstractApplicationContext doClose

INFO: Closing org.springframework.context.support.ClassPathXmlApplicationContext@8bd1b6a: startup date [Tue Oct 20 14:34:28 AEDT 2015]; root of context hierarchy

Oct 20, 2015 2:34:45 PM org.springframework.beans.factory.support.DefaultSingletonBeanRegistry destroySingletons

INFO: Destroying singletons in org.springframework.beans.factory.support.DefaultListableBeanFactory@4c12331b: defining beans [propertiesHolder,postgresDataSource,org.springframework.aop.config.internalAutoProxyCreator,org.springframework

.transaction.annotation.AnnotationTransactionAttributeSource#0,org.springframework.transaction.interceptor.TransactionInterceptor#0,org.springframework.transaction.config.internalTransactionAdvisor,txnManager,upgradeActionExecutorManager

]; root of factory hierarchy

2015-10-20 14:34:45,258 - INFO - runScript:83 - exit code: 0

2015-10-20 14:34:45,259 - INFO - runScript:74 - Running command: /sbin/service vpostgres stop

2015-10-20 14:34:46,519 - INFO - runScript:81 - stdout: Stopping vRealize Operations vPostgres Database...

Stopped vRealize Operations vPostgres Database.

2015-10-20 14:34:46,519 - INFO - runScript:82 - stderr:

2015-10-20 14:34:46,519 - INFO - runScript:83 - exit code: 0

2015-10-20 14:34:46,521 - INFO - <module>:308 - Completed Sql DB Upgrade

2015-10-20 14:34:51,653 - ERROR - <module>:337 - One or more of the DB upgrades failed, exiting with exit code: 2, continue for now

stats-post-upgrade_20151020-142925.log

2015-10-20 14:34:10,678 - ERROR - Script command: "/usr/java/default/bin/java -cp "/storage/db/vcops/tmpoa082L/*:/usr/lib/vmware-vcops/controller/plugins/*" -Dlog4j.configuration=/usr/lib/vmware-vcops/user/conf/persistence/log4j.properties -DALIVE_BASE=/usr/lib/vmware-vcops com.vmware.statsplatform.persistence.util.DBUpgrade adminRoleEnabled,dataRoleEnabled,uiRoleEnabled" failed with exit code: 2

2015-10-20 14:34:10,679 - ERROR - Master DB Upgrade failed with exit code: 2, continue to run master postgres DB Upgrade

2015-10-20 14:34:10,679 - INFO - Running Master Postgres DB Upgrade

2015-10-20 14:34:10,679 - INFO - Running command: /usr/java/default/bin/java -cp "/storage/db/vcops/tmpoa082L/*" -Dlog4j.configuration=/usr/lib/vmware-vcops/user/conf/persistence/log4j.properties -DADAPTER_HOME=/usr/lib/vmware-vcops/user/plugins/inbound com.vmware.statsplatform.persistence.util.CentralSqlDbUpgrade.CentralSqlDbUpgrade adminRoleEnabled,dataRoleEnabled,uiRoleEnabled

2015-10-20 14:34:16,437 - INFO - stdout:

2015-10-20 14:34:16,438 - INFO - stderr: SLF4J: Class path contains multiple SLF4J bindings.

SLF4J: Found binding in [jar:file:/storage/db/vcops/tmpoa082L/slf4j-log4j12-1.7.2.jar!/org/slf4j/impl/StaticLoggerBinder.class]

SLF4J: Found binding in [jar:file:/storage/db/vcops/tmpoa082L/slf4j-jdk14.jar!/org/slf4j/impl/StaticLoggerBinder.class]

SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.

SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]

2015-10-20 14:34:16,438 - INFO - exit code: 0

2015-10-20 14:34:16,438 - INFO - Running Cassandra DB Upgrade

2015-10-20 14:34:16,438 - INFO - Running command: /usr/java/default/bin/java -cp "/storage/db/vcops/tmpoa082L/*:/usr/lib/vmware-vcops/common/lib/*" -Dlog4j.configuration=/usr/lib/vmware-vcops/user/conf/persistence/log4j.properties -DALIVE_BASE=/usr/lib/vmware-vcops com.vmware.vcops.dbupgrade.cassandra.CassandraDbUpgrade adminRoleEnabled,dataRoleEnabled,uiRoleEnabled

2015-10-20 14:34:26,116 - INFO - stdout:

2015-10-20 14:34:26,116 - INFO - stderr: SLF4J: Class path contains multiple SLF4J bindings.

SLF4J: Found binding in [jar:file:/storage/db/vcops/tmpoa082L/slf4j-log4j12-1.7.2.jar!/org/slf4j/impl/StaticLoggerBinder.class]

SLF4J: Found binding in [jar:file:/storage/db/vcops/tmpoa082L/slf4j-jdk14.jar!/org/slf4j/impl/StaticLoggerBinder.class]

SLF4J: Found binding in [jar:file:/usr/lib/vmware-vcops/common/lib/slf4j-jdk14-1.7.2.jar!/org/slf4j/impl/StaticLoggerBinder.class]

SLF4J: Found binding in [jar:file:/usr/lib/vmware-vcops/common/lib/slf4j-log4j12-1.7.2.jar!/org/slf4j/impl/StaticLoggerBinder.class]

SLF4J: Found binding in [jar:file:/usr/lib/vmware-vcops/common/lib/vcops-suiteapi-client-1.0-all.jar!/org/slf4j/impl/StaticLoggerBinder.class]

SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.

I have even gone so far as to verify the MD5 check sum on the files to make sure they downloaded correctly.

I have done plenty of Googleing and there doesn't seem to be a resolution to this problem. Any help is welcome.

Thanks

0 Kudos
5 Replies
mark_j
Virtuoso
Virtuoso

Call GSS and open an SR, as it looks like the db upgrade failed. This ca be due to a numbers of reasons.

If you find this or any other answer useful please mark the answer as correct or helpful.
0 Kudos
Lost_Steak
Enthusiast
Enthusiast

Still trying to resolve this before calling VMware support, but I don't think I am able to. Other things that I have tried

1. I can successfully upgrade to 6.0.3 but it still fails when taking the next step to 6.1 (again in the same spot "Steps Completed" "4 of 9 - Preapply Validated")


Additional information:

We have a vRealize Automation and a vRealize Business installation which I believe have links into vRealize Operations. I have a hunch that this has put new entries into the database which is causing the problem

Has anyone successfully upgraded vROPs 6.1 which is linked with vRealize Automation and vRealize Business?

0 Kudos
MichaelRyom
Hot Shot
Hot Shot

Hi Tom

Seen errors like that  - failing on db. You should call GSS, right away. You might just be wasting time...

There are know errors in regards to upgrading to 6.1 - GSS will be the best way to go.

Blogging at https://MichaelRyom.dk
0 Kudos
A13x
Hot Shot
Hot Shot

I am having the exact same error message and opened up a SR with VMware. Did you get this resolved in the end?

0 Kudos
Lost_Steak
Enthusiast
Enthusiast

No I didn't resolve the issue and I didn't open an SR. I just redeployed the appliance and threw away the old metrics. Opening a SR and then dealing with it is in the too hard basket and I didn't want to just cover up a problem only to have it come back again in the next upgrade.

The redeploy took under 2 hours to be up and running again and recreate my rules.

I have kept the old appliance in an powered off state and if historical stats are required, I'll power it on again (with precautions).

0 Kudos