Upgrade vRealize Automation 7.3.0 to 7.4 failed

[anba89]

Hi all

I tried to upgrade vRA from 7.3.0 to 7.4 but got the following error:

Last Install:

Failed to install updates(Error while running post-install scripts) on Tuesday, 2018 April 17 23:09:58 UTC+2 VA-check: finished Pre-install: finished Post-install: failed Update failed (code 0-2). Check logs in /opt/vmware/var/log/vami or retry update later.

In the /opt/vmware/var/log/vami/updatecli.log I found this:

NOTICE:  Found extension citext and it exists in pg_catalog

Can not read the Orchestrator pid from /var/lib/vco/app-server/logs/tomcat.pid

+ res=2

+ echo 'Script /etc/bootstrap/postupdate.d/20-vcac failed, error status 2'

+ exit 2

+ rm -f /tmp/postupdate-err-log

+ exit 2

+ trapfunc

+ excode=2

+ test 2 -gt 0

+ vami_update_msg set post-install 'Post-install: failed'

+ test -x /usr/sbin/vami-update-msg

+ /usr/sbin/vami-update-msg set post-install 'Post-install: failed'

+ sleep 1

+ test 2 -gt 0 -o 0 -gt 0

+ vami_update_msg set update-status 'Update failed (code 0-2). Check logs in /opt/vmware/var/log/vami or retry update later.'

+ test -x /usr/sbin/vami-update-msg

+ /usr/sbin/vami-update-msg set update-status 'Update failed (code 0-2). Check logs in /opt/vmware/var/log/vami or retry update later.'

+ exit

17/04/2018 19:09:55 [ERROR] Failed with exit code 512

17/04/2018 19:09:55 [INFO] Update status: Running VMware tools reconfiguration

17/04/2018 19:09:55 [INFO] Running /opt/vmware/share/vami/vami_reconfigure_tools

vmware-toolbox-cmd is /usr/bin/vmware-toolbox-cmd

Configuring VAMI VMware tools service wrapper.

17/04/2018 19:09:55 [INFO] Update status: Done VMware tools reconfiguration

17/04/2018 19:09:55 [INFO] Update status: Error while running post-install scripts

17/04/2018 19:09:55 [ERROR] Failure: updatecli exiting abnormally

17/04/2018 19:09:55 [INFO] Install Finished

Any idea what's wrong?

Thanks,

Andi

[gradinka]

browse logs under /var/log/bootstrap (esp. "postupdate.log") for more details ... post here.

[anba89]

The postupdate logs:

/var/log/bootstrap # tail -n 100 postupdate.log

SSLOPTS: ?sslmode=verify-ca&sslrootcert=/var/vmware/vpostgres/current/.postgresql/root.crt

Orchestrator's root folder: /var/lib/vco

Orchestrator Configuration Tool. Version: 7.4.0.8071781 Build: 8071781

Can not read the Orchestrator pid from /var/lib/vco/app-server/logs/tomcat.pid

Start configuring database settings.

Configuring database settings finished successfully.

Start updating database scheme.

ch.dunes.vso.db.check.CheckDuplicatesException:

Duplicate entries found in the Orchestrator database:

Resource element duplicates:

        1 item with ID '8a8115675c67e982015c67eb4a4403e1' and name 'zh_MO.properties'

        1 item with ID '8a8115675c67e982015c67eb4a4403e1' and name 'ja_JP.properties'

        1 item with ID '8a8115675c67e982015c67eb4a4403e1' and name 'en.properties'

        1 item with ID '8a8115675c67e982015c67eb4a4403e1' and name 'ja.properties'

        1 item with ID '8a8115675c67e982015c67eb4a4403e1' and name 'es_ES.properties'

        1 item with ID '8a8115675c67e982015c67eb4a4403e1' and name 'zh_CN.properties'

        1 item with ID '8a8115675c67e982015c67eb4a4403e1' and name 'fr_FR.properties'

        1 item with ID '8a8115675c67e982015c67eb4a4403e1' and name 'es.properties'

        1 item with ID '8a8115675c67e982015c67eb4a4403e1' and name 'zh_Hant.properties'

        1 item with ID '8a8115675c67e982015c67eb4a4403e1' and name 'de_DE.properties'

        1 item with ID '8a8115675c67e982015c67eb4a4403e1' and name 'zh_HK.properties'

        1 item with ID '8a8115675c67e982015c67eb4a4403e1' and name 'zh.properties'

        1 item with ID '8a8115675c67e982015c67eb4a4403e1' and name 'zh_TW.properties'

        1 item with ID '8a8115675c67e982015c67eb4a4403e1' and name 'zh_Hans.properties'

        1 item with ID '8a8115675c67e982015c67eb4a4403e1' and name 'en_US.properties'

        1 item with ID '8a8115675c67e982015c67eb4a4403e1' and name 'ko_KR.properties'

        1 item with ID '8a8115675c67e982015c67eb4a4403e1' and name 'ko.properties'

        1 item with ID '8a8115675c67e982015c67eb4a4403e1' and name 'fr.properties'

        1 item with ID '8a8115675c67e982015c67eb4a4403e1' and name 'de.properties'

Resolve duplicates by deleting unnecessary items.

java.lang.RuntimeException: ch.dunes.vso.db.check.CheckDuplicatesException:

Duplicate entries found in the Orchestrator database:

Resource element duplicates:

        1 item with ID '8a8115675c67e982015c67eb4a4403e1' and name 'zh_MO.properties'

        1 item with ID '8a8115675c67e982015c67eb4a4403e1' and name 'ja_JP.properties'

        1 item with ID '8a8115675c67e982015c67eb4a4403e1' and name 'en.properties'

        1 item with ID '8a8115675c67e982015c67eb4a4403e1' and name 'ja.properties'

        1 item with ID '8a8115675c67e982015c67eb4a4403e1' and name 'es_ES.properties'

        1 item with ID '8a8115675c67e982015c67eb4a4403e1' and name 'zh_CN.properties'

        1 item with ID '8a8115675c67e982015c67eb4a4403e1' and name 'fr_FR.properties'

        1 item with ID '8a8115675c67e982015c67eb4a4403e1' and name 'es.properties'

        1 item with ID '8a8115675c67e982015c67eb4a4403e1' and name 'zh_Hant.properties'

        1 item with ID '8a8115675c67e982015c67eb4a4403e1' and name 'de_DE.properties'

        1 item with ID '8a8115675c67e982015c67eb4a4403e1' and name 'zh_HK.properties'

        1 item with ID '8a8115675c67e982015c67eb4a4403e1' and name 'zh.properties'

        1 item with ID '8a8115675c67e982015c67eb4a4403e1' and name 'zh_TW.properties'

        1 item with ID '8a8115675c67e982015c67eb4a4403e1' and name 'zh_Hans.properties'

        1 item with ID '8a8115675c67e982015c67eb4a4403e1' and name 'en_US.properties'

        1 item with ID '8a8115675c67e982015c67eb4a4403e1' and name 'ko_KR.properties'

        1 item with ID '8a8115675c67e982015c67eb4a4403e1' and name 'ko.properties'

        1 item with ID '8a8115675c67e982015c67eb4a4403e1' and name 'fr.properties'

        1 item with ID '8a8115675c67e982015c67eb4a4403e1' and name 'de.properties'

Resolve duplicates by deleting unnecessary items.

        at com.vmware.o11n.integration.initialization.DatabaseInitializer.initializeSchema(DatabaseInitializer.java:77)

        at com.vmware.o11n.configuration.database.install.DatabaseInstallerImpl.installDatabase(DatabaseInstallerImpl.java:30)

        at com.vmware.o11n.cli.configuration.commands.db.DbCommand.execute(DbCommand.java:100)

        at com.vmware.o11n.cli.configuration.commands.ConfigurationCommand.executeCmd(ConfigurationCommand.java:103)

        at com.vmware.o11n.cli.configuration.ConfigurationCli.executeCommand(ConfigurationCli.java:143)

        at com.vmware.o11n.cli.configuration.ConfigurationCli.main(ConfigurationCli.java:121)

Caused by: ch.dunes.vso.db.check.CheckDuplicatesException:

Duplicate entries found in the Orchestrator database:

Resource element duplicates:

        1 item with ID '8a8115675c67e982015c67eb4a4403e1' and name 'zh_MO.properties'

        1 item with ID '8a8115675c67e982015c67eb4a4403e1' and name 'ja_JP.properties'

        1 item with ID '8a8115675c67e982015c67eb4a4403e1' and name 'en.properties'

        1 item with ID '8a8115675c67e982015c67eb4a4403e1' and name 'ja.properties'

        1 item with ID '8a8115675c67e982015c67eb4a4403e1' and name 'es_ES.properties'

        1 item with ID '8a8115675c67e982015c67eb4a4403e1' and name 'zh_CN.properties'

        1 item with ID '8a8115675c67e982015c67eb4a4403e1' and name 'fr_FR.properties'

        1 item with ID '8a8115675c67e982015c67eb4a4403e1' and name 'es.properties'

        1 item with ID '8a8115675c67e982015c67eb4a4403e1' and name 'zh_Hant.properties'

        1 item with ID '8a8115675c67e982015c67eb4a4403e1' and name 'de_DE.properties'

        1 item with ID '8a8115675c67e982015c67eb4a4403e1' and name 'zh_HK.properties'

        1 item with ID '8a8115675c67e982015c67eb4a4403e1' and name 'zh.properties'

        1 item with ID '8a8115675c67e982015c67eb4a4403e1' and name 'zh_TW.properties'

        1 item with ID '8a8115675c67e982015c67eb4a4403e1' and name 'zh_Hans.properties'

        1 item with ID '8a8115675c67e982015c67eb4a4403e1' and name 'en_US.properties'

        1 item with ID '8a8115675c67e982015c67eb4a4403e1' and name 'ko_KR.properties'

        1 item with ID '8a8115675c67e982015c67eb4a4403e1' and name 'ko.properties'

        1 item with ID '8a8115675c67e982015c67eb4a4403e1' and name 'fr.properties'

        1 item with ID '8a8115675c67e982015c67eb4a4403e1' and name 'de.properties'

Resolve duplicates by deleting unnecessary items.

        at ch.dunes.vso.db.check.ValidateDb.checkForDuplicates(ValidateDb.java:126)

        at com.vmware.o11n.integration.initialization.DatabaseInitializer.initializeSchema(DatabaseInitializer.java:74)

        ... 5 more

Error updating the DB.

Script /etc/bootstrap/postupdate.d/20-vcac failed, error status 2

[gradinka]

Seems like some duplicate entries at the Orchestrator database 😕

I'll circle back with the team and report. Please open an SR with vmware support for tracking!

[anba89]

Thanks, I will open a SR with VMware support.

[123asd]

I´m facing the same issue. Please share the solution if there is one.

[ivand]

You can open Java client and remove duplicates before upgrade of vRA.

You need to navigate to resource elements view in designer mode and examine the structure. You should check what of the duplicates is used and delete the other one.

Hope this helps

[anba89]

According to VMware support engineering is still trying to find the root cause of this issue.

Hopefully they will have some news by Monday/Tuesday.

[gradinka]

what ivand recommends should help;

rootcause is really duplicate elements in vRO / vRO DB. Those should be cleared manually before the upgrade.

question: have you used the "Code Stream Management Pack for IT DevOps (Houdini)" at some point in the past?

[123asd]

No, we haven´t used Houdini.

[MikeNox]

This was caused because you have duplicate resource elements and configuration items.  Most likely from importing packages (I've seen it happen a few times).

Find each of these items (there will be more than 1 each) and delete the extras.  Then re-run the install.

[anba89 · Accepted solution]

I was able to successfully upgrade today from 7.3.0 to 7.4.0 with VMware support.

We have done the following steps:

1. Restore the vRA Appliance and the IaaS Host to the latest snapshot (before start upgrading to 7.4.0)
2. Take a Backup
3. Start vRA and IaaS Host
4. Connect via SSH to the vRA Appliance and run the following commands
   1. su postgres
   2. /opt/vmware/vpostgres/current/bin/psql
   3. \c vcac
   4. run a select statement --> added at the end of the post.
   5. if the output looks like this, then you are not able to upgrade:
   6. 
   7. In my case the problem was the Horizon Plug-In in vRO, I had to uninstall it. Since I don't need it, that wasn't a problem. If it's needed, you might have to check with support.
   8. Manually removing plug-ins from vRealize Orchestrator 7.3 and later (2151653): VMware Knowledge Base
5. Unfortunately, the entries were still there after the plug-in was deleted. And I had to delete the Horizon Resources folder on the vRO.
6. Connect to the vRO Java Client
   1. Go to Tools --> User preferences --> Select Delete non empty folder permitted (deselect it after you deleted the folder)
   2. Change the view to Design
   3. Go to the Resources Tab
   4. Open Library and delete the Folder Horizon --> !!  In my case the problem was the Horizon Plug-In in vRO, I had to uninstall it. Since I don't need it, that wasn't a problem. If it's needed, you might have to check with support !!
7. After deleting the folder, execute the select script again.
8. Then it should look like this.
   1. 
9. Then you can start upgrading to 7.4.0. It took me about 60 minutes including IaaS host.

Soon there should be public KB for this issue.

Hope this helps.

Andi

Select Statement:

select * from (

select categoryid, name, count(*)-1, 'vmo_configelement' as type from vmo_configelement group by categoryid, name having count(*) > 1

union

select parentcategoryid, name, count(*)-1, 'vmo_configelementcategory' as type from vmo_configelementcategory group by parentcategoryid, name having count(*) > 1

union

select elementid, '' as name, count(*)-1, 'vmo_configelementcontent' as type from vmo_configelementcontent group by elementid having count(*) > 1

union

select '' as id, name, count(*)-1, 'vmo_package' as type from vmo_package group by name having count(*) > 1

union

select packageid, '' as name, count(*)-1, 'vmo_packagecontent' as type from vmo_packagecontent group by packageid having count(*) > 1

union

select policyid, '' as name, count(*)-1, 'vmo_policycontent' as type from vmo_policycontent group by policyid having count(*) > 1

union

select categoryid, name, count(*)-1, 'vmo_resourceelement' as type from vmo_resourceelement group by categoryid, name having count(*) > 1

union

select elementid, '' as name, count(*)-1, 'vmo_resourceelementcontent' as type from vmo_resourceelementcontent group by elementid having count(*) > 1

union

select scriptmodulecategoryid, name, count(*)-1, 'vmo_scriptmodule' as type from vmo_scriptmodule group by scriptmodulecategoryid, name having count(*) > 1

union

select '' as id, name, count(*)-1, 'vmo_scriptmodulecategory' as type from vmo_scriptmodulecategory group by name having count(*) > 1

union

select scriptmoduleid, '' as name, count(*)-1, 'vmo_scriptmodulecontent' as type from vmo_scriptmodulecontent group by scriptmoduleid having count(*) > 1

union

select dunesobjectid, version, count(*)-1, 'vmo_versionhistorycontent' as type from vmo_versionhistorycontent group by dunesobjectid, version having count(*) > 1

union

select workflowid, '' as name, count(*)-1, 'vmo_workflowcontent' as type from vmo_workflowcontent group by workflowid having count(*) > 1

) as u;

[Windspirit]

BRILLIANT WORK anba89

I had the same problem...however it wasn't a plugin in my case but an Config element that had the same name but was in differnt folders. Removing the config elemnet and cleaning up its dependencies (all workflows that used that conifg element) solved the problem.

[magoon1]

Ours was similar but not plugin related.  We are not using Horizon plugin but are using the embedded vIDM which has some Horizon code so it wasn't applicable.  Here is the order of events we did with VMware Technical Support:

KB reference link:  Upgrade vRealize Automation 7.3.0 to 7.4 failed

1.  We captured the /var/log/bootstrap on primary appliance after failure.

     Portion relevent within file:

          Configuration element duplicates:

                  1 item with categoryid '8a7dc2326158633d01617737063140e4' and name 'InstanceDetails'

                  1 item with categoryid '8a7dc2326158633d01617737063140e4' and name 'Allocation'

          Configuration element category duplicates:

                  1 item with parentcategoryid 'null' and name 'Azure'

2.  Reverted snaps

3.  Putty into primary vRA VA as root

4.  Ran following commands:

          su postgres

          /opt/vmware/vpostgres/current/bin/psql

          \c vcac

          \dt

5.  In our case, the following tables were affected:

          vmo_configelement

          vmo_configelementcategory

----Performed following for vmo_configelement table----

a.  Ran the following command to capture the appropriate column names:

          select * from vmo_configelement;

b.  Ran the following command:

          select id, categoryid, name from vmo_configelement order by name;

               We saw the second duplicate IDs for names InstanceDetails and Allocation:

                    fae18e53-79b8-4a2e-8da9-c3353d288b98

                    c8b1645a-e4a1-430b-a6a3-eae4b98109e5

c.  Ran the following commands to delete from the vmo_configelement table:

          delete from vmo_configelement where id='fae18e53-79b8-4a2e-8da9-c3353d288b98';

          delete from vmo_configelement where id='c8b1645a-e4a1-430b-a6a3-eae4b98109e5';

d.  Ran the following command to confirm deletion from the vmo_configelement table:

           select id, categoryid, name from vmo_configelement order by name;

----Performed following for vmo_configelementcategory table----

aa.  Ran the following command for the vmo_configelementcategory table to capture the appropriate column names:

          select * from vmo_configelement;

bb.  Ran the following command for the vmo_configelementcategory table:

          select id, categoryid, name from vmo_configelement order by name;

               We saw the second duplicate id for name Azure:

                    8a7dc2326158633d01617737063140e4

cc.  Ran the following commands to delete from the vmo_configelementcategory table:

          delete from vmo_configelementcategory where id='8a7dc2326158633d01617737063140e4' and name='Azure';

dd.  Ran the following command to confirm deletion from thevmo_configelementcategory table:

           select id, categoryid, name from vmo_configelementcategory order by name;

----Table cleanup complete

Note:  You can install and execute the RPM (listed in KB) and should receive the following:

The command does not need database configuration update.

Rerun the update