Upgrade to vCSA 6.5 fails, "this appliance cannot be used or repaired", source and target vCenter left unreachable

[EdSp]

Hi,

I was trying to upgrade our vCenter from 6.0 (6.0.0U2) to 6.5U1. The 2 external platform service controllers (PSC's) were first successfully upgraded. Next and last was the upgrade of the vCenter Appliance itself. All seemed to go well: stage 1 (deploy) done; running stage 2 (upgrade).

The upgrade failed with the message: "This appliance cannot be used or repaired because a failure was encountered. You need to deploy a new appliance. Failure details"

The failure details show that the java process has failed:

An error occurred while invoking external command : 'Command: ['/usr/java/jre-vmware/bin/java', u'-Djava.endorsed.dirs=/var/opt/apache-tomcat-8.5.8/endorsed', u'-Djava.security.properties=/etc/vmware/java/vmware-override-java.security', u'-Djava.ext.dirs=/usr/java/jre-vmware/lib/ext:/usr/java/packages/lib/ext:/opt/vmware/jre_ext/', u'-classpath', u'/usr/lib/vmware-eam/dbmigrate/dbmigrate.jar:/etc/vmware/../vmware-eam/dbmigrate/:/etc/vmware/../vmware-eam/:/usr/lib/vmware-eam/web/webapps/eam/WEB-INF/classes/:/usr/lib/vmware-eam/web/webapps/eam/WEB-INF/lib/*:/usr/lib/vmware/common-jars/spring-core-4.3.5.RELEASE.jar:/usr/lib/vmware/common-jars/spring-context-4.3.5.RELEASE.jar:/usr/lib/vmware/common-jars/spring-beans-4.3.5.RELEASE.jar:/usr/lib/vmware/common-jars/spring-web-4.3.5.RELEASE.jar:/usr/lib/vmware/common-jars/log4j-1.2.17.jar:/usr/lib/vmware/common-jars/commons-lang-2.6.jar:/usr/lib/vmware/common-jars/commons-codec-1.6.jar:/usr/lib/vmware/common-jars/commons-logging-1.1.3.jar:/usr/lib/vmware/common-jars/httpclient-4.3.3.jar:/usr/lib/vmware/common-jars/httpcore-4.3.2.jar:/usr/lib/vmware/common-jars/stax2-api-3.0.1.jar:/usr/lib/vmware/common-jars/woodstox-core-asl-4.0.5.jar:/usr/lib/vmware/common-jars/spring-webmvc-4.3.5.RELEASE.jar:/usr/lib/vmware/common-jars/spring-expression-4.3.5.RELEASE.jar:/usr/lib/vmware/common-jars/commons-io-1.4.jar:/usr/lib/vmware/common-jars/slf4j-api-1.7.12.jar:/usr/lib/vmware/common-jars/slf4j-log4j12-1.7.12.jar', u'com.vmware.eam.dbmigrate.Main']

Stderr: Picked up JAVA_TOOL_OPTIONS: -Xms32M -Xmx128M '

Unable to migrate ESX Agent Manager persistent data

At this point I cannot access either the source (6.0) or target (new 6.5) appliance with the vSphere Client (thick/web).

I searched around and it looks like a possibility that the message printed on Stderr "Picked up ..." may be why the Java process is seen as "failed"???

Any tips greatly appreciated.

Thanks,

Ed

Message was edited by: EdSp Added exact source version (6.0.0U2).

[EdSp]

I have reverted the involved ESXi's to their last snapshot and am now rerunning the upgrade.

[EdSp]

The re-run of the update from 6.0u2 to 6.5u1 resulted in the same JAVA error, which fails when trying to start ESX Agent Manager process (EAM). The message is: Unable to migrate ESX Agent Manager persistent data

This is in stage 2, "Data transfer and appliance setup is in progress", step 2 "Set up target vCenter Server and start services".

I noticed the following:

• the vCenter 6.0 VM is shut down.
• the vCenter 6.5 VM is up and running
• login with the vSphere Client fails
• login with the vSphere Web Client (on either original or temp IP) is successful
• Going to https://<vCenter65>:5480, I can see the error message again or click to get the Support Bundle.

(edit) Found the following information in the EAM logfile (/var/log/vmware/eam/eam_dbmigrate.log) that is possibly of interest:

:

2017-08-22T13:15:03.886Z | ERROR | main | VcConnection.java | 650 | Error communicating with the vCenter server.

com.vmware.vim.vmomi.client.exception.SslException: com.vmware.vim.vmomi.core.exception.CertificateValidationException: Server certificate assertion not verified and thumbprint not matched

:

Caused by: javax.net.ssl.SSLException: hostname in certificate didn't match: <192.168.133.203> != <ei>

        at org.apache.http.conn.ssl.AbstractVerifier.verify(AbstractVerifier.java:231)

        at org.apache.http.conn.ssl.StrictHostnameVerifier.verify(StrictHostnameVerifier.java:61)

        at org.apache.http.conn.ssl.AbstractVerifier.verify(AbstractVerifier.java:152)

:

I am currently stuck with this vCenter 6.5 upgrade and don't know how to proceed or know for sure what exactly is causing this failure.

Thanks,

Ed

Edit: Looks like we are on 6.0u3 after all, which is not supported for direct upgrade to 6.5u1. Going to try the procedure found here: Cannot upgrade VCSA 6.0 to 6.5 - Internal error occurs during Export of vSphere ESX Agent Manager

[MohanBollu]

Hi,

Your upgrade path from 6.0u3\6.0u2 to 6.5u1 is supported as per interop matrix of vCenter.

VMware Product Interoperability Matrices

Could you please provide complete eam firstboot log.

Thanks,

Mohan

[Wunderwuzi]

I have had the same problem. Searching the root with vmware support, they have no solution. According to them, this could be a certificate problem. Trying to uninstall our certificates and reinstall the original certificate provided by vmware, fails too. Spent three weeks with this problem and vmware support.

Therefore, creating a new vCenter 6.0 with the same parameters and original certifcate the upgrade works fine.

[Raphael_Gassman]

Hi, did you ever managed to get a solution or workaround? Would appreciate any hint.

[Raphael_Gassman]

Hi, did you ever managed to get a solution or workaround or was the version interoperability for upgrade 6.0u3 to 6.5 the issue? I'm running into the same problem during vCenter Upgrade 6.0u1 to 6.5u1 Would appreciate any hint.