VCSA 6.0 -> 6.5 CLI Upgrade Fails at 58% with Vmware VirtualCenter failed firstboot.

[GBartsch]

We have a 6.0 vCenter Server Appliance with two external PSC [all 6.0.0.30800 build 9448190 / 6.0 Update 3h] we are attempting to upgrade to 6.5 U2e build 11347054.

We've been completely unsuccessful trying to use the GUI updater, as for some unknown reason the GUI upgrader will not connect to the vCenter/PSC during the initial connection in Stage 1.

However, with the CLI upgrader using a .JSON file, we've upgraded the PSCs (two external PSCs) without issue.

The vCenter Server Appliance, however, fails to upgrade with an error at 58%:

Progress: 58% Starting VMware vCenter Server...

Error:

     Problem Id: install.vpxd.action.failed

     Component key: vpxd

     Detail:

          Vmware VirtualCenter failed firstboot.

          An error occurred while invoking external command : 'Command: ['/usr/sbin/vpxd', '-L'] Strerr: '

     Resolution: Please search for these symptoms in the VMware Knowledge Base.....

...

     vCSACliInstallLogger - DEBUG - Running command on vm [new vCenter name]: /bin/bash --login -c 'ls `install-parameter upgrade.import.directory` /system-data/revert_networking.py'

     vCSACliInstallLogger - DEBUG - Running command on vm [new vCenter name]: /bin/bash --login -c '/opt/vmware/bin/python `install-parameter upgrade.import.directory` /system-data/revert_networking.py'

     vCSACliInstallLogger - ERROR - Fail to revert the target vm IP address: Failed to run and wait for command in guest with error 'Command '[u'/opt/vmware/bin/python', u'`install-parameter upgrade.import.directory`/system-data/revert_networking.py']' exited with non-zero status 1'

We were able to find a KB with the 'Command: ['/usr/sbin/vpxd', '-L'] Strerr: ' issue listed, and it seems to refer to duplicate vDS and vDPG names.  However we were not able to find any dupes.

(KB 2147547 for the vDS / vDPG issue: VMware Knowledge Base and a related one showing how to connect to postgres VMware Knowledge Base KB 2147285.)

There is only one additional issue seen in the vcsa-installer.log.  We see a message "Failed normalizing ip: [FQDN of the vCenter being upgraded"

Does anyone have any ideas on this one?

[RickVerstegen]

Already rebooted the vCenter Server Appliance before upgrading? Also verify that the FQDN is resolvable.

[GBartsch]

We were able upgrade the PSCs (2 external PSCs) without issue.

I have rebooted and tried this upgrade several times.  Same exact error every time...

Forward and Reverse DNS lookup zones are populated.  You can ping all of the appliances from each other. NSLOOKUPs working without issue.

[msripada]

If the new appliance 6.5 is created, please check the vpxd.log in the newly created 6.5 appliance which can shed some light on why the service firstboot failed.

Thanks,

MS

[GBartsch]

So, not a total solution, but we downloaded the 6.5 U2G version and tried that this AM....

....and guess what?  It got past 58%!

Then it blew up on the VMware Authentication Services at 80%, which there is some sort of KB about that in 6.7.

[GBartsch · Accepted solution]

It turned out that the 6.0 vCenter server had some duplicate ROOT and INTERMEDIATE certificates in the TRUSTED ROOT STORE.

Somehow some root CAs were not imported with the certificate thumbprint as the ALIAS for the certs; they were in fact the file name used for the certs when they were brought to the vCenter.

As such, there were two sets of the same certificates with different aliases.

This was causing the upgrade to explode when attempting to start the VPXD service. (...at 58%.)

PSC's did not appear to have the same issue; only the VPXD (virtual center server service) seemed to have this issue.

It is abolutely a bug that having two copies of the the same trusted root and intermediate certificates would cause a service to not start.

[GBartsch]

I was speaking with support the other day, and this issue is not commonly known.

I'll see what I can do to reach back to engineering and have that fixed.  However, it's pretty rare thing...

[GBartsch]

Some Updates:

After upgradeing the vCenter 6.0 appliances to 6.5, we noticed that with 6.7 we saw some other issues:

The PSCs when going from 6.5 -> 6.7 also had some duplicate root certificates.

At the end of the day, all appliances, vCenter or PSC, need to have any duplicate roots removed JUST prior to the upgrade.

It is critical that you do not restart any services. Simply SSH into the box just prior to upgrade (after you've shut down the entire vCenter appliance environment and take snapshots) and kill the duplicate certificates.

With the 6.5 upgrade installer, it doesn't notify you of any cert issues. The 6.7 upgrade installer is MUCH better, and will warn you of these issues.