GBartsch
Enthusiast
Enthusiast

VCSA 6.0 -> 6.5 CLI Upgrade Fails at 58% with Vmware VirtualCenter failed firstboot.

Jump to solution

We have a 6.0 vCenter Server Appliance with two external PSC [all 6.0.0.30800 build 9448190 / 6.0 Update 3h] we are attempting to upgrade to 6.5 U2e build 11347054.

We've been completely unsuccessful trying to use the GUI updater, as for some unknown reason the GUI upgrader will not connect to the vCenter/PSC during the initial connection in Stage 1.

However, with the CLI upgrader using a .JSON file, we've upgraded the PSCs (two external PSCs) without issue.

The vCenter Server Appliance, however, fails to upgrade with an error at 58%:

Progress: 58% Starting VMware vCenter Server...

Error:

     Problem Id: install.vpxd.action.failed

     Component key: vpxd

     Detail:

          Vmware VirtualCenter failed firstboot.

          An error occurred while invoking external command : 'Command: ['/usr/sbin/vpxd', '-L'] Strerr: '

     Resolution: Please search for these symptoms in the VMware Knowledge Base.....

...

     vCSACliInstallLogger - DEBUG - Running command on vm [new vCenter name]: /bin/bash --login -c 'ls `install-parameter upgrade.import.directory` /system-data/revert_networking.py'

     vCSACliInstallLogger - DEBUG - Running command on vm [new vCenter name]: /bin/bash --login -c '/opt/vmware/bin/python `install-parameter upgrade.import.directory` /system-data/revert_networking.py'

     vCSACliInstallLogger - ERROR - Fail to revert the target vm IP address: Failed to run and wait for command in guest with error 'Command '[u'/opt/vmware/bin/python', u'`install-parameter upgrade.import.directory`/system-data/revert_networking.py']' exited with non-zero status 1'

We were able to find a KB with the 'Command: ['/usr/sbin/vpxd', '-L'] Strerr: ' issue listed, and it seems to refer to duplicate vDS and vDPG names.  However we were not able to find any dupes.

(KB 2147547 for the vDS / vDPG issue: VMware Knowledge Base and a related one showing how to connect to postgres VMware Knowledge Base KB 2147285.)

There is only one additional issue seen in the vcsa-installer.log.  We see a message "Failed normalizing ip: [FQDN of the vCenter being upgraded"

Does anyone have any ideas on this one?

0 Kudos
1 Solution

Accepted Solutions
GBartsch
Enthusiast
Enthusiast

It turned out that the 6.0 vCenter server had some duplicate ROOT and INTERMEDIATE certificates in the TRUSTED ROOT STORE.

Somehow some root CAs were not imported with the certificate thumbprint as the ALIAS for the certs; they were in fact the file name used for the certs when they were brought to the vCenter.

As such, there were two sets of the same certificates with different aliases.

This was causing the upgrade to explode when attempting to start the VPXD service. (...at 58%.)

PSC's did not appear to have the same issue; only the VPXD (virtual center server service) seemed to have this issue.

It is abolutely a bug that having two copies of the the same trusted root and intermediate certificates would cause a service to not start.

View solution in original post

0 Kudos
8 Replies
RickVerstegen
Expert
Expert

Already rebooted the vCenter Server Appliance before upgrading? Also verify that the FQDN is resolvable.

Blog: https://rickverstegen84.wordpress.com/ Twitter: https://twitter.com/verstegenrick
0 Kudos
GBartsch
Enthusiast
Enthusiast

We were able upgrade the PSCs (2 external PSCs) without issue.

I have rebooted and tried this upgrade several times.  Same exact error every time...

Forward and Reverse DNS lookup zones are populated.  You can ping all of the appliances from each other. NSLOOKUPs working without issue.

0 Kudos
msripada
Virtuoso
Virtuoso

If the new appliance 6.5 is created, please check the vpxd.log in the newly created 6.5 appliance which can shed some light on why the service firstboot failed.

Thanks,

MS

0 Kudos
GBartsch
Enthusiast
Enthusiast

So, not a total solution, but we downloaded the 6.5 U2G version and tried that this AM....

....and guess what?  It got past 58%!

Then it blew up on the VMware Authentication Services at 80%, which there is some sort of KB about that in 6.7.

0 Kudos
GBartsch
Enthusiast
Enthusiast

It turned out that the 6.0 vCenter server had some duplicate ROOT and INTERMEDIATE certificates in the TRUSTED ROOT STORE.

Somehow some root CAs were not imported with the certificate thumbprint as the ALIAS for the certs; they were in fact the file name used for the certs when they were brought to the vCenter.

As such, there were two sets of the same certificates with different aliases.

This was causing the upgrade to explode when attempting to start the VPXD service. (...at 58%.)

PSC's did not appear to have the same issue; only the VPXD (virtual center server service) seemed to have this issue.

It is abolutely a bug that having two copies of the the same trusted root and intermediate certificates would cause a service to not start.

View solution in original post

0 Kudos
GBartsch
Enthusiast
Enthusiast

I was speaking with support the other day, and this issue is not commonly known.

I'll see what I can do to reach back to engineering and have that fixed.  However, it's pretty rare thing...

0 Kudos
GBartsch
Enthusiast
Enthusiast

Some Updates:

After upgradeing the vCenter 6.0 appliances to 6.5, we noticed that with 6.7 we saw some other issues:

The PSCs when going from 6.5 -> 6.7 also had some duplicate root certificates.

At the end of the day, all appliances, vCenter or PSC, need to have any duplicate roots removed JUST prior to the upgrade.

It is critical that you do not restart any services. Simply SSH into the box just prior to upgrade (after you've shut down the entire vCenter appliance environment and take snapshots) and kill the duplicate certificates.

With the 6.5 upgrade installer, it doesn't notify you of any cert issues. The 6.7 upgrade installer is MUCH better, and will warn you of these issues.

0 Kudos
fainabuff
Contributor
Contributor

Somehow some root CAs were not imported with the certificate thumbprint as the ALIAS for the certs; they were in fact the file name used for the certs when they were brought to the vCenter.

Any Update compass connect

0 Kudos