GotToBeStrong
Contributor
Contributor

Migration assistant pre-check fails on source vcenter SSO certificate

Good Day,

Attempting to migrate from vCenter Server (Windows) 6.0.U3i to 6.5 (Appliance) which I'm reading is supported.  Migration-Assistant  pre-checks at the source vCenter Server fail on SSO service certificates.

I've recently (this week) re-generated all certificates with the VMCA option using both selections 3 and 6 in the vCenter certificate manager tool.  So, I know machine and service certificates are current and there is a current, valid self-signed CA cert as both of those operations completed successfully (additionally, the vSphere Client recognized the new thumbprints and prompted me to install the new certificates - after each of the above operations).  I even had to re-generate a new certificate for VUM separately.  I can also validate the CA cert when querying the cryptography using the vSphere tools in the Windows file system (vecs-cli and dir-cli).

(Background) The origins of this vCenter Server was 5.1 or so.  Historically, when this vCenter Server was originally built (many years ago) we had used internal CA certificates which required that we include the CA and intermediate CA cert paths in the signed certs we uploaded.  Now, those original CA and Intermediate CA certificates have both expired (long ago at this point) and were renewed at the CA level, so invalidated the machine certificate.  This did not seem to break anything.  We have since replaced the machine and service certificates with (as mentioned above) VMCA signed certificates.  (Everything continues to function at this point, however we haven't done anything funky like attempting to remove and reattach hosts to vCenter.)

In anticipation of our planed upgrade/migration, I didn't see any problem with applying VMCA certificates in place of renewing the domain CA signed certs as it was much faster to complete, would only be temporary for the duration of the migration and I needed to ensure that there were current/valid certificates for the migration.

So, despite the fact that I've supplied/applied (I believe) new, valid certificates to all of the correct endpoints in the vCenter server and everything is still functioning; the migration assistant is still complaining about the SSO service(s) certs and shuts down - preventing the migration.

After quite a bit of research, I've found that:
The valid certificates seem to be assigned to the services.
The current CA (VMCA) is available in the Trusted_Root store
There are multiple, expired certificates (old machine cert signed by expired CA, expired domain CA cert, expired intermediate domain CA) in both the Trusted_Root store and the trustedcert stores on the Windows vCenter Server (Migration Source).

I can validate this by using the same commands above (vecs-cli and dir-cli).

 

My question(s) is/are:

I found an article on removing old CA/Certs from vCenter (VM KB 2146011); so in theory if I follow those steps I should be able to selectively remove only the expired CA and Certs from those 2 certificate stores.
What I'm unsure of is this:  When I remove the expired certificates from those stores, assuming all that remains are valid CA and Certs - will the services displaying those certificates (results from vecs-cli and dir-cli) switch over their SSL trust to what remains, or will I have to first assign the correct Cert/CA thumbprints to those services, even though they seem to present current/correct certificates during normal operation.

If I have to manually assign the current VMCA (by defining the certificate thumbprint or however) to the SSO services PRIOR to removing the old CA/Certs from the certificate stores, then I need to please find some instructions on the manual steps involved in that process.  I have read a number of articles that only apply to older versions of vCenter Server (4.x/5.x when this operation was more complicated) but don't seem to apply to vCenter 6.x and also don't seem to apply to my specific issue in that I seemingly need to only address the SSO certificate(s) - and not the platform services as a whole - in order to allow the migration-assistant to continue.

My hopeful assumption would be that I can simply follow the steps in the KB above and remove the expired CA, Intermediate-CA certificates from the respective vCenter Certificate Stores; restart services (or reboot the vCenter Server Windows instance) and have the services pick up on the fact that all that remains is the current CA/Certs and apply them on-the-fly.

So, if anyone has any insight, advise or has run into this conundrum before, please feel free to let me know and/or point me at any relevant documentation covering how to manually change only the SSL Trust for these services one at a time - by service GUID so I can ensure that they are only referencing current, valid certs and CA and (hopefully) then proceed with the migration.

Alternatively, is there a way to force the migration to ignore any invalid CA certificates (a command line switch perhaps)?  Given that the machine cert and all service cert(s) are supposedly up to date based on the certificate manager operations that I just performed; realistically, the expired certs shouldn't have any impact at all...

 

                   Please advise,

 

 

Update:  I've managed to make some progress and have removed all expired certificates from all keystores using the dir-cli command.  I also found out that i was missing a cert in the STS store, so i exported the machine_cert and imported it as the sts_cert as per a vmware kb.  So now i have valid certs in all cert stores on the vcenter server, however i still have one (hopefully only one) conundrum...

The VMWare Secure Token Service (STS) service under Windows is still referencing an old cert.  I found a kb article about generating a new key and cert; combining them into a pkcs file and importing that into the java jks file for vCenter Server, however the next step - refreshing the cert in the service - requires web client access which is impossible as Flash has been sunset and I have no legacy Windows floating around capable of reaching the vCenter server.

Is there a command-line way of finalizing that STS service certificate refresh once I've imported it into the .jks file as per the KB?  The web-client only executes API commands which essentially equates to a command line anyway.  There has to be a way for me to finalize this fix and move on with my migration...???

I found fixsts.ps1 which I believe might address my issue, however it doesn't work on my current edition of vCenter, however I believe works with vcenter 6.5; so alternatively, I'm thinking an in-place upgrade (if the certificate issue won't roadblock that as well) and then try to fix the cert, then migrate...

0 Kudos
0 Replies