vCenter 5.5 certificate replacement issue

[Godric7000]

I'm having problems getting the self signed certificates for vCenter v5.5 replaced with Enterprise CA certificates.  I am using the certificate replacement tool from VMware.  The SSO certificate is successfully replaced so I move on to having the Inventory service trust the SSO certificate.  They succeeds as well but I think all its doing is bouncing the Inventory service.  I then go to install the Inventory service certificate and get:

[Sat 03/15/2014 - 15:51:06.27]: The services that are restarted as a part of this operation are: vCenter Inventory Service.

Enter the location to the new Inventory Service SSL chain (default value is: c:\certs\Inventory\chain.pem):

Enter the location to the new Inventory Service private key (default value is: c:\certs\Inventory\rui.key):

Enter the Single Sign-On Administrator user (default value is: administrator@vsphere.local):

Enter the Single Sign-On Administrator password (will not be echoed):

[.] The supplied certificate chain is valid.

[Sat 03/15/2014 - 15:51:20.40]: Last operation update Inventory Service SSL certificate failed :

[Sat 03/15/2014 - 15:51:20.41]: Cannot determine if Inventory Service is registered with Single Sign-On - errorlevel is 1

If I look at the logs, I see the following:

[Sat 03/15/2014 - 15:51:15.43]: The Inventory Service is installed at "C:\Program Files\VMware\Infrastructure\Inventory Service"

[Sat 03/15/2014 - 15:51:15.44]: Rollback path is "C:\ssl-certificate-updater-tool-1308332\backup"

[Sat 03/15/2014 - 15:51:15.45]: Rollback path is "C:\ssl-certificate-updater-tool-1308332\backup\IS"

[Sat 03/15/2014 - 15:51:18.03]: Determining whether Inventory Service is registered with Single Sign-On ...

Intializing registration provider...

Getting SSL certificates for https://FP-CSVC01.domain.loc:7444/lookupservice/sdk

com.vmware.vim.vmomi.core.exception.CertificateValidationException: Server certificate chain not verified

Return code is: SslHandshakeFailed

1

[Sat 03/15/2014 - 15:51:20.39]: "Cannot determine if Inventory Service is registered with Single Sign-On - errorlevel is 1"

[Sat 03/15/2014 - 15:51:20.39]: Exiting Inventory Service update SSL certificate due to errors

Obviously there is something about the CA chain that it doesn't like.  If I look at my cert store on the vCenter server, I have my Root CA in the Trusted Root store.  In the Intermediate Store I have both the Root and Intermediate.  If I browse to the lookup service (https://FP-CSVC01.domain.loc:7444/lookupservice/sdk) from a web browser, the certificate shows as valid and throws no errors so there should be nothing wrong with the certificate.

The format of the chain.pem in the Inventory directory is correct as well.  It is the Inventory cert, followed by the Intermediate cert, followed by the Root cert.  No extra spaces anywhere.

I have also tried to manually replace the certificates and it essentially fails at the same spot.  SSO replacement goes fine then I go to unregister the Inventory service and the SSL handshake fails.

Funny thing is that if I am using vCenter 5.1, I get past this all without issue.

Thoughts??

[akodenkiri]

Can you please follow below KB for SSL certificate replacement ? Looks like you are providing wrong SSO password while replacing the SSL certificate.

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=205734...

-Ak

[Godric7000]

I did follow that KB for 5.5.  I even tried the manual replacement not using the certificate replacement program and had the same issues.

The password is not incorrect.  I put my passwords into a password database then copy/paste them so there are no issues with typing the password.  To my knowledge, the program will throw an error saying the password is incorrect anyhow.

[admin]

What special character are you using in your SSO password?

I take this as a given but better ask than be sorry afterwasrds: FP-CSVC01.domain.loc is actually in the subject alternate names of your SSO certificate?

What does c:\program files\vmware\infrastructure\vmware\cis\vmware-sso\ssolscli listServices https://FP-CSVC01.domain.loc:7444/lookupservice/sdk return on a admin command line? Twice error 100 or does it list some endpoints?

[Godric7000]

Frank...

The only special character is a single exclamation point which I believe is a supported character.  I tried doing a vCenter 5.1 install with the same password and made it past this point.

The real domain name was omitted from this post for security reasons.  I accidentally forgot to list the subdomain in the FQDN so it is more like: FP-CSVC01.subdomain.domain.loc

I just ran the command you suggested and I get two certificate chain errors, a return code OperationFailed, and 100

[admin]

This usually means that you don't have the correct subject alternate name in your certificate, did you include the IP? If yes, can you ran the same command but use the IP address instead of the fqdn?

Believe it or not but I have seen issues sometimes with "!", as this can serve as an escape character now as well sometimes ... good experience with "@" and "." so far from my side.

[pwyde]

I am experiencing the exact same problem. vCenter Server 5.5 installation is on a brand new VM running Windows Server 2012 R2. vCenter Server is also new. I've been following Derek Seaman's excellent blog post about two-tier PKI infrastructure and vSphere 5.5 installation.

I am able to replace the vCenterSSO certificate. When I move to replacing Web Client certificate with a self-signed CA certificate, I receive the error below in vCenter Certificate Automation Tool v5.5.

==================================================================

7. Update the vSphere Web Client and Log Browser SSL Certificates

     1. Update the Web Client Trust to Single Sign-On

     2. Update the Web Client Trust to Inventory Service

     3. Update the Web Client Trust to vCenter Server

     4. Update the Web Client SSL Certificate

     5. Update the Log Browser Trust to Single Sign-On

     6. Update the Log Browser SSL Certificate

     7. Rollback to the previous Web Client SSL Certificate

     8. Rollback to the previous Log Browser SSL Certificate

     9. Return to the main menu to update other services

The chosen service is: 4

[2014-06-08 -  1:25:36,70]: The services that are restarted as a part of this op

eration are: vSphere Web Client

Enter location to the new Web Client SSL chain (default value is: C:\Tools\Certi

ficates\vCenterWebClient\chain.pem):

Enter location to the new Web Client private key (default value is: C:\Tools\Cer

tificates\vCenterWebClient\rui.key):

Enter Single Sign-On Administrator user (default value is: administrator@vsphere

.local):

Enter Single Sign-On Administrator password (will not be echoed):

[.] The supplied certificate chain is valid.

[2014-06-08 -  1:25:50,86]: Last operation update vSphere Web Client SSL certifi

cate failed :

[2014-06-08 -  1:25:50,86]: Cannot validate the Lookup Service connection - erro

rlevel is 1

I get the below error when executing ssolscli.cmd.

C:\Program Files\VMware\Infrastructure\VMware\CIS\vmware-sso>ssolscli.cmd listSe

rvices https://192.168.12.8:7444/lookupservice/sdk

Intializing registration provider...

Getting SSL certificates for https://192.168.12.8:7444/lookupservice/sdk

com.vmware.vim.vmomi.core.exception.CertificateValidationException: Server certi

ficate chain not verified

com.vmware.vim.vmomi.core.exception.CertificateValidationException: Server certi

ficate chain not verified

Return code is: OperationFailed

100

Were you or anybody else able to resolve the above issue? Any thoughts and/or suggestions are greatly appreciated.

[Godric7000]

HA!   You did the same thing I did.  Followed Derek's blog on PKI followed by his vCenter 5.5 blog.  The problem is with using his CAPolicy.inf file.  He has AlternateSignatureAlgorithm enabled which turns on an unsupported signature algorithm.  If you eliminate that line from the inf and reinstall you should be ok. 

I found the issue when I compared my lab vCenter 5.1 deployment cert where when I created the PKI I just ran through the install and didn't use a capolicy.inf.  The signature algorithms were different.  That's when I found this one vCenter security doc that talks about what signature algorithms were and were not supported.  The one Derek's installs was not supported!

hope that helps

[pwyde]

Thanks a million for the tip! So basically I have to tear down my whole PK? Meaning revoke Root CA cert and Sub CA cert and re-install both the Root CA and Sub CA with the updated CAPolicy.inf?

[Godric7000]

More than likely yes.  I just blew my entire environment away and started fresh because monkeying with the CA servers after I found out what was wrong totally blew up my SQL servers which were using SSL encryption. 

[pwyde]

I just took down my whole Windows domain environment (luckily it was a virtualized lab) and re-did the whole thing from domain controllers to enterprise subordinate CA and boom! The self-signed SSL certificates in vSphere worked! So I am very thankful and once again I must say thanks a million for pointing me to the root cause of the problem. I can now finally implement the self-signed SSL certificates in my production environment.

[Godric7000]

You don't know how many hours I spent taking rebuilding pieces of the setup, doing manual certificate imports, etc before I got it working.  Thankfully snapshots exist. 

Glad I could help.

[Kingofbytes]

Godric7000

THANK YOU.  Indeed my whole install was doomed from the beginning because of the "AlternateSignatureAlgorithm" issue.

My CA was server 2008 R2.  There was NO AlternateSignatureAlgorithm registry entry.  Guess what.  That means AlternateSignatureAlgorithm defaults to "1" even without the entry in the registry.  So you must explicitly put in the entry and set it to 0.

Once I did that, I redid all the certs with Derek's powershell toolkit as well as the Automation tool and things are working well!  THANK YOU for pointing me to the original sin!

[digidwain1]

Okay, what do you mean by alternateSignature Algorithm? I was following the VMWare KB not this Derek thread.  I get no errors on the ssls but I do get an unable to connect to server issue when I am trying to replace the expired certs.  I even reset the DB password - no love.