After a week of deploying VCSA 5.5 over and over again, probably 18 times, and seeing what must be every SSL related error possible, I am asking advice here for those who may have stumbled through this before. (Or know Linux)
On my last attempt I've installed fresh, configured the server, generated cert reqs through OpenSSL 0.9.8y, approved them through my internal CA, and attempted VMWare KB:
All goes well this time until step 22: Run these commands to register the vCenter Inventory Service back to vCenter Single Sign-On:cd /etc/vmware-sso/register-hooks.d
./02-inventoryservice --mode install --ls-server https://VCSA1.domain.com:7444/lookupservice/sdk --user administrator@vsphere.local --password sso_administrator_password
And I am rewarded with this:
Getting SSL certificates for https://VCSA1.mydomain.com:7444/lookupservice/sdk |
com.vmware.vim.vmomi.core.exception.CertificateValidationException: Server certificate chain not verified
Return code is: SslHandshakeFailed
What gives?
The SSO service never gave me any errors when doing the cert install in previous steps. The FQDN is the same in the cert as it is on the VCSA on initial setup, and yes the same case.
VMWare KB yield nothing relevant as does a Google search.
HELP! I'm going crazy trying to figure this out.
Anyone help me out here?
I've worked through the KB article 8 times, and followed Derek Seaman's blog on this, but still always the same result.
I've had the same problem, but my issue is:
com.vmware.vim.vmomi.core.exception.CertificateValidationException: Server certificate assertion not verified and thumbprint not matched
Return code is: SslHandshakeFailed (when trying to unregister or reregister the Inventory Service after successfully replacing the vCenterSSO certificate).
I believe that if you follow this article you will be able to replace your bad certs:
If you give it a try and is working please let me know. My feeling is that the ssl trust chain is broken and you will need to recreate it in a similar way.
Thank you.
Have you read Derek Seaman's bog on SSL in ESX 5.5? You can probably start here: http://www.derekseaman.com/2013/10/vsphere-5-5-install-pt-5-ssl-deep.html
You can check the chain using openssl verify. In Windows the command would be:
openssl verify -CAfile C:\pathtopem\chain.pem -verbose c:\pathtoCerfile\filename.cer
I don't know what the linux equivalent is.
When I ran into this issue, it was because the intermediate cert wasn't included in the chain. Open the cert and look at the sections and make sure your root cert is at the bottom and the vcenter cert is at the top.
I am in the same boat as you are. I even posted a question to the editor of the article that after following the steps, I get the same issue. I have to be honest here that I didn't use a Microsoft CA nor a public CA, and ran all the OpenSSL commands directly on the appliance. So, I started by creating a root CA with the key, then I generate the CSRs etc from the appliance. To ensure that the root CA that I created was valid, I copied it to /etc/ssl/certs and ran c_rehash /etc/ssl/certs, which then lists my new CA.
As it always goes, I made progress right after my last post.
After getting the error, it seems that the certificate is put in place, because if you browse to https://vcenter:7444/lookupservice/sdk, the correct certificate does appear. I then ran OpenSSL s_client to verify that the certificate is valid and this is what I got:
vCenter55:/ # openssl s_client -connect 192.168.33.128:7444 -status ### The command I ran the first time
CONNECTED(00000003)
OCSP response: no response sent
depth=1 /C=ZA/ST=Gauteng/L=Pretoria/O=company/OU=Certificate Authority/CN=company Root CA
verify error:num=19:self signed certificate in certificate chain ### Seems the appliance doesn't like the self-signed certificate
verify return:0
---
Certificate chain
0 s:/C=ZA/ST=Gauteng/O=company/OU=VMware vCenter Service Certificate/CN=vCenter55.company.co.za
i:/C=ZA/ST=Gauteng/L=Pretoria/O=company/OU=Certificate Authority/CN=company Root CA
1 s:/C=ZA/ST=Gauteng/L=Pretoria/O=company/OU=Certificate Authority/CN=company Root CA
i:/C=ZA/ST=Gauteng/L=Pretoria/O=company/OU=Certificate Authority/CN=company Root CA
---
Server certificate
-----BEGIN CERTIFICATE-----
MIID9jCCAt6gAwIBAgICAS.......FBQAwfjELMAkGA1UEBhMCWkEx
-----END CERTIFICATE-----
subject=/C=ZA/ST=Gauteng/O=company/OU=VMware vCenter Service Certificate/CN=vCenter55.company.co.za
issuer=/C=ZA/ST=Gauteng/L=Pretoria/O=company/OU=Certificate Authority/CN=company Root CA
---
No client certificate CA names sent
---
SSL handshake has read 2309 bytes and written 441 bytes
---
New, TLSv1/SSLv3, Cipher is AES256-SHA
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
SSL-Session:
Protocol : TLSv1
Cipher : AES256-SHA
Session-ID: 540771BD58058D6BD2F7C0B673A0D5740FC964C9179DC83DDA9EDA0BCAEB06C7
Session-ID-ctx:
Master-Key: 8BDD035D2FCB5645DECF21B5BB26B6C46C6A964DBD8B5E54EA4CEF1893B75E2D2C2C904E1162B808BA7BBD5CFDDEE22E
Key-Arg : None
Start Time: 1409774013
Timeout : 300 (sec)
Verify return code: 19 (self signed certificate in certificate chain) ### The return code 19, as seen above, is and error
---
vCenter55:/ # openssl s_client -connect 192.168.33.128:7444 -CApath /etc/ssl/certs ### This time I ran it while specifying the folder where my root CA is kept
CONNECTED(00000003)
depth=1 /C=ZA/ST=Gauteng/L=Pretoria/O=company/OU=Certificate Authority/CN=company Root CA
verify return:1 ### No error this time.
depth=0 /C=ZA/ST=Gauteng/O=company/OU=VMware vCenter Service Certificate/CN=vCenter55.company.co.za
verify return:1
---
Certificate chain
0 s:/C=ZA/ST=Gauteng/O=company/OU=VMware vCenter Service Certificate/CN=vCenter55.company.co.za
i:/C=ZA/ST=Gauteng/L=Pretoria/O=company/OU=Certificate Authority/CN=company Root CA
1 s:/C=ZA/ST=Gauteng/L=Pretoria/O=company/OU=Certificate Authority/CN=company Root CA
i:/C=ZA/ST=Gauteng/L=Pretoria/O=company/OU=Certificate Authority/CN=company Root CA
---
Server certificate
-----BEGIN CERTIFICATE-----
MIID9jCCAt6gAwIBAgI......wfjELMAkGA1UEBhMCWkEx
-----END CERTIFICATE-----
subject=/C=ZA/ST=Gauteng/O=company/OU=VMware vCenter Service Certificate/CN=vCenter55.company.co.za
issuer=/C=ZA/ST=Gauteng/L=Pretoria/O=company/OU=Certificate Authority/CN=company Root CA
---
No client certificate CA names sent
---
SSL handshake has read 2309 bytes and written 465 bytes
---
New, TLSv1/SSLv3, Cipher is AES256-SHA
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
SSL-Session:
Protocol : TLSv1
Cipher : AES256-SHA
Session-ID: 54077230E37AC53541373C907E213A8ED19EA02DF5EAFA47C28BF114DA3D68E1
Session-ID-ctx:
Master-Key: F8226AA2B758500D90B0137632F14752FB617E749577C7B4826CD541B1DE6D8BA8F4C3FA24CE59F734E8D5176D1F43AB
Key-Arg : None
Start Time: 1409774128
Timeout : 300 (sec)
Verify return code: 0 (ok) ### This time the cert is OK
Turning my attention to the 02-inventoryservice script...
Just some more information about where the issue seems to be. None of the below made a difference but I add it for the record.
Going through the 02-inventoryservice script, it reaches a point where it calls the vi_regtool, which is a Java application. When you get the message "Initializing registration provider" and "Getting SSL certificates for https://...", it is within in this Java application. More precisely, it is when it runs the command 'exec -a vi_regtool $JAVA_BIN "$LOG4J_CONF" $JAVA_OPTS -jar "$VI_REGTOOL_JAR" "$@"'.
Since Java has its own certificate store, I added my self-signed certificate into the Java cacerts store. I even created an intermediate CA to sign the server certs with and added this cert into the store too. On vCSA the Java JRE home is at /usr/java/jre-vmware. To add the CA to the Java store, run this command while withing the JRE HOME folder: bin/keytool -import -trustcacerts -alias MyRootCA -file RootCA.crt -keystore lib/security/cacerts. This adds the cert successfully, and even after rebooting the appliance, I still cannot run 02-inventoryservice to completion.
I get: server certificate assertion not verified and thumbprint not matched.
Return code is: SSLHandshakeFailed.
Andre
I realized that there are a number of things that I missed, so I wrote a script to automate the entire process. Testers wanted!
Andre's VMware Notes: Certificator
Andre Combrinck
Hello WTF,
I will go for the shameless self-branding :
http://bidabe.zapto.org/?p=316
I used this script on four vCSA so far => no problem !
This script automates SSL certificates signing for vCSA 5.5 and guide you through:
– Generate Certificate Signing Request (CSR)
– Request / Download Certificate
– Install new certificates.
It runs on vCenter Server Virtual Appliance 5.5 and is intended to work with Microsoft Internal CA.
It checks the system requirements: OpenSSL version, vCSA version, DNS Records and IP Address
SSH into your vCSA (puTTY) :
# vCSAFQDN:~ # vim /usr/bin/uptcrt
:set paste
(Paste the whole script from clipboard)
:wq
# chmod a+x /usr/bin/uptcrt
# uptcrt
I would be surprised if you still experience issues but please let me know.
There are a few things that can break your setup (notably encoding between Windows and Linux), this script will avoid that.
regards,
Florian Bidabé
Information Systems