VMware Cloud Community
WHISKEYTANG0F0X
Contributor
Contributor

Week long headache with CA signed certificates on VCSA 5.5

After a week of deploying VCSA 5.5 over and over again, probably 18 times, and seeing what must be every SSL related error possible, I am asking advice here for those who may have stumbled through this before. (Or know Linux)

On my last attempt I've installed fresh, configured the server, generated cert reqs through OpenSSL 0.9.8y, approved them through my internal CA, and attempted VMWare KB:

VMware KB: Configuring Certificate Authority (CA) signed certificates for vCenter Server Appliance 5...

All goes well this time until step 22: Run these commands to register the vCenter Inventory Service back to vCenter Single Sign-On:

cd /etc/vmware-sso/register-hooks.d

./02-inventoryservice --mode install --ls-server https://VCSA1.domain.com:7444/lookupservice/sdk --user administrator@vsphere.local --password sso_administrator_password


And I am rewarded with this:

Getting SSL certificates for https://VCSA1.mydomain.com:7444/lookupservice/sdk

com.vmware.vim.vmomi.core.exception.CertificateValidationException: Server certificate chain not verified

Return code is: SslHandshakeFailed

What gives?


The SSO service never gave me any errors when doing the cert install in previous steps. The FQDN is the same in the cert as it is on the VCSA on initial setup, and yes the same case.


VMWare KB yield nothing relevant as does a Google search.


HELP! I'm going crazy trying to figure this out.




Tags (5)
Reply
0 Kudos
10 Replies
WHISKEYTANG0F0X
Contributor
Contributor

Anyone help me out here?

I've worked through the KB article 8 times, and followed Derek Seaman's blog on this, but still always the same result.

Reply
0 Kudos
bedobash
Enthusiast
Enthusiast

I've had the same problem, but my issue is:

com.vmware.vim.vmomi.core.exception.CertificateValidationException: Server certificate assertion not verified and thumbprint not matched

Return code is: SslHandshakeFailed (when trying to unregister or reregister the Inventory Service after successfully replacing the vCenterSSO certificate).

Reply
0 Kudos
BorgOne
Contributor
Contributor

I believe that if you follow this article you will be able to replace your bad certs:

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=205722...

If you give it a try and is working please let me know. My feeling is that the ssl trust chain is broken and you will need to recreate it in a similar way.

Thank you.

Reply
0 Kudos
flynmooney
Enthusiast
Enthusiast

Have you read Derek Seaman's bog on SSL in ESX 5.5?  You can probably start here: http://www.derekseaman.com/2013/10/vsphere-5-5-install-pt-5-ssl-deep.html

Reply
0 Kudos
PhillyDiane
Contributor
Contributor

You can check the chain using openssl verify. In Windows the command would be:

openssl verify -CAfile C:\pathtopem\chain.pem -verbose  c:\pathtoCerfile\filename.cer

I don't know what the linux equivalent is.

When I ran into this issue, it was because the intermediate cert wasn't included in the chain. Open the cert and look at the sections and make sure your root cert is at the bottom and the vcenter cert is at the top.

Reply
0 Kudos
Andr3201110141
Enthusiast
Enthusiast

I am in the same boat as you are. I even posted a question to the editor of the article that after following the steps, I get the same issue. I have to be honest here that I didn't use a Microsoft CA nor a public CA, and ran all the OpenSSL commands directly on the appliance. So, I started by creating a root CA with the key, then I generate the CSRs etc from the appliance. To ensure that the root CA that I created was valid, I copied it to /etc/ssl/certs and ran c_rehash /etc/ssl/certs, which then lists my new CA.

Good design comes from intelligence.
Reply
0 Kudos
Andr3201110141
Enthusiast
Enthusiast

As it always goes, I made progress right after my last post.

After getting the error, it seems that the certificate is put in place, because if you browse to https://vcenter:7444/lookupservice/sdk, the correct certificate does appear. I then ran OpenSSL s_client to verify that the certificate is valid and this is what I got:

vCenter55:/ # openssl s_client -connect 192.168.33.128:7444 -status ### The command I ran the first time

CONNECTED(00000003)

OCSP response: no response sent

depth=1 /C=ZA/ST=Gauteng/L=Pretoria/O=company/OU=Certificate Authority/CN=company Root CA

verify error:num=19:self signed certificate in certificate chain ### Seems the appliance doesn't like the self-signed certificate

verify return:0

---

Certificate chain

0 s:/C=ZA/ST=Gauteng/O=company/OU=VMware vCenter Service Certificate/CN=vCenter55.company.co.za

   i:/C=ZA/ST=Gauteng/L=Pretoria/O=company/OU=Certificate Authority/CN=company Root CA

1 s:/C=ZA/ST=Gauteng/L=Pretoria/O=company/OU=Certificate Authority/CN=company Root CA

   i:/C=ZA/ST=Gauteng/L=Pretoria/O=company/OU=Certificate Authority/CN=company Root CA

---

Server certificate

-----BEGIN CERTIFICATE-----

MIID9jCCAt6gAwIBAgICAS.......FBQAwfjELMAkGA1UEBhMCWkEx

-----END CERTIFICATE-----

subject=/C=ZA/ST=Gauteng/O=company/OU=VMware vCenter Service Certificate/CN=vCenter55.company.co.za

issuer=/C=ZA/ST=Gauteng/L=Pretoria/O=company/OU=Certificate Authority/CN=company Root CA

---

No client certificate CA names sent

---

SSL handshake has read 2309 bytes and written 441 bytes

---

New, TLSv1/SSLv3, Cipher is AES256-SHA

Server public key is 2048 bit

Secure Renegotiation IS supported

Compression: NONE

Expansion: NONE

SSL-Session:

    Protocol  : TLSv1

    Cipher    : AES256-SHA

    Session-ID: 540771BD58058D6BD2F7C0B673A0D5740FC964C9179DC83DDA9EDA0BCAEB06C7

    Session-ID-ctx:

    Master-Key: 8BDD035D2FCB5645DECF21B5BB26B6C46C6A964DBD8B5E54EA4CEF1893B75E2D2C2C904E1162B808BA7BBD5CFDDEE22E

    Key-Arg   : None

    Start Time: 1409774013

    Timeout   : 300 (sec)

    Verify return code: 19 (self signed certificate in certificate chain) ### The return code 19, as seen above, is and error

---

vCenter55:/ # openssl s_client -connect 192.168.33.128:7444 -CApath /etc/ssl/certs ### This time I ran it while specifying the folder where my root CA is kept

CONNECTED(00000003)

depth=1 /C=ZA/ST=Gauteng/L=Pretoria/O=company/OU=Certificate Authority/CN=company Root CA

verify return:1 ### No error this time.

depth=0 /C=ZA/ST=Gauteng/O=company/OU=VMware vCenter Service Certificate/CN=vCenter55.company.co.za

verify return:1

---

Certificate chain

0 s:/C=ZA/ST=Gauteng/O=company/OU=VMware vCenter Service Certificate/CN=vCenter55.company.co.za

   i:/C=ZA/ST=Gauteng/L=Pretoria/O=company/OU=Certificate Authority/CN=company Root CA

1 s:/C=ZA/ST=Gauteng/L=Pretoria/O=company/OU=Certificate Authority/CN=company Root CA

   i:/C=ZA/ST=Gauteng/L=Pretoria/O=company/OU=Certificate Authority/CN=company Root CA

---

Server certificate

-----BEGIN CERTIFICATE-----

MIID9jCCAt6gAwIBAgI......wfjELMAkGA1UEBhMCWkEx

-----END CERTIFICATE-----

subject=/C=ZA/ST=Gauteng/O=company/OU=VMware vCenter Service Certificate/CN=vCenter55.company.co.za

issuer=/C=ZA/ST=Gauteng/L=Pretoria/O=company/OU=Certificate Authority/CN=company Root CA

---

No client certificate CA names sent

---

SSL handshake has read 2309 bytes and written 465 bytes

---

New, TLSv1/SSLv3, Cipher is AES256-SHA

Server public key is 2048 bit

Secure Renegotiation IS supported

Compression: NONE

Expansion: NONE

SSL-Session:

    Protocol  : TLSv1

    Cipher    : AES256-SHA

    Session-ID: 54077230E37AC53541373C907E213A8ED19EA02DF5EAFA47C28BF114DA3D68E1

    Session-ID-ctx:

    Master-Key: F8226AA2B758500D90B0137632F14752FB617E749577C7B4826CD541B1DE6D8BA8F4C3FA24CE59F734E8D5176D1F43AB

    Key-Arg   : None

    Start Time: 1409774128

    Timeout   : 300 (sec)

    Verify return code: 0 (ok) ### This time the cert is OK

Turning my attention to the 02-inventoryservice script...

Good design comes from intelligence.
Reply
0 Kudos
Andr3201110141
Enthusiast
Enthusiast

Just some more information about where the issue seems to be. None of the below made a difference but I add it for the record.

Going through the 02-inventoryservice script, it reaches a point where it calls the vi_regtool, which is a Java application. When you get the message "Initializing registration provider" and "Getting SSL certificates for https://...", it is within in this Java application. More precisely, it is when it runs the command 'exec -a vi_regtool $JAVA_BIN "$LOG4J_CONF" $JAVA_OPTS -jar "$VI_REGTOOL_JAR" "$@"'.

Since Java has its own certificate store, I added my self-signed certificate into the Java cacerts store. I even created an intermediate CA to sign the server certs with and added this cert into the store too. On vCSA the Java JRE home is at /usr/java/jre-vmware. To add the CA to the Java store, run this command while withing the JRE HOME folder: bin/keytool -import -trustcacerts -alias MyRootCA -file RootCA.crt -keystore lib/security/cacerts. This adds the cert successfully, and even after rebooting the appliance, I still cannot run 02-inventoryservice to completion.

I get: server certificate assertion not verified and thumbprint not matched.

Return code is: SSLHandshakeFailed.

Andre

Good design comes from intelligence.
Reply
0 Kudos
Andr3201110141
Enthusiast
Enthusiast

I realized that there are a number of things that I missed, so I wrote a script to automate the entire process. Testers wanted!

Andre's VMware Notes: Certificator

Andre Combrinck

Good design comes from intelligence.
Reply
0 Kudos
FlorianBidabe
Contributor
Contributor

Hello WTF,

I will go for the shameless self-branding :

http://bidabe.zapto.org/?p=316

I used this script on four vCSA so far => no problem !

This script automates SSL certificates signing for vCSA 5.5 and guide you through:

– Generate Certificate Signing Request (CSR)

– Request / Download Certificate

– Install new certificates.

It runs on vCenter Server Virtual Appliance 5.5 and is intended to work with Microsoft Internal CA.

It checks the system requirements: OpenSSL version, vCSA version, DNS Records and IP Address

SSH into your vCSA (puTTY) :

# vCSAFQDN:~ # vim /usr/bin/uptcrt

:set paste

(Paste the whole script from clipboard)

:wq

# chmod a+x /usr/bin/uptcrt

# uptcrt

I would be surprised if you still experience issues but please let me know.

There are a few things that can break your setup (notably encoding between Windows and Linux), this script will avoid that.

regards,

Florian Bidabé

Information Systems

Reply
0 Kudos