VMware Cloud Community
mc1903cae
Enthusiast
Enthusiast
Jump to solution

vCenter 6.7 / VMCA as a Subordinate CA / Incomplete certification path on ESXi 6.7 hosts; but ESXi 6.5 hosts work OK.

VCSA with Embedded PSC v6.7 (Build 9451876)

VMCA configured as Subordinate CA to a Windows 2012 R2 Enterprise Root CA. (SHA256 Hash / 2048 bit Key)

VMCA replaces the SSL certificate on a ESXi v6.5 (Build 5969303) host and the 'certification path' is complete. All works as expected, no browser errors.

VMCA replaces the SSL certificate on a ESXi v6.7 (Build 8169922) host and the 'certification path' is incomplete. Still get the standard browser errors. The root CA and VMCA certificates are NOT in the path, only the ESXi host certificate!

ESXi v6.5 Host - Complete Certification Path.
Good SSL Certificate- ESXi 6.5 Host.png

A dump of the SSL connection using the TestSSLServer utility (GitHub - pornin/TestSSLServer ) shown below.

Connection: mc-esxi-v-204.momusconsulting.com:443

SNI: mc-esxi-v-204.momusconsulting.com

  TLSv1.0:

     server selection: uses client preferences

     3-- (key:  RSA) RSA_WITH_AES_128_CBC_SHA

     3-- (key: RSA)  RSA_WITH_AES_256_CBC_SHA

     3f- (key: RSA) ECDHE_RSA_WITH_AES_128_CBC_SHA

     3f- (key: RSA) ECDHE_RSA_WITH_AES_256_CBC_SHA

  TLSv1.1: idem

  TLSv1.2:

     server selection: enforce server preferences

     3f- (key: RSA) ECDHE_RSA_WITH_AES_256_GCM_SHA384

     3f- (key: RSA) ECDHE_RSA_WITH_AES_128_GCM_SHA256

     3-- (key: RSA)  RSA_WITH_AES_256_GCM_SHA384

     3-- (key: RSA)  RSA_WITH_AES_128_GCM_SHA256

     3f- (key: RSA) ECDHE_RSA_WITH_AES_256_CBC_SHA384

     3f- (key: RSA) ECDHE_RSA_WITH_AES_256_CBC_SHA

     3f- (key: RSA) ECDHE_RSA_WITH_AES_128_CBC_SHA256

     3f- (key: RSA) ECDHE_RSA_WITH_AES_128_CBC_SHA

     3-- (key: RSA)  RSA_WITH_AES_256_CBC_SHA256

     3-- (key: RSA)  RSA_WITH_AES_256_CBC_SHA

     3-- (key: RSA)  RSA_WITH_AES_128_CBC_SHA256

     3-- (key: RSA)  RSA_WITH_AES_128_CBC_SHA

=========================================

+++++ SSLv3/TLS: 1 certificate chain(s)

+++ chain: length=3

names match:        yes

includes root:      yes

signature hash(es): SHA-256

+ certificate order: 0

thumprint:  A18830247B90395EE003D706CE3AEB3CDA96BC6D

serial:     E032A1675443F48D

subject: EMAILADDRESS=admin@momusconsulting.com,CN=mc-esxi-v-204.momusconsulting.com,OU=Momus Labs,O=Momus Consulting,L=Basingstoke,ST=Basingstoke,C=GB

issuer:     CN=VMCA-mc-vcsa-v-204,OU=Momus Labs,O=Momus Consulting,L=Basingstoke,ST=Hampshire,C=GB

valid from: 2018-10-06 14:22:12 UTC

valid to:   2020-10-05 12:06:47 UTC

key type:   RSA

key size:   2048

sign hash:  SHA-256

server names:

   mc-esxi-v-204.momusconsulting.com

+ certificate order: 1

thumprint:  6313EF9061D1ED748298F0DB7D693F6CC2099046

serial:     5D0000000BA3C47E6295F579B400000000000B

subject:    CN=VMCA-mc-vcsa-v-204,OU=Momus Labs,O=Momus Consulting,L=Basingstoke,ST=Hampshire,C=GB

issuer:     CN=Momus Root CA on mc-addc-v-101,DC=momusconsulting,DC=com

valid from: 2018-10-06 12:06:47 UTC

valid to:   2020-10-05 12:06:47 UTC

key type:   RSA

key size:   2048

sign hash:  SHA-256

+ certificate order: 2

thumprint:  A3BD98D6B6C712A510E11669A84D0571C2D2F0F1

serial:     65F1DEEF09DD1A9A436075662D731F0F

subject:    CN=Momus Root CA on mc-addc-v-101,DC=momusconsulting,DC=com

issuer:     CN=Momus Root CA on mc-addc-v-101,DC=momusconsulting,DC=com

valid from: 2018-10-05 15:11:29 UTC

valid to:   2028-10-05 15:21:28 UTC

key type:   RSA

key size:   2048

sign hash:  SHA-256

(self-issued)

=========================================

Server compression support: no

Server sends a random system time.

Secure renegotiation support: yes

Encrypt-then-MAC support (RFC 7366): no

SSLv2 ClientHello format (for SSLv3+): yes

Minimum EC size (no extension):   256

Minimum EC size (with extension): 256

ECDH parameter reuse:  no

Supported curves (size and name) ('*' = selected by server):

  * 256 secp256r1 (P-256)

=========================================

  WARN[CS006]: Server supports cipher suites with no forward secrecy.

ESXi v6.7 Host - Incomplete Certification Path.
Bad SSL Certificate - ESXi 6.7 Host.png

Again, a dump of the SSL connection is shown below.

Connection: mc-esxi-v-205.momusconsulting.com:443

SNI: mc-esxi-v-205.momusconsulting.com

  TLSv1.2:

     server selection: enforce server preferences

     3f- (key: RSA) ECDHE_RSA_WITH_AES_256_GCM_SHA384

     3f- (key: RSA) ECDHE_RSA_WITH_AES_128_GCM_SHA256

     3-- (key: RSA)  RSA_WITH_AES_256_GCM_SHA384

     3-- (key: RSA)  RSA_WITH_AES_128_GCM_SHA256

     3f- (key: RSA) ECDHE_RSA_WITH_AES_256_CBC_SHA384

     3f- (key: RSA) ECDHE_RSA_WITH_AES_256_CBC_SHA

     3f- (key: RSA) ECDHE_RSA_WITH_AES_128_CBC_SHA256

     3f- (key: RSA)  ECDHE_RSA_WITH_AES_128_CBC_SHA

     3-- (key: RSA)  RSA_WITH_AES_256_CBC_SHA256

     3-- (key: RSA)  RSA_WITH_AES_256_CBC_SHA

     3-- (key: RSA)  RSA_WITH_AES_128_CBC_SHA256

     3-- (key: RSA)  RSA_WITH_AES_128_CBC_SHA

=========================================

+++++ SSLv3/TLS: 1 certificate chain(s)

+++ chain: length=1

names match:        yes

includes root:      no

signature hash(es): SHA-256

+ certificate order: 0

thumprint:  9CB7BEC3BD58491A36069B182093F22BE9813042

serial:     FD682ECC9662D00C

subject: EMAILADDRESS=admin@momusconsulting.com,CN=mc-esxi-v-205.momusconsulting.com,OU=Momus Labs,O=Momus Consulting,L=Basingstoke,ST=Basingstoke,C=GB

issuer:     CN=VMCA-mc-vcsa-v-204,OU=Momus Labs,O=Momus Consulting,L=Basingstoke,ST=Hampshire,C=GB

valid from: 2018-10-06 14:44:04 UTC

valid to:   2020-10-05 12:06:47 UTC

key type:   RSA

key size:   2048

sign hash:  SHA-256

server names:

   mc-esxi-v-205.momusconsulting.com

=========================================

Server compression support: no

Server sends a random system time.

Secure renegotiation support: yes

Encrypt-then-MAC support (RFC 7366): no

SSLv2 ClientHello format (for SSLv3+): yes

Minimum EC size (no extension):   256

Minimum EC size (with extension): 256

ECDH parameter reuse:  no

Supported curves (size and name) ('*' = selected by server):

  * 256 secp256r1 (P-256)

=========================================

  WARN[CS006]: Server supports cipher suites with no forward secrecy.

Any ideas?

Thanks

M

1 Solution

Accepted Solutions
mc1903cae
Enthusiast
Enthusiast
Jump to solution

Found out how to fix it; but I cannot take the glory..! That goes to to @weg0t0eleven on Reddit (https://www.reddit.com/r/vmware/comments/8z4zal/certificate_chain_on_esxi_nodes/)

On each ESXi 6.7 / 6.7 U1 host, edit the /etc/vmware/rhttpproxy/config.xml file to remove the <!-- and --> comment tags for the <keyStoreFile> line (approx line 77).

Save the file and reboot the host.

/etc/vmware/rhttpproxy/config.xml

   <!-- Remove the following node to disable SSL -->

   <ssl>

      <!-- The server private key file -->

      <privateKey>/etc/vmware/ssl/rui.key</privateKey>

      <!-- The server side certificate file -->

      <certificate>/etc/vmware/ssl/rui.crt</certificate>

      <!-- Client-side CAFile verify location -->

      <!-- <keyStoreFile>/etc/vmware/ssl/castore.pem</keyStoreFile> -->

   </ssl>

   <!-- Remove the following node to disable SSL -->

   <ssl>

      <!-- The server private key file -->

      <privateKey>/etc/vmware/ssl/rui.key</privateKey>

      <!-- The server side certificate file -->

      <certificate>/etc/vmware/ssl/rui.crt</certificate>

      <!-- Client-side CAFile verify location -->

      <keyStoreFile>/etc/vmware/ssl/castore.pem</keyStoreFile>

   </ssl>

Once you have made this change on a 6.7 host you can upgrade to 6.7 U1 and the certificate chain remains complete.

If you upgrade a 6.5 host to 6.7/6.7U1 you will need to do make this change & reboot after the upgrade has completed.

Happy days.

View solution in original post

11 Replies
gorciakj
Contributor
Contributor
Jump to solution

I've come across the same issue.  I updated my ESXi 6.7 host to 6.7.0, 10302608 and renewed the certificate and am still seeing the same behavior.

mc1903cae
Enthusiast
Enthusiast
Jump to solution

gorciakj

It's good to know that I am not alone! :smileygrin:

What is your vCenter version/build/platform?

I haven't had a chance to test it with a 6.7 U1 VCSA and 6.7 U1 ESXi Host.

M

Reply
0 Kudos
gorciakj
Contributor
Contributor
Jump to solution

Also running VCSA 6.7 (Build 9433931)

VMCA also configured as a subordinate to Server 2012 R2 CA

The ESXi 6.7 host I updated was to 6.7U1, have not tried updating my VCSA yet.

All my 6.5 hosts have a complete certificate chain just as you've mentioned as well.

mc1903cae
Enthusiast
Enthusiast
Jump to solution

Thanks again gorciakj

I will try upgrading my test system at the weekend to see if a 6.7U1 VCSA makes any difference!

I will post what I find up on this thread.

M

Reply
0 Kudos
mc1903cae
Enthusiast
Enthusiast
Jump to solution

gorciakj I found some time today to test this with 6.7 U1 VCSA. Sadly the issue is still there for the 6.7 U1 ESXi hosts!

VCSA with Embedded PSC v6.7 U1 (Build 10244745) - Fresh install NOT an upgrade.

VMCA configured as Subordinate CA to a Windows 2012 R2 Enterprise Root CA. (SHA256 Hash / 2048 bit Key)

:smileycheck: VMCA replaces the SSL certificate on a ESXi v6.5 U1 (Build 5969303) host and the 'certification path' is complete. All works as expected, no browser errors.

:smileycheck: VMCA replaces the SSL certificate on a ESXi v6.5 U2 (Build 8294253) host and the 'certification path' is complete. All works as expected, no browser errors.

:smileyx: VMCA replaces the SSL certificate on a ESXi v6.7 U1 (Build 10302608) host and the 'certification path' is incomplete. Still get the standard browser errors. The root CA and VMCA certificates are NOT in the path, only the ESXi host certificate!

I don't have any support, so I cannot raise a SR to take this any further.

M

Reply
0 Kudos
mc1903cae
Enthusiast
Enthusiast
Jump to solution

Found out how to fix it; but I cannot take the glory..! That goes to to @weg0t0eleven on Reddit (https://www.reddit.com/r/vmware/comments/8z4zal/certificate_chain_on_esxi_nodes/)

On each ESXi 6.7 / 6.7 U1 host, edit the /etc/vmware/rhttpproxy/config.xml file to remove the <!-- and --> comment tags for the <keyStoreFile> line (approx line 77).

Save the file and reboot the host.

/etc/vmware/rhttpproxy/config.xml

   <!-- Remove the following node to disable SSL -->

   <ssl>

      <!-- The server private key file -->

      <privateKey>/etc/vmware/ssl/rui.key</privateKey>

      <!-- The server side certificate file -->

      <certificate>/etc/vmware/ssl/rui.crt</certificate>

      <!-- Client-side CAFile verify location -->

      <!-- <keyStoreFile>/etc/vmware/ssl/castore.pem</keyStoreFile> -->

   </ssl>

   <!-- Remove the following node to disable SSL -->

   <ssl>

      <!-- The server private key file -->

      <privateKey>/etc/vmware/ssl/rui.key</privateKey>

      <!-- The server side certificate file -->

      <certificate>/etc/vmware/ssl/rui.crt</certificate>

      <!-- Client-side CAFile verify location -->

      <keyStoreFile>/etc/vmware/ssl/castore.pem</keyStoreFile>

   </ssl>

Once you have made this change on a 6.7 host you can upgrade to 6.7 U1 and the certificate chain remains complete.

If you upgrade a 6.5 host to 6.7/6.7U1 you will need to do make this change & reboot after the upgrade has completed.

Happy days.

wagnewal
Contributor
Contributor
Jump to solution

Thanks for the very good explanation how to solve it, I also had to fight with the same problem after I enrolled a new VMCA certificate!

I finally fixed it with a find and replace "sed" command and "cssh" in paralell on many affected hosts in one step:

cd /etc/vmware/rhttpproxy/

grep -q "<version>6.6.0.0</version>" config.xml && sed -i.backup "/Client-side\ CAFile\ verify\ location/ {n;s/<\!\-\- \(.*\) \-\->/\1/}" config.xml

/etc/init.d/rhttpproxy restart

Sed creates a backup(config.xml.backup) of the original file in the same directory.

VMware posted this KB entry regarding the same issue:

VMware Knowledge Base

Reply
0 Kudos
mc1903cae
Enthusiast
Enthusiast
Jump to solution

wagnewal​ nice, very nice..! Thank you for sharing!

It's also good to see that VMware now have a KB for this issue; fingers crossed for a fix in an 6.7 U2.

Cheers

M

Reply
0 Kudos
franckehret
Enthusiast
Enthusiast
Jump to solution

Hi there,

Thank you so much, worked for me as well (6.7 U1 - latest patches) ! Smiley Happy

mc1903cae
Enthusiast
Enthusiast
Jump to solution

Reading the ESXi v6.7 Update 2 release notes it looks like this issue has been fixed. I haven't tested it myself yet.

VMware ESXi 6.7 Update 2 Release Notes

PR 2212140: Renewing a host certificate might not push the full chain of trust to the ESXi host

When you renew a certificate, only the first certificate from the supplied chain of trust might be stored on the ESXi host. Any intermediate CA certificates are truncated. Because of the missing certificates, the chain to the root CA cannot be built. This leads to a warning for an untrusted connection.

This issue is resolved in this release.

Reply
0 Kudos
mc1903cae
Enthusiast
Enthusiast
Jump to solution

I have been meaning to update this for a while...

Whatever @VMware fixed with PR2212140 in the ESXi v6.7 U2 release it was not this!

This issue persists with ESXi v6.7 U2 (Build 13006603) and also with the recent release of ESXi v6.7 U3 (Build 14320388).

Editing the /etc/vmware/rhttpproxy/config.xml file as per the posts above still works.

M

Reply
0 Kudos