Fix CSR commonname or set VMCA to Custom Certificate Authority

[marcussf]

I am maintaining a few ESXi hosts not connected to vCenter. Some of those are native 6.5 installations while others have been upgraded from 6.0 to 6.5.

Upgrading to 6.5 build 16576891 apparently broke the certificates on hosts that originally ran 6.0. /etc/vmware/ssl/castore.pem was zapped, i.e. had 0 bytes after the upgrade which caused a few problems like sfcbd not starting. I fixed that by running /sbin/generate-certificates and restarting sfcbd-watchdog and rhttpproxy. While that seems to have restored all services, the web UI CSR generation now uses the hosts IPv4 address as CSR commonname. Asking for the hostname in an SSH session reveals the correct name I would like to see in the CSR request.

That again, and the fact that the web UI is now HSTS-enabled, causes the issue that browsers (rightfully) complain about an insecure connection to the web UI. Specifically Firefox will not even allow an exception.

I am seeking your advice on the best way to fix this. I am happy to either fix the setting that causes the IPv4-based commonname or set the VMCA mode to Custom Certificate Authority and replace rui.key and rui.crt myself. Concerning the latter I was only able to find instructions on how to set it Through vCenter unfortunately.

[Lalegre]

Hey marcussf​,

Do you have the DNS suffix and hostname configuration if you login into the DCUI? If not i would suggest to populate it and try again.

I am not sure about what you are talking here: web UI CSR generation

What you can do is delete the rui.crt and rui.keyand try to run the /sbin/generate-certificates again. Also please take a look at the next KB which referes to the castore.pem file: VMware Knowledge Base

[marcussf]

Hi @Lalegre,   

the hostname and DNS suffix is correct in both the DCUI and when issueing hostname in an SSH session.

When talking about web UI CSR generation I was referring to the ESXi web admin page, specifically the 'Import new certificate' -> 'Generate IP signing request' function on https://esxihost.fqdn/ui/#/host/manage/security/certificates

I am afraid re-running /sbin/generate-certificates did nothing. Here's the analysis of the CSR:

# openssl req -text -noout -verify -in esxihost.fqdn.csr

Certificate Request:

    Data:

        Version: 0 (0x0)

        Subject: C=US, ST=California, L=Palo Alto, O=VMware, Inc, OU=VMware ESX Server Default Certificate/ema

ilAddress=ssl-certificates@vmware.com, CN=212.nnn.nnn.nnn/unstructuredName=1576589792,564d7761726520496e632e

        Subject Public Key Info:

            Public Key Algorithm: rsaEncryption

                Public-Key: (2048 bit)

[..]

As you can see the commonname is the IPv4 of the host and not the FQDN and as a result the cert is issued for the IP, not the hostname. On an IPv6-only ESXI host the CN is 0.0.0.0 by the way.

[Lalegre]

Hey marcussf​,

I perfectly got it what you are facing and the only i could think is about a corruption on maybe some files that are inside the ESXi for regenerate the certificates. From here with my point of view i can give you some options:

• Run the ISO of the ESXi and re-install the ESXi but preserving the VMFS.
• Reset the ESXi so factory (If possible)
• Regenerate the self-signed certificates using OpenSSL (If you do not care about having these certificates i would go with this one)

[marcussf]

Hi,

I would favour option 3, but for that I would need to set the VMCA to be in in 'Custom' mode. Do you know how to reconfigure the VMCA without vCenter?

Thanks!

[Lalegre]

Hey,

I understood that this ESXi that you are updating the certificate is outside the vCenter Server and is standalone. Configuring VMCA in Custom Mode is for using Custom CA Certificates on vCenter Server and your ESXi hosts (The ones connected to it)

Is this your actual scenario?

[marcussf]

That's correct.

In the meantime I have successfully replaced rui.key and rui.crt. I am unsure though if future updates may break the cert again. It is my impression that Custom Mode VMCA will prevent that even for standalone ESXi hosts.

[Lalegre]

Setting the VMCA to Custom mode is for trusting the ESXi with custom CA certificates signed by a Custom CA. So in the future if you want to work this way you will need to switch to this mode and after that join your ESXi into vCenter.

If you do not do this and keep the default then your ESXi hosts will no work inside vCenter Server as the SSL handshake fails because the CA is not trusted.

Using ESXi standalone is completely different because by default the certificates are self-signed and not managed by any CA (Not VMCA nor Custom CA)

[marcussf]

I am fine with using my own CA to sign the certificates. But is it fine to use a standalone ESXi without Custom Mode VMCA or will future build updates break my certs?

[Lalegre · Accepted solution]

I think you are getting confused and that is what i was trying to explain.

VMCA is the Certification Authority that comes embedded in vCenter Server as a service since version 6.0 so if you are using the ESXi Standalone you do not have VMCA and your certificates could be or self-signed or Custom CA (Microsoft CA, Public CA, etc) but NOT the VMCA because does not exist.

So if you decide to use Custom CA certificates like from your private Microsoft CA it is fine but in the future if you install vCenter Server, before importing the ESXi inside the vCenter Inventory you will need to also generate custom CA certificates to vCenter Server and to change from "vmca" mode in vCenter to "custom" mode.

• "vmca" mode means that the VMCA embeded in vCenter will issue certificates for all the ESXi joined into vCenter.
• "custom" mode means that an external CA (e.g Microsoft CA) will be used for trust the certificates of the ESXis.

Basically if you change right now the certificate on your ESXi Standalone and in the future you join to vCenter Server using the "vmca" custom mode, the SSL certificate of the ESXi will be replaced with one signed by the VMCA. On the opposite if you join the ESXi into vCenter Server using the "custom" mode it will maintain the already replaced certificate in the ESXi and will verify its validity which the External CA you have configured.

Now from my experience and point of view, in most of the cases is not needed to use Custom CA certificates in your ESXi as they are already in a secure handshake with the vCenter Server however if you try to login to the UI of the Standalone ESXi you will see that is not-trusted but your browser but if there is not a strict policy on your company i would suggest to go with this options as you do not needed to remember to change your certificates for all your ESXis in the future and you will avoid more points of failure. Also there are some KBs related to using Custom CA Certificates with vSAN that shows a lot of issues.

Finally i recommend you to read the next link which is short but will explain you in detail the difference between the SSL modes: Certificate Mode Switch Workflows

If you have any more thoughts just let me know!

[marcussf]

Thanks for all your help!

While the issue is resolved, I suspect that auto-generated certs will become a bigger issue as browsers start to be more strict about them with Firefox being on the forefront. Specifically with HSTS I do not think there's an alternative to installing official certs or certs signed by your own CA. But that's something for the VMWare devs to think about.