Well, custom SSL certificates are still a pretty big mess. Since there is not much documentation on how to create the SSL certificates, I tried to follow the self-signed ones the installer creates as closely as possible.
Certificates with the following attributes worked for me on both the vCenter and the ESXi hosts:
Subject should contain the $FQDN as CN.
At least the following X509v3 extensions should be present:
X509v3 Basic Constraints:
CA:FALSE
X509v3 Key Usage:
Digital Signature, Non Repudiation, Key Encipherment
X509v3 Extended Key Usage:
TLS Web Client Authentication, TLS Web Server Authentication
X509v3 Subject Alternative Name:
DNS:$FQDN
The installer does not add Non Repudiation, but it does not seem to harm and our CA's policy requires it.
Another source of endless fun are intermediate CAs. ESXi does not deliver a certificate chain. If your vCenter does not find a valid path to a trusted Root CA, it won't accept the ESXi hosts certificates. Support for certificate chains is lacking in that much software, I started to add all our intermediate CAs to our base OS images as trusted root CAs.
Deploying the certificates on a vCenter turns out to be an adventure as well. With vSphere5 there is documentation on how to add custom certificates to Update Manager and vCenter server, but not to the Inventory Service or the Web Client. This leads the vSphere client to nag about unknown SSL certificates about every ten seconds. The way I work around this is to put the SSL certificates on the vCenter server in the right location before installing the software. The installer does not overwrite them but instead uses them. This means no issues with system status or hardware status.
For a 2008R2 vCenter with default installation paths, one should create the following directories containing the custom certificates both in PEM and PKCS12 format, the last one encrypted with "testpassword".
vCenter Server:
C:\ProgramData\VMware\VMware VirtualCenter\SSL
Update Manager:
C:\Program Files (x86)\VMware\Infrastructure\Update Manager\SSL
Inventory Service:
C:\Program Files\VMware\Infrastructure\Inventory Service\ssl
vSphere Web Client:
C:\Program Files\VMware\Infrastructure\vSphere Web Client\DMServer\config\ssl
At that point everything worked fine except FDM. I engaged support and got a workaround. After lots of debugging I have a hypothesis. If you add an ESXi server, that uses a certificate your vCenter OS trusts, there is no dialogue, that asks you to verify and accept the hosts SSL thumbprint. In that case VPX_HOST.EXPECTED_SSL_THUMPRINT does not get populated for that host, which is something FDM relies on.
The workaround is to add the right SSL thumprint to said cloumn for each host while the vCenter service is not running. I wrote a quick-and-dirty PS script which accomplishes this with as little downtime as possible. The script expects the user running it to be local administrator on the vCenter server and have SQL read/write access to the vCenter database.
Regards,
Thorsten
Fixed a typo
.
