topic Re: vCenter fails to start after certificate replacement in vCenter™ Server Discussions

vCenter fails to start after certificate replacement

snikers — Thu, 19 Dec 2019 17:32:21 GMT

Using vcenter 6.7 Administration - > Certificates have added root CA certificate of Letsencrypt and replaced Machine certificate with signed one provide certificate and key

After reboot vcenter doesn`t start anymore:

2019-12-19T17:22:23.429Z info vpxd[05606] [Originator@6876 sub=ThreadPool] Entering worker thread loop
2019-12-19T17:22:23.430Z info vpxd[05605] [Originator@6876 sub=ThreadPool] Thread enlisted
2019-12-19T17:22:23.430Z info vpxd[05605] [Originator@6876 sub=ThreadPool] Entering worker thread loop
2019-12-19T17:22:23.459Z error vpxd[05321] [Originator@6876 sub=Main opID=CheckCertificateExpiry-6058ed8] Unable to get certificate count for APPLMGMT_PASSWORD from VECS localhost, error: 0
2019-12-19T17:22:23.548Z info vpxd[05332] [Originator@6876 sub=ThreadPool] Spawning additional worker - allocated: 144, idle: 19
2019-12-19T17:22:23.553Z info vpxd[05617] [Originator@6876 sub=ThreadPool] Thread enlisted
2019-12-19T17:22:23.553Z info vpxd[05617] [Originator@6876 sub=ThreadPool] Entering worker thread loop
2019-12-19T17:22:23.572Z warning vpxd[05113] [Originator@6876 sub=LSClient] Caught exception while getting service with Id :e2136204-f25b-4a2b-a5ac-67b473cfd253. N7Vmacore9ExceptionE(Cannot initialize service registration stub)
--> [context]zKq7AVECAAAAAGC34QAOdnB4ZAAA4AArbGlidm1hY29yZS5zbwAAWCUbABWWGAH5kWV2cHhkAAHOlWUB9qFlASkvoAIqhQJsaWJhdXRoemNsaWVudC5zbwABvdeeAToJVAGKaFQBGcZSA5AFAmxpYmMuc28uNgABpb5S[/context]
2019-12-19T17:22:23.573Z warning vpxd[05113] [Originator@6876 sub=LSClient] Caught exception while retrieve endpoint. N7Vmacore9ExceptionE(Cannot initialize service registration stub)
--> [context]zKq7AVECAAAAAGC34QAPdnB4ZAAA4AArbGlidm1hY29yZS5zbwAAWCUbABWWGAH5kWV2cHhkAAE4l2UBKJllASuiZQEpL6ACKoUCbGliYXV0aHpjbGllbnQuc28AAb3XngE6CVQBimhUARnGUgOQBQJsaWJjLnNvLjYAAaW+Ug==[/context]
2019-12-19T17:22:23.574Z warning vpxd[05113] [Originator@6876 sub=LSClient] endpoint not found for Product: com.vmware.cis, Type: cs.inventory
2019-12-19T17:22:23.574Z warning vpxd[05113] [Originator@6876 sub=LSClient] Caught exception while getting service with Id :e2136204-f25b-4a2b-a5ac-67b473cfd253. N7Vmacore9ExceptionE(Cannot initialize service registration stub)
--> [context]zKq7AVECAAAAAGC34QAOdnB4ZAAA4AArbGlidm1hY29yZS5zbwAAWCUbABWWGAH5kWV2cHhkAAHOlWUB9qFlASkvoAI3hQJsaWJhdXRoemNsaWVudC5zbwABvdeeAToJVAGKaFQBGcZSA5AFAmxpYmMuc28uNgABpb5S[/context]
2019-12-19T17:22:23.575Z warning vpxd[05113] [Originator@6876 sub=LSClient] Caught exception while retrieve endpoint. N7Vmacore9ExceptionE(Cannot initialize service registration stub)
--> [context]zKq7AVECAAAAAGC34QAPdnB4ZAAA4AArbGlidm1hY29yZS5zbwAAWCUbABWWGAH5kWV2cHhkAAE4l2UBKJllASuiZQEpL6ACN4UCbGliYXV0aHpjbGllbnQuc28AAb3XngE6CVQBimhUARnGUgOQBQJsaWJjLnNvLjYAAaW+Ug==[/context]
2019-12-19T17:22:23.597Z warning vpxd[05113] [Originator@6876 sub=LSClient] endpoint not found for Product: com.vmware.cis, Type: cs.inventory
2019-12-19T17:22:23.718Z warning vpxd[05113] [Originator@6876 sub=VpxdAuthClient] [ConnectAndLogin] Failed to loginBySamlToken: N7Vmacore3Ssl18SSLVerifyExceptionE(SSL Exception: Verification parameters:
--> PeerThumbprint: 49:68:90:15:2C:75:C6:7C:C7:B4:55:EB:87:E2:E6:29:92:21:A8:72
--> ExpectedThumbprint:
--> ExpectedPeerName: localhost
--> The remote host certificate has these problems:
-->
--> * Host name does not match the subject name(s) in certificate.)
--> [context]zKq7AVECAAAAAGC34QANdnB4ZAAA4AArbGlidm1hY29yZS5zbwAAWCUbAP6dGACeQCIAaXEiABtFIgDTSSIAOaIjAHFvIwA6ciMAnVYrAdRzAGxpYnB0aHJlYWQuc28uMAAC3Y4ObGliYy5zby42AA==[/context]
2019-12-19T17:22:23.719Z info vpxd[05113] [Originator@6876 sub=VpxdAuthClient] fallback to loginByCertificate
2019-12-19T17:22:23.729Z error vpxd[05113] [Originator@6876 sub=ServerAccess] Remote login failed: N7Vmacore3Ssl18SSLVerifyExceptionE(SSL Exception: Verification parameters:
--> PeerThumbprint: 49:68:90:15:2C:75:C6:7C:C7:B4:55:EB:87:E2:E6:29:92:21:A8:72
--> ExpectedThumbprint:
--> ExpectedPeerName: localhost
--> The remote host certificate has these problems:
-->
--> * Host name does not match the subject name(s) in certificate.)

When resetting certificates using /usr/lib/vmware-vmca/bin/certificate-manager it works again

There is no ESXi host connected to vCenter just in case...

Re: vCenter fails to start after certificate replacement

T180985 — Thu, 19 Dec 2019 19:00:20 GMT

Looks like the cert is incorrectly configured

* Host name does not match the subject name(s) in certificate.

Re: vCenter fails to start after certificate replacement

Vijay2027 — Thu, 19 Dec 2019 23:56:42 GMT

Run the below commands and make sure all 3 gives you hostnmae of vCSA

1. PNID of the vCenter server: # /usr/lib/vmware-vmafd/bin/vmafd-cli get-pnid --server-name localhost

2. Hostname of vCenter server: # hostname -f

3. And SAN (Subject Alternative Name) field of machine ssl cert: # openssl x509 -in machine.cer -noout -text | grep DNS:

Re: vCenter fails to start after certificate replacement

meanevo — Sun, 19 Jan 2020 18:21:51 GMT

same here... cant figure out why

tail -f /var/log/vmware/vpxd/vpxd.log

--> ExpectedThumbprint:
--> ExpectedPeerName: localhost
--> The remote host certificate has these problems:
-->
--> * Host name does not match the subject name(s) in certificate.)

the following command gives same result

/usr/lib/vmware-vmafd/bin/vmafd-cli get-pnid --server-name localhost

/usr/lib/vmware-vmafd/bin/vecs-cli entry list --store MACHINE_SSL_CERT --text | grep -A1 Alternative

openssl x509 -in <path_to_certificate_file> -noout -text | grep -A1 Alternative

hostname -f

Re: vCenter fails to start after certificate replacement

slairipp — Wed, 13 May 2020 02:51:09 GMT

Did you ever find a fix for this issue? Having the exact same issue here.

Re: vCenter fails to start after certificate replacement

mohitsharma444 — Sun, 17 May 2020 05:25:38 GMT

I had the same issue for past few weeks. Updated to 6.7.0.44000 and looks like this is resolved. I've run my playbook for renewing letsencrypt certificates a bunch of times, rebooted vcsa and everything seems to be stable so far.

Re: vCenter fails to start after certificate replacement

Virtbay — Tue, 19 May 2020 18:40:47 GMT

It could be also due to duplicate certificate in the trusted root store . Try running the below command and match the serial numbers . If you find duplicate serial numbers then you would have to remove them.

/usr/lib/vmware-vmafd/bin/vecs-cli entry list --store TRUSTED_ROOTS --text| grep -A 6 -i alias |less

Re: vCenter fails to start after certificate replacement

jgrade82 — Thu, 18 Feb 2021 22:22:54 GMT

Was this fixed??

I am facing the same issue. Followed the thread but nothing seems to work

Re: vCenter fails to start after certificate replacement

kursanthoy — Thu, 19 May 2022 18:40:21 GMT

vcenter 6.7.0.52000

have the same issue after renewal letencrypt certificate:

vcenter.yyy.com

# /usr/lib/vmware-vmafd/bin/vmafd-cli get-pnid --server-name localhost
vcenter.yyy.com
# hostname -f
vcenter.yyy.com
# /usr/lib/vmware-vmafd/bin/vecs-cli entry list --store MACHINE_SSL_CERT --text | grep -A1 Alternative
X509v3 Subject Alternative Name:
DNS:vcenter.yyy.com

--> PeerThumbprint: 98:FE:16:42:E3:CF:43:2B:63:C5:9D:79:9C:77:FB:BD:B2:2A:07:FA
--> ExpectedThumbprint:
--> ExpectedPeerName: localhost
--> The remote host certificate has these problems:
-->
--> * Host name does not match the subject name(s) in certificate.)
--> [context]zKq7AVECAAAAAAt9JgENdnB4ZAAAPFYrbGlidm1hY29yZS5zbwAAJEUbAFqxGAAehyIA9b4iAJuLIgBTkCIAie8jAMG8IwCKvyMA+asrAdRzAGxpYnB0aHJlYWQuc28uMAAC7Y8ObGliYy5zby42AA==[/context]
2022-05-19T13:40:08.483Z info vpxd[36481] [Originator@6876 sub=VpxdAuthClient] fallback to loginByCertificate
2022-05-19T13:40:08.487Z error vpxd[36481] [Originator@6876 sub=ServerAccess] Remote login failed: N7Vmacore3Ssl18SSLVerifyExceptionE(SSL Exception: Verification parameters:
--> PeerThumbprint: 98:FE:16:42:E3:CF:43:2B:63:C5:9D:79:9C:77:FB:BD:B2:2A:07:FA
--> ExpectedThumbprint:
--> ExpectedPeerName: localhost
--> The remote host certificate has these problems:
-->
--> * Host name does not match the subject name(s) in certificate.)
--> [context]zKq7AVECAAAAAAt9JgENdnB4ZAAAPFYrbGlidm1hY29yZS5zbwAAJEUbAFqxGAAehyIA9b4iAJuLIgBTkCIAie8jAMG8IwCKvyMA+asrAdRzAGxpYnB0aHJlYWQuc28uMAAC7Y8ObGliYy5zby42AA==[/context]
2022-05-19T13:40:08.488Z error vpxd[36481] [Originator@6876 sub=AuthzStorageProvider] [AuthzStorageProvider::CreateAuthzMgr] Failed to connect to IS: <N5Vmomi5Fault17HostCommunication9ExceptionE(Fault cause: vmodl.fault.HostCommunication
--> )
--> [context]zKq7AVECAAAAAAt9JgESdnB4ZAAAPFYrbGlidm1hY29yZS5zbwAAJEUbAFqxGAEGEFR2cHhkAAGu9FoBmOJjAV+voAG6mKACru4BbGliYXV0aHpjbGllbnQuc28AAlcHAgLSjgICsoYCAdkvnwFSJ1QBooZUAfnjUgPgBgJsaWJjLnNvLjYAAYXcUg==[/context]>
2022-05-19T13:40:08.490Z error vpxd[36481] [Originator@6876 sub=Default] Failed to instantiate AuthzStorageProvider: N5Vmomi5Fault17HostCommunication9ExceptionE(Fault cause: vmodl.fault.HostCommunication
--> )
--> [context]zKq7AVECAAAAAAt9JgESdnB4ZAAAPFYrbGlidm1hY29yZS5zbwAAJEUbAFqxGAEGEFR2cHhkAAGu9FoBmOJjAV+voAG6mKACru4BbGliYXV0aHpjbGllbnQuc28AAlcHAgLSjgICsoYCAdkvnwFSJ1QBooZUAfnjUgPgBgJsaWJjLnNvLjYAAYXcUg==[/context]

any thoughts?