VMware Cloud Community
snikers
Enthusiast
Enthusiast

vCenter fails to start after certificate replacement

Using vcenter 6.7 Administration - > Certificates have added root CA certificate of Letsencrypt and replaced Machine certificate with signed one provide certificate and key

After reboot vcenter doesn`t start anymore:

2019-12-19T17:22:23.429Z info vpxd[05606] [Originator@6876 sub=ThreadPool] Entering worker thread loop

2019-12-19T17:22:23.430Z info vpxd[05605] [Originator@6876 sub=ThreadPool] Thread enlisted

2019-12-19T17:22:23.430Z info vpxd[05605] [Originator@6876 sub=ThreadPool] Entering worker thread loop

2019-12-19T17:22:23.459Z error vpxd[05321] [Originator@6876 sub=Main opID=CheckCertificateExpiry-6058ed8] Unable to get certificate count for APPLMGMT_PASSWORD from VECS localhost, error: 0

2019-12-19T17:22:23.548Z info vpxd[05332] [Originator@6876 sub=ThreadPool] Spawning additional worker - allocated: 144, idle: 19

2019-12-19T17:22:23.553Z info vpxd[05617] [Originator@6876 sub=ThreadPool] Thread enlisted

2019-12-19T17:22:23.553Z info vpxd[05617] [Originator@6876 sub=ThreadPool] Entering worker thread loop

2019-12-19T17:22:23.572Z warning vpxd[05113] [Originator@6876 sub=LSClient] Caught exception while getting service with Id :e2136204-f25b-4a2b-a5ac-67b473cfd253. N7Vmacore9ExceptionE(Cannot initialize service registration stub)

--> [context]zKq7AVECAAAAAGC34QAOdnB4ZAAA4AArbGlidm1hY29yZS5zbwAAWCUbABWWGAH5kWV2cHhkAAHOlWUB9qFlASkvoAIqhQJsaWJhdXRoemNsaWVudC5zbwABvdeeAToJVAGKaFQBGcZSA5AFAmxpYmMuc28uNgABpb5S[/context]

2019-12-19T17:22:23.573Z warning vpxd[05113] [Originator@6876 sub=LSClient] Caught exception while retrieve endpoint. N7Vmacore9ExceptionE(Cannot initialize service registration stub)

--> [context]zKq7AVECAAAAAGC34QAPdnB4ZAAA4AArbGlidm1hY29yZS5zbwAAWCUbABWWGAH5kWV2cHhkAAE4l2UBKJllASuiZQEpL6ACKoUCbGliYXV0aHpjbGllbnQuc28AAb3XngE6CVQBimhUARnGUgOQBQJsaWJjLnNvLjYAAaW+Ug==[/context]

2019-12-19T17:22:23.574Z warning vpxd[05113] [Originator@6876 sub=LSClient] endpoint not found for Product: com.vmware.cis, Type: cs.inventory

2019-12-19T17:22:23.574Z warning vpxd[05113] [Originator@6876 sub=LSClient] Caught exception while getting service with Id :e2136204-f25b-4a2b-a5ac-67b473cfd253. N7Vmacore9ExceptionE(Cannot initialize service registration stub)

--> [context]zKq7AVECAAAAAGC34QAOdnB4ZAAA4AArbGlidm1hY29yZS5zbwAAWCUbABWWGAH5kWV2cHhkAAHOlWUB9qFlASkvoAI3hQJsaWJhdXRoemNsaWVudC5zbwABvdeeAToJVAGKaFQBGcZSA5AFAmxpYmMuc28uNgABpb5S[/context]

2019-12-19T17:22:23.575Z warning vpxd[05113] [Originator@6876 sub=LSClient] Caught exception while retrieve endpoint. N7Vmacore9ExceptionE(Cannot initialize service registration stub)

--> [context]zKq7AVECAAAAAGC34QAPdnB4ZAAA4AArbGlidm1hY29yZS5zbwAAWCUbABWWGAH5kWV2cHhkAAE4l2UBKJllASuiZQEpL6ACN4UCbGliYXV0aHpjbGllbnQuc28AAb3XngE6CVQBimhUARnGUgOQBQJsaWJjLnNvLjYAAaW+Ug==[/context]

2019-12-19T17:22:23.597Z warning vpxd[05113] [Originator@6876 sub=LSClient] endpoint not found for Product: com.vmware.cis, Type: cs.inventory

2019-12-19T17:22:23.718Z warning vpxd[05113] [Originator@6876 sub=VpxdAuthClient] [ConnectAndLogin] Failed to loginBySamlToken: N7Vmacore3Ssl18SSLVerifyExceptionE(SSL Exception: Verification parameters:

--> PeerThumbprint: 49:68:90:15:2C:75:C6:7C:C7:B4:55:EB:87:E2:E6:29:92:21:A8:72

--> ExpectedThumbprint:

--> ExpectedPeerName: localhost

--> The remote host certificate has these problems:

-->

--> * Host name does not match the subject name(s) in certificate.)

--> [context]zKq7AVECAAAAAGC34QANdnB4ZAAA4AArbGlidm1hY29yZS5zbwAAWCUbAP6dGACeQCIAaXEiABtFIgDTSSIAOaIjAHFvIwA6ciMAnVYrAdRzAGxpYnB0aHJlYWQuc28uMAAC3Y4ObGliYy5zby42AA==[/context]

2019-12-19T17:22:23.719Z info vpxd[05113] [Originator@6876 sub=VpxdAuthClient] fallback to loginByCertificate

2019-12-19T17:22:23.729Z error vpxd[05113] [Originator@6876 sub=ServerAccess] Remote login failed: N7Vmacore3Ssl18SSLVerifyExceptionE(SSL Exception: Verification parameters:

--> PeerThumbprint: 49:68:90:15:2C:75:C6:7C:C7:B4:55:EB:87:E2:E6:29:92:21:A8:72

--> ExpectedThumbprint:

--> ExpectedPeerName: localhost

--> The remote host certificate has these problems:

-->

--> * Host name does not match the subject name(s) in certificate.)

When resetting certificates using /usr/lib/vmware-vmca/bin/certificate-manager it works again

There is no ESXi host connected to vCenter just in case...

Reply
0 Kudos
8 Replies
T180985
Expert
Expert

Looks like the cert is incorrectly configured

* Host name does not match the subject name(s) in certificate.

Please mark helpful or correct if my answer resolved your issue. How to post effectively on VMTN https://communities.vmware.com/people/daphnissov/blog/2018/12/05/how-to-ask-for-help-on-tech-forums
Reply
0 Kudos
Vijay2027
Expert
Expert

Run the below commands and make sure all 3 gives you hostnmae of vCSA

1. PNID of the vCenter server: # /usr/lib/vmware-vmafd/bin/vmafd-cli get-pnid --server-name localhost

2. Hostname of vCenter server: # hostname -f

3. And SAN (Subject Alternative Name) field of machine ssl cert: # openssl x509 -in machine.cer -noout -text | grep DNS:

Reply
0 Kudos
meanevo
Contributor
Contributor

same here... cant figure out why

tail -f /var/log/vmware/vpxd/vpxd.log

--> ExpectedThumbprint:

--> ExpectedPeerName: localhost

--> The remote host certificate has these problems:

-->

--> * Host name does not match the subject name(s) in certificate.)

the following command gives same result

/usr/lib/vmware-vmafd/bin/vmafd-cli get-pnid --server-name localhost

/usr/lib/vmware-vmafd/bin/vecs-cli entry list --store MACHINE_SSL_CERT --text | grep -A1 Alternative

openssl x509 -in <path_to_certificate_file> -noout -text | grep -A1 Alternative

hostname -f

slairipp
Contributor
Contributor

Did you ever find a fix for this issue?  Having the exact same issue here.

Reply
0 Kudos
mohitsharma444
Contributor
Contributor

I had the same issue for past few weeks. Updated to 6.7.0.44000 and looks like this is resolved. I've run my playbook for renewing letsencrypt certificates a bunch of times, rebooted vcsa and everything seems to be stable so far.

Reply
0 Kudos
Virtbay
Enthusiast
Enthusiast

It could be also due to duplicate certificate in the trusted root store . Try running the below command and match the serial numbers . If you find duplicate serial numbers then you would have to remove them.

/usr/lib/vmware-vmafd/bin/vecs-cli entry list --store TRUSTED_ROOTS --text| grep -A 6 -i alias |less

Regards VB Please mark helpful or correct if my answer resolved your issue.
Reply
0 Kudos
jgrade82
Contributor
Contributor

Was this fixed?? 

I am facing the same issue. Followed the thread but nothing seems to work

Reply
0 Kudos
kursanthoy
Contributor
Contributor

vcenter 6.7.0.52000

have the same issue after renewal letencrypt certificate:

vcenter.yyy.com

# /usr/lib/vmware-vmafd/bin/vmafd-cli get-pnid --server-name localhost
vcenter.yyy.com
# hostname -f
vcenter.yyy.com
# /usr/lib/vmware-vmafd/bin/vecs-cli entry list --store MACHINE_SSL_CERT --text | grep -A1 Alternative
X509v3 Subject Alternative Name:
DNS:vcenter.yyy.com


--> PeerThumbprint: 98:FE:16:42:E3:CF:43:2B:63:C5:9D:79:9C:77:FB:BD:B2:2A:07:FA
--> ExpectedThumbprint:
--> ExpectedPeerName: localhost
--> The remote host certificate has these problems:
-->
--> * Host name does not match the subject name(s) in certificate.)
--> [context]zKq7AVECAAAAAAt9JgENdnB4ZAAAPFYrbGlidm1hY29yZS5zbwAAJEUbAFqxGAAehyIA9b4iAJuLIgBTkCIAie8jAMG8IwCKvyMA+asrAdRzAGxpYnB0aHJlYWQuc28uMAAC7Y8ObGliYy5zby42AA==[/context]
2022-05-19T13:40:08.483Z info vpxd[36481] [Originator@6876 sub=VpxdAuthClient] fallback to loginByCertificate
2022-05-19T13:40:08.487Z error vpxd[36481] [Originator@6876 sub=ServerAccess] Remote login failed: N7Vmacore3Ssl18SSLVerifyExceptionE(SSL Exception: Verification parameters:
--> PeerThumbprint: 98:FE:16:42:E3:CF:43:2B:63:C5:9D:79:9C:77:FB:BD:B2:2A:07:FA
--> ExpectedThumbprint:
--> ExpectedPeerName: localhost
--> The remote host certificate has these problems:
-->
--> * Host name does not match the subject name(s) in certificate.)
--> [context]zKq7AVECAAAAAAt9JgENdnB4ZAAAPFYrbGlidm1hY29yZS5zbwAAJEUbAFqxGAAehyIA9b4iAJuLIgBTkCIAie8jAMG8IwCKvyMA+asrAdRzAGxpYnB0aHJlYWQuc28uMAAC7Y8ObGliYy5zby42AA==[/context]
2022-05-19T13:40:08.488Z error vpxd[36481] [Originator@6876 sub=AuthzStorageProvider] [AuthzStorageProvider::CreateAuthzMgr] Failed to connect to IS: <N5Vmomi5Fault17HostCommunication9ExceptionE(Fault cause: vmodl.fault.HostCommunication
--> )
--> [context]zKq7AVECAAAAAAt9JgESdnB4ZAAAPFYrbGlidm1hY29yZS5zbwAAJEUbAFqxGAEGEFR2cHhkAAGu9FoBmOJjAV+voAG6mKACru4BbGliYXV0aHpjbGllbnQuc28AAlcHAgLSjgICsoYCAdkvnwFSJ1QBooZUAfnjUgPgBgJsaWJjLnNvLjYAAYXcUg==[/context]>
2022-05-19T13:40:08.490Z error vpxd[36481] [Originator@6876 sub=Default] Failed to instantiate AuthzStorageProvider: N5Vmomi5Fault17HostCommunication9ExceptionE(Fault cause: vmodl.fault.HostCommunication
--> )
--> [context]zKq7AVECAAAAAAt9JgESdnB4ZAAAPFYrbGlidm1hY29yZS5zbwAAJEUbAFqxGAEGEFR2cHhkAAGu9FoBmOJjAV+voAG6mKACru4BbGliYXV0aHpjbGllbnQuc28AAlcHAgLSjgICsoYCAdkvnwFSJ1QBooZUAfnjUgPgBgJsaWJjLnNvLjYAAYXcUg==[/context]



any thoughts?




Tags (1)
Reply
0 Kudos