Vcenter server enhanced mode certificate update problem

[smelnik]

Hello everyone!

I`ve got two vcenter servers(vcenter1 and vcenter2) with external PSC(psc1 and psc2). The problem is that on vcenter2 certificates were updated but on psc havent added new certificates to active. So if i log in vcenter2 i can manage vcenter2 and vcenter1, but if i log into vcenter1 i see message "Cannot connect to one or more vcenter servers". 

I`ve googled for case like this, but could not find anything. Can anyone point me how to solve this problem?

[scott28tt]

@smelnik 

Moderator: Moved to vCenter Server Discussions

[msripada]

when you login to VC1, we need to check the webclient logs for the vc1 and see why its not able to communicate with vc2.

do you have all ports opened with psc/vcs in your environment? Is the behavior same with SSO administrator as well or only with domain accounts?

thanks,

MS

[smelnik]

Hello msripada,

there are no errors in web client logs, but in /var/log/vmware/vapi/endpoint/endpoint.log is see errors like this:

Spoiler

com.vmware.vim.query.client.exception.ClientException: java.util.concurrent.ExecutionException: com.vmware.vim.vmomi.client.exception.SslException: com.vmware.vim.vmomi.core.exception.CertificateValidationException: Server certificate chain is not trusted and thumbprint verification is not configured
        at com.vmware.vim.query.client.impl.QueryAuthenticationManagerImpl.loginBySamlToken(QueryAuthenticationManagerImpl.java:232)
        at com.vmware.vapi.endpoint.cis.router.InvProviderClientFactory.createProviderClient(InvProviderClientFactory.java:105)
        at com.vmware.vapi.endpoint.cis.router.InvSvcBuilder.createInvServiceClientList(InvSvcBuilder.java:345)
        at com.vmware.vapi.endpoint.cis.router.InvSvcBuilder.buildInt(InvSvcBuilder.java:296)
        at com.vmware.vapi.endpoint.cis.router.InvSvcBuilder.rebuild(InvSvcBuilder.java:254)
        at com.vmware.vapi.state.impl.DefaultStateManager.rebuild(DefaultStateManager.java:406)
        at com.vmware.vapi.state.impl.DefaultStateManager$2.doReconfig(DefaultStateManager.java:444)
        at com.vmware.vapi.state.impl.DefaultStateManager$2.run(DefaultStateManager.java:433)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.util.concurrent.ExecutionException: com.vmware.vim.vmomi.client.exception.SslException: com.vmware.vim.vmomi.core.exception.CertificateValidationException: Server certificate chain is not trusted and thumbprint verification is not configured
        at com.vmware.vim.vmomi.core.impl.BlockingFuture.get(BlockingFuture.java:81)
        at com.vmware.vim.query.client.impl.QueryAuthenticationManagerImpl.loginBySamlToken(QueryAuthenticationManagerImpl.java:230)
        ... 14 more
Caused by: com.vmware.vim.vmomi.client.exception.SslException: com.vmware.vim.vmomi.core.exception.CertificateValidationException: Server certificate chain is not trusted and thumbprint verification is not configured
        at com.vmware.vim.vmomi.client.common.impl.ResponseImpl.setError(ResponseImpl.java:256)
        at com.vmware.vim.vmomi.client.http.impl.HttpExchange.run(HttpExchange.java:51)
        at com.vmware.vim.vmomi.client.http.impl.HttpProtocolBindingBase.executeRunnable(HttpProtocolBindingBase.java:226)
        at com.vmware.vim.vmomi.client.http.impl.HttpProtocolBindingImpl.send(HttpProtocolBindingImpl.java:110)
        at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl$CallExecutor.sendCall(MethodInvocationHandlerImpl.java:613)
        at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl$CallExecutor.executeCall(MethodInvocationHandlerImpl.java:594)
        at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl.completeCall(MethodInvocationHandlerImpl.java:345)
        at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl.invokeOperation(MethodInvocationHandlerImpl.java:305)
        at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl.invoke(MethodInvocationHandlerImpl.java:179)
        at com.sun.proxy.$Proxy91.loginBySamlToken(Unknown Source)
        at com.vmware.vim.query.client.impl.QueryAuthenticationManagerImpl.loginBySamlToken(QueryAuthenticationManagerImpl.java:228)
        ... 14 more
Caused by: com.vmware.vim.vmomi.core.exception.CertificateValidationException: Server certificate chain is not trusted and thumbprint verification is not configured
        at com.vmware.vim.vmomi.client.http.impl.ClientExceptionTranslator.translate(ClientExceptionTranslator.java:54)
        ... 25 more
Caused by: com.vmware.vim.vmomi.core.exception.CertificateValidationException: SSL handshake from 0.0.0.0/0.0.0.0:53206 to vcenter2/172.22.0.253:443 failed in 25 ms
        at com.vmware.vim.vmomi.client.http.impl.ThumbprintTrustManager$HostnameVerifier.handleHandshakeException(ThumbprintTrustManager.java:597)
        at com.vmware.vim.vmomi.client.http.impl.ThumbprintTrustManager$HostnameVerifier.verify(ThumbprintTrustManager.java:422)
        at com.vmware.vim.vmomi.client.http.impl.VlsiSslSocketFactory.verifyHostname(VlsiSslSocketFactory.java:129)
        at com.vmware.vim.vmomi.client.http.impl.VlsiSslSocketFactory.createLayeredSocket(VlsiSslSocketFactory.java:122)
        at com.vmware.vim.vmomi.client.http.impl.VlsiSslSocketFactory.connectSocket(VlsiSslSocketFactory.java:88)
        at org.apache.http.impl.conn.HttpClientConnectionOperator.connect(HttpClientConnectionOperator.java:117)
        at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:314)
        at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:363)
        at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:219)
        at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:195)
        at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:86)
        at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:108)
        at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:186)
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:57)
        at com.vmware.vim.vmomi.client.http.impl.HttpExchange.run(HttpExchange.java:45)
        ... 23 more
Caused by: javax.net.ssl.SSLHandshakeException: com.vmware.vim.vmomi.client.exception.VlsiCertificateException: Server certificate chain is not trusted and thumbprint verification is not configured
        at sun.security.ssl.Alerts.getSSLException(Alerts.java:198)
        at sun.security.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1967)
        at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:331)
        at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:325)
        at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1689)
        at sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:226)
        at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1082)
        at sun.security.ssl.Handshaker.process_record(Handshaker.java:1010)
        at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1079)
        at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1388)
        at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1416)
        at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1400)
        at com.vmware.vim.vmomi.client.http.impl.ThumbprintTrustManager$HostnameVerifier.verify(ThumbprintTrustManager.java:420)
        ... 37 more
Caused by: com.vmware.vim.vmomi.client.exception.VlsiCertificateException: Server certificate chain is not trusted and thumbprint verification is not configured
        at com.vmware.vim.vmomi.client.http.impl.ThumbprintTrustManager.checkServerTrusted(ThumbprintTrustManager.java:206)
        at sun.security.ssl.AbstractTrustManagerWrapper.checkServerTrusted(SSLContextImpl.java:1099)
        at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1671)
        ... 45 more
Caused by: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
        at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:450)
        at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:317)
        at sun.security.validator.Validator.validate(Validator.java:262)
        at sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:330)
        at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:235)
        at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:113)
        at com.vmware.vim.vmomi.client.http.impl.ThumbprintTrustManager.checkServerTrusted(ThumbprintTrustManager.java:191)
        ... 47 more
Caused by: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
        at sun.security.provider.certpath.SunCertPathBuilder.build(SunCertPathBuilder.java:141)
        at sun.security.provider.certpath.SunCertPathBuilder.engineBuild(SunCertPathBuilder.java:126)
        at java.security.cert.CertPathBuilder.build(CertPathBuilder.java:280)
        at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:445)
        ... 53 more

[msripada]

MACHINE SSL of vcenter 2 is having issues with trust mismatch. You can use lsdoctor https://kb.vmware.com/s/article/80469 but you need to have maintenance to shutdown and take powered off snapshots of all vcenter/pscs in the environment. Use lsdoctor -t once you have snaps and backups ready. 

thanks,

MS

[scott28tt]

@smelnik 

Moderator: Please use the "spoiler" function when posting large text dumps to make the thread readable by others, I have edited your most recent post so you can see the difference.

You add a "spoiler" to a post using the triangle icon on the extended toolbar of the post creator/editor:

[smelnik]

Thanks for answer.

i`ve tried lsdoctor util, but got this:

Spoiler

Provide password for administrator@vsphere.local:
2021-02-02T08:41:43 INFO __init__: Retrieved services from SSO site: vniikr-local
2021-02-02T08:41:43 INFO findAndFix: Checking services for trust mismatches...
2021-02-02T08:41:43 INFO findAndFix: Attempting to reregister d84cec37-1301-405f-8e9c-b16978d673d7 for vcenter2.vsphere.site
2021-02-02T08:41:44 INFO findAndFix: Attempting to reregister 096d9cdf-2d5c-4b64-ae78-af1e5d964648 for vcenter2.vsphere.site
2021-02-02T08:41:44 INFO findAndFix: Attempting to reregister d84cec37-1301-405f-8e9c-b16978d673d7_authz for vcenter2.vsphere.site
2021-02-02T08:41:44 INFO findAndFix: Attempting to reregister 88649dfd-d65d-4c29-8790-1c0c7b224010 for vcenter2.vsphere.site
2021-02-02T08:41:45 INFO findAndFix: Attempting to reregister d84cec37-1301-405f-8e9c-b16978d673d7_kv for vcenter2.vsphere.site
2021-02-02T08:41:45 INFO findAndFix: Attempting to reregister f0da7786-fbf6-4b05-83e4-38481f4cbd03 for vcenter2.vsphere.site
2021-02-02T08:41:46 INFO findAndFix: Attempting to reregister vniikr-local:4e7099b2-bc08-49fa-8cdc-2a6865c1c57e for psc02.vsphere.site
2021-02-02T08:41:46 INFO findAndFix: Attempting to reregister 34486bc5-9a97-4def-97e2-8dcc837b59dd for psc02.vsphere.site
2021-02-02T08:41:46 INFO findAndFix: Attempting to reregister 0fa71877-966b-4710-b033-a02a661022fa for vcenter2.vsphere.site
2021-02-02T08:41:46 INFO findAndFix: Attempting to reregister vniikr-local:a3151943-ab9d-4c62-b1b8-79fb776cf282 for psc02.vsphere.site
2021-02-02T08:43:53 WARNING findAndFix: 172.22.0.250 is now blacklisted.
2021-02-02T08:43:54 INFO findAndFix: Attempting to reregister a2eeadec-8442-421f-8c5d-8fd07c62ceab for vcenter2.vsphere.site
2021-02-02T08:43:54 WARNING unregister_service: Failed to unregister_service [a2eeadec-8442-421f-8c5d-8fd07c62ceab]: '', sys.exc_info(
2021-02-02T08:43:54 WARNING unregister_service: Failed to unregister_service [a2eeadec-8442-421f-8c5d-8fd07c62ceab]: '', str(e)
2021-02-02T08:43:54 WARNING unregister_service: Failed to unregister_service [a2eeadec-8442-421f-8c5d-8fd07c62ceab]: BadStatusLine("''
2021-02-02T08:43:54 WARNING unregister_service: Failed to unregister_service [a2eeadec-8442-421f-8c5d-8fd07c62ceab]: Traceback (most r
File "/root/lsdoctor/lsdoctor-master/lib/utils.py", line 768, in unregister_service
self.service_content.serviceRegistration.Delete(svc_id)
File "/usr/lib/vmware/site-packages/pyVmomi/VmomiSupport.py", line 557, in <lambda>
self.f(*(self.args + (obj,) + args), **kwargs)
File "/usr/lib/vmware/site-packages/pyVmomi/VmomiSupport.py", line 363, in _InvokeMethod
return self._stub.InvokeMethod(self, info, args)
File "/usr/lib/vmware/site-packages/pyVmomi/SoapAdapter.py", line 1410, in InvokeMethod
resp = conn.getresponse()
File "/usr/lib/python2.7/httplib.py", line 1161, in getresponse
response.begin()
File "/usr/lib/python2.7/httplib.py", line 448, in begin
version, status, reason = self._read_status()
File "/usr/lib/python2.7/httplib.py", line 412, in _read_status
raise BadStatusLine(line)
BadStatusLine: ''
, traceback.format_exc()
2021-02-02T08:43:54 ERROR unregister_service: Failed to unregister service a2eeadec-8442-421f-8c5d-8fd07c62ceab, esclate the error
2021-02-02T08:43:54 ERROR findAndFix: Failed to re-register a2eeadec-8442-421f-8c5d-8fd07c62ceab
Traceback (most recent call last):
File "lsdoctor.py", line 520, in <module>
main()
File "lsdoctor.py", line 492, in main
trustFix(params, username, password)
File "lsdoctor.py", line 359, in trustFix
trust_check.check()
File "/root/lsdoctor/lsdoctor-master/lib/trust.py", line 197, in check
self.findAndFix()
File "/root/lsdoctor/lsdoctor-master/lib/trust.py", line 180, in findAndFix
self.ls.unregister(serviceId)
File "/root/lsdoctor/lsdoctor-master/lib/utils.py", line 1265, in unregister
self.lsClient.unregister_service(svc_id)
File "/root/lsdoctor/lsdoctor-master/lib/utils.py", line 724, in add_securityctx_to_requests
return req_method(self, *args, **kargs)
File "/root/lsdoctor/lsdoctor-master/lib/utils.py", line 768, in unregister_service
self.service_content.serviceRegistration.Delete(svc_id)
File "/usr/lib/vmware/site-packages/pyVmomi/VmomiSupport.py", line 557, in <lambda>
self.f(*(self.args + (obj,) + args), **kwargs)
File "/usr/lib/vmware/site-packages/pyVmomi/VmomiSupport.py", line 363, in _InvokeMethod
return self._stub.InvokeMethod(self, info, args)
File "/usr/lib/vmware/site-packages/pyVmomi/SoapAdapter.py", line 1410, in InvokeMethod
resp = conn.getresponse()
File "/usr/lib/python2.7/httplib.py", line 1161, in getresponse
response.begin()
File "/usr/lib/python2.7/httplib.py", line 448, in begin
version, status, reason = self._read_status()
File "/usr/lib/python2.7/httplib.py", line 412, in _read_status
raise BadStatusLine(line)
httplib.BadStatusLine: ''

But vcenter2 appeared in web client of vcenter1, but still not managebale.

Is ther a way to upload new certificates of vcenter2 to psc1?

[msripada]

I suggest you to kindly open case with GSS as things may get complicated if we tweak issues with certs

[JRavi]

@smelnik were you able to resolve this issue. If so please help us with the resolution. We are also seeing this issue in our environment.

[2nia]

1. Login to the PSC node and run "/usr/lib/vmware-vmdir/bin/vdcadmintool" to check vmdir state 
2. If vmdir state output is read only , use "/usr/lib/vmware-vmafd/bin/dir-cli state set --state NORMAL
3. and try "python lsdoctor.py -l" and  "python lsdoctor.py -t"