topic Re: "Unable to query vSphere health information" and "Unable to query vSAN health information" after certificate replacement - VCSA 6.7U2 in vCenter™ Server Discussions

"Unable to query vSphere health information" and "Unable to query vSAN health information" after certificate replacement - VCSA 6.7U2

irvingpop_chef — Wed, 23 Oct 2019 18:25:14 GMT

I attempted to replace my machine cert on my VCSA server. After a few attempts I gave up and performed a full certificate reset using the `/usr/lib/vmware-vmca/bin/certificate-manager` tool.

Now I'm seeing the following errors in the UI when looking at any Health or vSAN information. Anyone know how to resolve this? I upgraded to 6.7.0.40000 and that didn't help.

in my /var/log/vmware/vsphere-ui/logs/vsphere_client_virgo.log I see lots of:

Caused by: com.vmware.vsphere.client.vsandp.core.sessionmanager.common.NotAccessibleException: com.vmware.vim.vmomi.client.exception.SslException: com.vmware.vim.vmomi.core.exception.CertificateValidationException: Server certificate chain is not trusted and thumbprint doesn't match

at com.vmware.vsphere.client.vsandp.core.sessionmanager.common.PbmClient.getConnection(PbmClient.java:70)

at com.vmware.vsphere.client.vsan.base.impl.PbmDataProvider.getProfileIds(PbmDataProvider.java:181)

at com.vmware.vsphere.client.vsan.base.impl.PbmDataProvider.getStoragePolicies(PbmDataProvider.java:131)

at com.vmware.vsphere.client.vsan.base.impl.PbmDataProvider.getObjectCompatibleStoragePolicies(PbmDataProvider.java:118)

... 119 common frames omitted

Caused by: com.vmware.vim.vmomi.client.exception.SslException: com.vmware.vim.vmomi.core.exception.CertificateValidationException: Server certificate chain is not trusted and thumbprint doesn't match

at com.vmware.vim.vmomi.client.common.impl.ResponseImpl.setError(ResponseImpl.java:256)

at com.vmware.vim.vmomi.client.http.impl.HttpExchange.run(HttpExchange.java:56)

at com.vmware.vim.vmomi.client.http.impl.HttpProtocolBindingBase.executeRunnable(HttpProtocolBindingBase.java:226)

at com.vmware.vim.vmomi.client.http.impl.HttpProtocolBindingImpl.send(HttpProtocolBindingImpl.java:106)

at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl$CallExecutor.sendCall(MethodInvocationHandlerImpl.java:629)

...

Re: "Unable to query vSphere health information" and "Unable to query vSAN health information" after certificate replacement - VCSA 6.7U2

Vijay2027 — Thu, 24 Oct 2019 01:52:30 GMT

Looks like there is a mismatch between machine ssl and endpoints certs. Please open a SR with GSS.

Re: "Unable to query vSphere health information" and "Unable to query vSAN health information" after certificate replacement - VCSA 6.7U2

irvingpop_chef — Mon, 28 Oct 2019 15:15:35 GMT

Vijay2027 thanks for your quick response. Unfortunately putting in a SR isn't an option right now. Do you know how to check and/or manually fix the endpoint certs?

Re: "Unable to query vSphere health information" and "Unable to query vSAN health information" after certificate replacement - VCSA 6.7U2

Seyerl — Mon, 28 Oct 2019 16:46:33 GMT

I have the same issue, do you have a solution for that issue already?

Thank you!

Re: "Unable to query vSphere health information" and "Unable to query vSAN health information" after certificate replacement - VCSA 6.7U2

irvingpop_chef — Mon, 28 Oct 2019 17:00:08 GMT

I wish I did. When did these issues start for you?

For me, they first started when I tried using a machine cert created by Let's Encrypt. There were problems with this certificate. I ended up resetting all of the certificates. Since doing this, I've been plagued by issues.

Re: "Unable to query vSphere health information" and "Unable to query vSAN health information" after certificate replacement - VCSA 6.7U2

Seyerl — Mon, 28 Oct 2019 17:03:10 GMT

I have the issues since I changed the VCSA certificate authority as a sub cert to my AD CA...

But I didn't tried to reset this until now.

Is it for you also working in the flash web client?

Re: "Unable to query vSphere health information" and "Unable to query vSAN health information" after certificate replacement - VCSA 6.7U2

irvingpop_chef — Mon, 28 Oct 2019 18:11:32 GMT

Wow, surprisingly the flash client works!

I do see some errors in the /var/log/vmware/vsphere-client/logs/vsphere_client_virgo.log like:

[2019-10-28T11:05:28.557-07:00] [WARN ] http-nio-9090-exec-10 c.vmware.vim.vmomi.client.http.impl.AllowKnownThumbprintVerifier Mismatched thumbprint 20:00:72:BA:3E:85:D4:93:A2:78:A4:83:62:2C:62:6C:4E:46:64:FF, rejecting connection

but it doesn't seem to affect the operation of the client

Re: "Unable to query vSphere health information" and "Unable to query vSAN health information" after certificate replacement - VCSA 6.7U2

Seyerl — Mon, 28 Oct 2019 18:20:58 GMT

I see this error as well with the flash client, but I'm not sure if this is because of the other issue or something the flash client is doing, because I'm getting the same errors also with pages that are working in the HTML 5 client.

I'm leaving the office for today but tomorrow I will try to reset the certificate back to an VCSA self singed one, maybe this will change something.

Re: "Unable to query vSphere health information" and "Unable to query vSAN health information" after certificate replacement - VCSA 6.7U2

KocPawel — Mon, 28 Oct 2019 21:49:53 GMT

Did you try this?

VMware Knowledge Base

Remember to make a backup (at least snasphot of your vCenter) before you do anything to be able to go back

Re: "Unable to query vSphere health information" and "Unable to query vSAN health information" after certificate replacement - VCSA 6.7U2

irvingpop_chef — Mon, 28 Oct 2019 21:53:04 GMT

Yes. see the second sentence in the original message

Re: "Unable to query vSphere health information" and "Unable to query vSAN health information" after certificate replacement - VCSA 6.7U2

Vijay2027 — Tue, 29 Oct 2019 04:52:13 GMT

Before we check endpoints can you move the contents of the below folders to a backup location (Ex: /storage/core) and restart vsphere-client and vsphere-ui services.

usr/lib/vmware-vsphere-client/server/work

/usr/lib/vmware-virgo/server/pickup

Re: "Unable to query vSphere health information" and "Unable to query vSAN health information" after certificate replacement - VCSA 6.7U2

irvingpop_chef — Tue, 29 Oct 2019 18:11:33 GMT

thanks @Vijay2027, I moved out /usr/lib/vmware-vsphere-client/server/work but I didn't have a folder called /usr/lib/vmware-virgo/server/pickup in my install. no change after restart of both services.

root@vcenter [ /usr/lib/vmware-virgo/server ]# ls -la

total 116

drwxr-xr-x 10 root root 4096 Oct 28 10:02 .

drwxr-xr-x 3 root root 4096 Oct 10 05:57 ..

drwxr-xr-x 2 root root 4096 Oct 28 10:02 about_files

-rwxr-xr-x 1 root root 5588 May 30 11:30 About.html

-rwxr-xr-x 1 root root 3140 May 30 11:30 AboutKernel.html

-rwxr-xr-x 1 root root 4381 May 30 11:30 AboutNano.html

drwxr-xr-x 2 root root 4096 Oct 28 10:02 admin

-rwxr-xr-x 1 root root 14547 May 30 11:30 artifacts.xml

drwxr-xr-x 2 root root 4096 Oct 28 10:02 bin

drwxr-xr-x 4 root root 4096 Oct 28 10:02 configuration

-rwxr-xr-x 1 root root 12567 May 30 11:30 epl-v10.html

drwxr-xr-x 4 root root 4096 Oct 28 10:02 lib

-rwxr-xr-x 1 root root 8783 May 30 11:30 notice.html

drwxr-xr-x 4 root root 4096 Oct 10 05:57 p2

drwxr-xr-x 2 root root 12288 Oct 28 10:02 plugins

drwxr-xr-x 3 root root 4096 Oct 10 05:57 repository

-rwxr-xr-x 1 root root 3616 May 30 11:31 vmware-changes.txt

Re: "Unable to query vSphere health information" and "Unable to query vSAN health information" after certificate replacement - VCSA 6.7U2

Vijay2027 — Wed, 30 Oct 2019 01:32:42 GMT

I've sent you few commands in DM. Pls check.

Re: "Unable to query vSphere health information" and "Unable to query vSAN health information" after certificate replacement - VCSA 6.7U2

irvingpop_chef — Thu, 14 Nov 2019 18:55:01 GMT

Thanks again for your help Vijay2027 - you were right, it was definitely a complex and not self-fixable issue. I had a number of services that were running with the wrong certificate. GSS provided me with a script that fixed them all.

Re: "Unable to query vSphere health information" and "Unable to query vSAN health information" after certificate replacement - VCSA 6.7U2

Globalfight3r — Mon, 25 Nov 2019 16:26:29 GMT

Hi Vijay2027, could you please also send me the commands? I have exact the same problem.

Thank you in advance!

Re: "Unable to query vSphere health information" and "Unable to query vSAN health information" after certificate replacement - VCSA 6.7U2

HugoSarrazin — Tue, 26 Nov 2019 10:38:47 GMT

Hi,

exactly same problem here, can i have the commands too please ?

Re: "Unable to query vSphere health information" and "Unable to query vSAN health information" after certificate replacement - VCSA 6.7U2

Vijay2027 — Tue, 26 Nov 2019 11:12:17 GMT

This is a complex process. GSS has automated the process to fix certificate mismatch. Please open a SR.

Yow will have to follow the process as per VMware Knowledge Base to fix the mismatch.

Re: "Unable to query vSphere health information" and "Unable to query vSAN health information" after certificate replacement - VCSA 6.7U2

HugoSarrazin — Tue, 26 Nov 2019 13:39:34 GMT

I finally found out that it is a HTML5 problem. I do not have this issue with the Flex client !

Re: "Unable to query vSphere health information" and "Unable to query vSAN health information" after certificate replacement - VCSA 6.7U2

Vijay2027 — Tue, 26 Nov 2019 14:07:44 GMT

The issue will be at service registration of HTML client. You will still have to get this corrected.

Re: "Unable to query vSphere health information" and "Unable to query vSAN health information" after certificate replacement - VCSA 6.7U2

HugoSarrazin — Tue, 26 Nov 2019 14:47:41 GMT

reboot of vcenter and the problem disappeared...