VMware Cloud Community
irvingpop_chef
Contributor
Contributor
Jump to solution

"Unable to query vSphere health information" and "Unable to query vSAN health information" after certificate replacement - VCSA 6.7U2

I attempted to replace my machine cert on my VCSA server.  After a few attempts I gave up and performed a full certificate reset using the `/usr/lib/vmware-vmca/bin/certificate-manager` tool.

Now I'm seeing the following errors in the UI when looking at any Health or vSAN information.   Anyone know how to resolve this?  I upgraded to 6.7.0.40000 and that didn't help.

Screen Shot 2019-10-23 at 11.14.28 AM.png

Screen Shot 2019-10-23 at 11.14.43 AM.png

Screen Shot 2019-10-23 at 11.14.54 AM.png

in my /var/log/vmware/vsphere-ui/logs/vsphere_client_virgo.log I see lots of:

Caused by: com.vmware.vsphere.client.vsandp.core.sessionmanager.common.NotAccessibleException: com.vmware.vim.vmomi.client.exception.SslException: com.vmware.vim.vmomi.core.exception.CertificateValidationException: Server certificate chain is not trusted and thumbprint doesn't match

at com.vmware.vsphere.client.vsandp.core.sessionmanager.common.PbmClient.getConnection(PbmClient.java:70)

at com.vmware.vsphere.client.vsan.base.impl.PbmDataProvider.getProfileIds(PbmDataProvider.java:181)

at com.vmware.vsphere.client.vsan.base.impl.PbmDataProvider.getStoragePolicies(PbmDataProvider.java:131)

at com.vmware.vsphere.client.vsan.base.impl.PbmDataProvider.getObjectCompatibleStoragePolicies(PbmDataProvider.java:118)

... 119 common frames omitted

Caused by: com.vmware.vim.vmomi.client.exception.SslException: com.vmware.vim.vmomi.core.exception.CertificateValidationException: Server certificate chain is not trusted and thumbprint doesn't match

at com.vmware.vim.vmomi.client.common.impl.ResponseImpl.setError(ResponseImpl.java:256)

at com.vmware.vim.vmomi.client.http.impl.HttpExchange.run(HttpExchange.java:56)

at com.vmware.vim.vmomi.client.http.impl.HttpProtocolBindingBase.executeRunnable(HttpProtocolBindingBase.java:226)

at com.vmware.vim.vmomi.client.http.impl.HttpProtocolBindingImpl.send(HttpProtocolBindingImpl.java:106)

at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl$CallExecutor.sendCall(MethodInvocationHandlerImpl.java:629)

...

1 Solution

Accepted Solutions
Vijay2027
Expert
Expert
Jump to solution

Looks like there is a mismatch between machine ssl and endpoints certs. Please open a SR with GSS.

View solution in original post

30 Replies
Vijay2027
Expert
Expert
Jump to solution

Looks like there is a mismatch between machine ssl and endpoints certs. Please open a SR with GSS.

irvingpop_chef
Contributor
Contributor
Jump to solution

Vijay2027​ thanks for your quick response.  Unfortunately putting in a SR isn't an option right now.    Do you know how to check and/or manually fix the endpoint certs?

Reply
0 Kudos
Seyerl
Contributor
Contributor
Jump to solution

I have the same issue, do you have a solution for that issue already?

Thank you!

Reply
0 Kudos
irvingpop_chef
Contributor
Contributor
Jump to solution

I wish I did.  When did these issues start for you?

For me, they first started when I tried using a machine cert created by Let's Encrypt.  There were problems with this certificate.  I ended up resetting all of the certificates.  Since doing this, I've been plagued by issues.

Reply
0 Kudos
Seyerl
Contributor
Contributor
Jump to solution

I have the issues since I changed the VCSA certificate authority as a sub cert to my AD CA...

But I didn't tried to reset this until now.

Is it for you also working in the flash web client?

Reply
0 Kudos
irvingpop_chef
Contributor
Contributor
Jump to solution

Wow, surprisingly the flash client works!

I do see some errors in the /var/log/vmware/vsphere-client/logs/vsphere_client_virgo.log like:

[2019-10-28T11:05:28.557-07:00] [WARN ] http-nio-9090-exec-10         c.vmware.vim.vmomi.client.http.impl.AllowKnownThumbprintVerifier  Mismatched thumbprint 20:00:72:BA:3E:85:D4:93:A2:78:A4:83:62:2C:62:6C:4E:46:64:FF, rejecting connection

but it doesn't seem to affect the operation of the client

Reply
0 Kudos
Seyerl
Contributor
Contributor
Jump to solution

I see this error as well with the flash client, but I'm not sure if this is because of the other issue or something the flash client is doing, because I'm getting the same errors also with pages that are working in the HTML 5 client.

I'm leaving the office for today but tomorrow I will try to reset the certificate back to an VCSA self singed one, maybe this will change something.

Reply
0 Kudos
KocPawel
Hot Shot
Hot Shot
Jump to solution

Did you try this?

VMware Knowledge Base

Remember to make a backup (at least snasphot of your vCenter) before you do anything to be able to go back Smiley Happy

irvingpop_chef
Contributor
Contributor
Jump to solution

Yes.  see the second sentence in the original message Smiley Happy

Reply
0 Kudos
Vijay2027
Expert
Expert
Jump to solution

Before we check endpoints can you move the contents of the below folders to a backup location (Ex: /storage/core) and restart vsphere-client and vsphere-ui services.

usr/lib/vmware-vsphere-client/server/work

/usr/lib/vmware-virgo/server/pickup

Reply
0 Kudos
irvingpop_chef
Contributor
Contributor
Jump to solution

thanks @Vijay2027,  I moved out /usr/lib/vmware-vsphere-client/server/work but I didn't have a folder called /usr/lib/vmware-virgo/server/pickup in my install.  no change after restart of both services.

root@vcenter [ /usr/lib/vmware-virgo/server ]# ls -la

total 116

drwxr-xr-x 10 root root  4096 Oct 28 10:02 .

drwxr-xr-x  3 root root  4096 Oct 10 05:57 ..

drwxr-xr-x  2 root root  4096 Oct 28 10:02 about_files

-rwxr-xr-x  1 root root  5588 May 30 11:30 About.html

-rwxr-xr-x  1 root root  3140 May 30 11:30 AboutKernel.html

-rwxr-xr-x  1 root root  4381 May 30 11:30 AboutNano.html

drwxr-xr-x  2 root root  4096 Oct 28 10:02 admin

-rwxr-xr-x  1 root root 14547 May 30 11:30 artifacts.xml

drwxr-xr-x  2 root root  4096 Oct 28 10:02 bin

drwxr-xr-x  4 root root  4096 Oct 28 10:02 configuration

-rwxr-xr-x  1 root root 12567 May 30 11:30 epl-v10.html

drwxr-xr-x  4 root root  4096 Oct 28 10:02 lib

-rwxr-xr-x  1 root root  8783 May 30 11:30 notice.html

drwxr-xr-x  4 root root  4096 Oct 10 05:57 p2

drwxr-xr-x  2 root root 12288 Oct 28 10:02 plugins

drwxr-xr-x  3 root root  4096 Oct 10 05:57 repository

-rwxr-xr-x  1 root root  3616 May 30 11:31 vmware-changes.txt

Reply
0 Kudos
Vijay2027
Expert
Expert
Jump to solution

I've sent you few commands in DM. Pls check.

Reply
0 Kudos
irvingpop_chef
Contributor
Contributor
Jump to solution

Thanks again for your help Vijay2027​ - you were right, it was definitely a complex and not self-fixable issue.  I had a number of services that were running with the wrong certificate.  GSS provided me with a script that fixed them all.

Reply
0 Kudos
Globalfight3r
Contributor
Contributor
Jump to solution

Hi Vijay2027, could you please also send me the commands? I have exact the same problem.


Thank you in advance!

Reply
0 Kudos
HugoSarrazin
Contributor
Contributor
Jump to solution

Hi,

exactly same problem here, can i have the commands too please ?

Reply
0 Kudos
Vijay2027
Expert
Expert
Jump to solution

This is a complex process. GSS has automated the process to fix certificate mismatch. Please open a SR.

Yow will have to follow the process as per VMware Knowledge Base  to fix the mismatch.

Reply
0 Kudos
HugoSarrazin
Contributor
Contributor
Jump to solution

I finally found out that it is a HTML5 problem. I do not have this issue with the Flex client !

Reply
0 Kudos
Vijay2027
Expert
Expert
Jump to solution

The issue will be at service registration of HTML client. You will still have to get this corrected.

Reply
0 Kudos
HugoSarrazin
Contributor
Contributor
Jump to solution

reboot of vcenter and the problem disappeared...

Reply
0 Kudos