Highlighted
Contributor
Contributor

"Unable to query vSphere health information" and "Unable to query vSAN health information" after certificate replacement - VCSA 6.7U2

Jump to solution

I attempted to replace my machine cert on my VCSA server.  After a few attempts I gave up and performed a full certificate reset using the `/usr/lib/vmware-vmca/bin/certificate-manager` tool.

Now I'm seeing the following errors in the UI when looking at any Health or vSAN information.   Anyone know how to resolve this?  I upgraded to 6.7.0.40000 and that didn't help.

Screen Shot 2019-10-23 at 11.14.28 AM.png

Screen Shot 2019-10-23 at 11.14.43 AM.png

Screen Shot 2019-10-23 at 11.14.54 AM.png

in my /var/log/vmware/vsphere-ui/logs/vsphere_client_virgo.log I see lots of:

Caused by: com.vmware.vsphere.client.vsandp.core.sessionmanager.common.NotAccessibleException: com.vmware.vim.vmomi.client.exception.SslException: com.vmware.vim.vmomi.core.exception.CertificateValidationException: Server certificate chain is not trusted and thumbprint doesn't match

at com.vmware.vsphere.client.vsandp.core.sessionmanager.common.PbmClient.getConnection(PbmClient.java:70)

at com.vmware.vsphere.client.vsan.base.impl.PbmDataProvider.getProfileIds(PbmDataProvider.java:181)

at com.vmware.vsphere.client.vsan.base.impl.PbmDataProvider.getStoragePolicies(PbmDataProvider.java:131)

at com.vmware.vsphere.client.vsan.base.impl.PbmDataProvider.getObjectCompatibleStoragePolicies(PbmDataProvider.java:118)

... 119 common frames omitted

Caused by: com.vmware.vim.vmomi.client.exception.SslException: com.vmware.vim.vmomi.core.exception.CertificateValidationException: Server certificate chain is not trusted and thumbprint doesn't match

at com.vmware.vim.vmomi.client.common.impl.ResponseImpl.setError(ResponseImpl.java:256)

at com.vmware.vim.vmomi.client.http.impl.HttpExchange.run(HttpExchange.java:56)

at com.vmware.vim.vmomi.client.http.impl.HttpProtocolBindingBase.executeRunnable(HttpProtocolBindingBase.java:226)

at com.vmware.vim.vmomi.client.http.impl.HttpProtocolBindingImpl.send(HttpProtocolBindingImpl.java:106)

at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl$CallExecutor.sendCall(MethodInvocationHandlerImpl.java:629)

...

1 Solution

Accepted Solutions
Highlighted
Expert
Expert

Looks like there is a mismatch between machine ssl and endpoints certs. Please open a SR with GSS.

View solution in original post

0 Kudos
26 Replies
Highlighted
Expert
Expert

Looks like there is a mismatch between machine ssl and endpoints certs. Please open a SR with GSS.

View solution in original post

0 Kudos
Highlighted
Contributor
Contributor

Vijay2027​ thanks for your quick response.  Unfortunately putting in a SR isn't an option right now.    Do you know how to check and/or manually fix the endpoint certs?

0 Kudos
Highlighted
Contributor
Contributor

I have the same issue, do you have a solution for that issue already?

Thank you!

0 Kudos
Highlighted
Contributor
Contributor

I wish I did.  When did these issues start for you?

For me, they first started when I tried using a machine cert created by Let's Encrypt.  There were problems with this certificate.  I ended up resetting all of the certificates.  Since doing this, I've been plagued by issues.

0 Kudos
Highlighted
Contributor
Contributor

I have the issues since I changed the VCSA certificate authority as a sub cert to my AD CA...

But I didn't tried to reset this until now.

Is it for you also working in the flash web client?

0 Kudos
Highlighted
Contributor
Contributor

Wow, surprisingly the flash client works!

I do see some errors in the /var/log/vmware/vsphere-client/logs/vsphere_client_virgo.log like:

[2019-10-28T11:05:28.557-07:00] [WARN ] http-nio-9090-exec-10         c.vmware.vim.vmomi.client.http.impl.AllowKnownThumbprintVerifier  Mismatched thumbprint 20:00:72:BA:3E:85:D4:93:A2:78:A4:83:62:2C:62:6C:4E:46:64:FF, rejecting connection

but it doesn't seem to affect the operation of the client

0 Kudos
Highlighted
Contributor
Contributor

I see this error as well with the flash client, but I'm not sure if this is because of the other issue or something the flash client is doing, because I'm getting the same errors also with pages that are working in the HTML 5 client.

I'm leaving the office for today but tomorrow I will try to reset the certificate back to an VCSA self singed one, maybe this will change something.

0 Kudos
Highlighted
Hot Shot
Hot Shot

Did you try this?

VMware Knowledge Base

Remember to make a backup (at least snasphot of your vCenter) before you do anything to be able to go back Smiley Happy

0 Kudos
Highlighted
Contributor
Contributor

Yes.  see the second sentence in the original message Smiley Happy

0 Kudos
Highlighted
Expert
Expert

Before we check endpoints can you move the contents of the below folders to a backup location (Ex: /storage/core) and restart vsphere-client and vsphere-ui services.

usr/lib/vmware-vsphere-client/server/work

/usr/lib/vmware-virgo/server/pickup

0 Kudos
Highlighted
Contributor
Contributor

thanks @Vijay2027,  I moved out /usr/lib/vmware-vsphere-client/server/work but I didn't have a folder called /usr/lib/vmware-virgo/server/pickup in my install.  no change after restart of both services.

root@vcenter [ /usr/lib/vmware-virgo/server ]# ls -la

total 116

drwxr-xr-x 10 root root  4096 Oct 28 10:02 .

drwxr-xr-x  3 root root  4096 Oct 10 05:57 ..

drwxr-xr-x  2 root root  4096 Oct 28 10:02 about_files

-rwxr-xr-x  1 root root  5588 May 30 11:30 About.html

-rwxr-xr-x  1 root root  3140 May 30 11:30 AboutKernel.html

-rwxr-xr-x  1 root root  4381 May 30 11:30 AboutNano.html

drwxr-xr-x  2 root root  4096 Oct 28 10:02 admin

-rwxr-xr-x  1 root root 14547 May 30 11:30 artifacts.xml

drwxr-xr-x  2 root root  4096 Oct 28 10:02 bin

drwxr-xr-x  4 root root  4096 Oct 28 10:02 configuration

-rwxr-xr-x  1 root root 12567 May 30 11:30 epl-v10.html

drwxr-xr-x  4 root root  4096 Oct 28 10:02 lib

-rwxr-xr-x  1 root root  8783 May 30 11:30 notice.html

drwxr-xr-x  4 root root  4096 Oct 10 05:57 p2

drwxr-xr-x  2 root root 12288 Oct 28 10:02 plugins

drwxr-xr-x  3 root root  4096 Oct 10 05:57 repository

-rwxr-xr-x  1 root root  3616 May 30 11:31 vmware-changes.txt

0 Kudos
Highlighted
Expert
Expert

I've sent you few commands in DM. Pls check.

0 Kudos
Highlighted
Contributor
Contributor

Thanks again for your help Vijay2027​ - you were right, it was definitely a complex and not self-fixable issue.  I had a number of services that were running with the wrong certificate.  GSS provided me with a script that fixed them all.

0 Kudos
Highlighted
Contributor
Contributor

Hi Vijay2027, could you please also send me the commands? I have exact the same problem.


Thank you in advance!

0 Kudos
Highlighted
Contributor
Contributor

Hi,

exactly same problem here, can i have the commands too please ?

0 Kudos
Highlighted
Expert
Expert

This is a complex process. GSS has automated the process to fix certificate mismatch. Please open a SR.

Yow will have to follow the process as per VMware Knowledge Base  to fix the mismatch.

0 Kudos
Highlighted
Contributor
Contributor

I finally found out that it is a HTML5 problem. I do not have this issue with the Flex client !

0 Kudos
Highlighted
Expert
Expert

The issue will be at service registration of HTML client. You will still have to get this corrected.

0 Kudos
Highlighted
Contributor
Contributor

reboot of vcenter and the problem disappeared...

0 Kudos