no healthy upstream error

[TimGlen]

Hi Folks,  

I've seen several others post about this same error and I've tried \ and checked several things but I don't see a resolution based on the posts I've read. 

I have vCenter Version: 7.0.1.00200 ,  Build number:  17327517.

I am able to login to the vCenter Server GUI and SSH without problems. When I try and login to vCenter I receive the no healthy upstream.

Originally when I discovered this my log partition was full.   I followed a KB and cleaned it up.  No other partitions are over 50% util.  DNS works, NTP is configured and time is accurate.

I have a lot of services that aren't started but I **think** at least some of them should be.

When I try and start them using service-control I get some started then an error.

I'd greatly appreciate any tips or guidance. 🙂   Thanks!

Tim

[ptarnawski]

Hey @TimGlen ,

If you have that many services not running, give the appliance a reboot. If this wont help log SR with VMware, 

[a_p_]

What's in the logs?

Did you already check the certificates (https://kb.vmware.com/s/article/82332), i.e. whether one or more are expired.

André

[TimGlen]

Thanks @ptarnawski , but I've rebooted this appliance several times and similar results.  

[TimGlen]

@a_p_   Thanks for replying! 

I have looked at the certs and some of the certs that are backup are expired but that's it.  I don't believe that should be a problem.  See the attached screenshot for details. 

About the logs.     service-control --start --all   stops and errors while trying to start vpxd-svcs so I've cat and less that log , grepped for error and other things but honestly, I have no idea what I'm looking for and while I do see some errors I don't know what is relevant.   

I've zipped and attached the current vpxd & vpxd-svcs log files.  I would greatly appreciate another set of eyes on them if that is the proper direction to go or any other guidance.     Thanks folks! 

[a_p_]

Not sure, but there are a lot of errors regarding an invalid, and expired certificate.

2022-08-05T00:05:36.507-04:00 [Thread-13 ERROR com.vmware.vim.sso.client.impl.SoapBindingImpl opId=] Error communicating to the remote server https://vcenter.theglens.net/sts/STSService/vsphere.local
com.sun.xml.internal.ws.client.ClientTransportException: HTTP transport error: javax.net.ssl.SSLHandshakeException: java.security.cert.CertificateException: No subject alternative DNS name matching vcenter.theglens.net found.

2022-08-03T21:38:54.400-04:00 error vpxd[11710] [Originator@6876 sub=vmomi.soapStub[10469]] Resetting stub adapter for server <cs p:00007f83308d7f50, TCP:vcenter.theglens.net:443> : service state request failed: N7Vmacore3Ssl18SSLVerifyExceptionE(SSL Exception: Verification parameters:
--> PeerThumbprint: 89:94:99:91:6E:F3:FB:9C:EB:84:B8:A0:F7:9F:31:CA:66:77:8F:3E
--> ExpectedThumbprint: 
--> ExpectedPeerName: vcenter.theglens.net
--> The remote host certificate has these problems:
--> 
--> * certificate has expired)

Please check whether the certificate for the mentioned FQDN has recently been replaced, or has an issue.

Although less likely, you may also want to ensure that the STS certificate is ok (see https://kb.vmware.com/s/article/79248)

André

[ptarnawski]

Hey @TimGlen 

I was just about to write that it can be STS cert, but @a_p_ had already done it. I will add three articles related to that issue, maybe follow to check on STS:

https://angrysysops.com/2021/07/05/how-to-check-if-sts-certificate-is-about-to-expire-or-expired-alr...

https://angrysysops.com/2021/05/19/sts-certificate-expiration-signing-certificate-is-not-valid-error...

https://angrysysops.com/2022/03/01/refresh-a-vcenter-server-sts-certificate-using-the-vsphere-client...

[TimGlen]

Thank you both for your help.

This is the output from checksys.py,  looks like the STS certs are valid. 

I’m sorry, I should have mentioned this earlier.   

After I fixed the log partition out of space the errors persisted. At that time on August 4, I realized the Machine Cert had expired. At that time I followed the doc below

https://kb.vmware.com/s/article/2112283

I did receive an error during that process. The log \ error is below. 

The service-control.log from that time period is below. 

I'm uploading the certificate-manager.log to this message. 

Again, I really appreciate your assistance. 

Tim