Hi Guys
We've been Experiencing this error / bug / problem in vCenter 7 whenever we do the ff:
- Test High Availability (VMs restarts on the other server except for the vCenter Server)
After powering on the esxi host where the vCenter is residing and the vCenter Server the error occurs
Tried the ff: re-installing, checking VM Hardware Version, checking the OS: no Photon OS in the list. restarting services but nothing works
in the end we need to restore our back-up.
Screen shot below
Thanks!
Could you open the VAMI interface of the vCenter server? (https://vc_ip:5480) If it response, please check all VCSA services status ...
Don't try to restore a backup immediately or it it's not possible, please keep a original corrupted VM of the VCSA for investigation the root of problem
Could you please take a putty session to the vCSA and run the below command to confirm if you have any space issues,
hostname -f && date && uptime && vpxd -vl && chage -l root && df -h && service-control --status.
if there is no issues with the space then please use the below article to find if the sts certificate is expired,
Please feel free to raise a case with VMware support team to help you if you need an extra hand.
Thanks.
Please like and mark this as correct solution, if this helps you to solve your problem.
Hi,
Today I had the same incident, and many automatic vcsa services were stopped
services were starting slowly and the vcsa cpu was 100%
it has taken a long time to function normally, we will have to monitor this incidence
In my case, "no healthy upstream" was caused by expired certificates. After starting STS service, login issues persist - "user name and password required". After rebooting, numerous services failed to start and i received: Exception in invoking authentication handler [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: certificate has expired (_ssl.c:1076). I followed the first link to regenerate certs - option 8. Followed the second link to regenerate STS cert. Rebooted and everything works!
I have the same problems with a fresh installation of VCSA 7.0.0c.
I gave the VM more CPUs and waited 30 minutes after booting, but I still get the message "no healthy upstream".
The service "vAPI-Endpoint" reports the error "Failed to connect to f592d7af-0df2-4c7f-8e29-a9f03efd7a34 \ com.vmware.vcenter.nsxd.vapi vAPI provider."
A restart doesn't help either. What should I do?
Hi,
Check your vCenter hosts file.
If it looks like this
10.200.111.25 vcenter.mydomain.local vcenter
10.200.111.25 vcenter.mydomain.local vcenter
then backup it first and after that change it to this
127.0.0.1 vcenter.mydomain.local vcenter localhost
::1 vcenter.mydomain.local vcenter localhost ipv6-localhost ipv6-loopback
Reboot the vCenter and check if you can access it
This is what support did when i raised a ticket.
Regards,
foxmr
Usually, we see this message in the web client while all the services are starting. So wait for some time and check the web client.
If you still see the same message in the web client, try checking the status of the services from the command-line (service-control --status --all) or through VAMI (https://vc-ip:5480).
If all the services are not UP and running, the best way is to restart all the services (service-control --stop --all) & (service-control --start --all). Now when all the services are starting VC should throw an error if there are any potential issues.
It worked for me.
Thanks foxmr for sharing
I faced the same problem I make new deployment i synchronize vCenter installation to public NTP servers, and synchronize DNS to the same NTP server, the deployment succeed and vCenter works in timely manner after restart.
vCenter 7 deployment failing and "no healthy upstream" forever error - System Admin Labs
Same problem I think. Expired certificate.
But in VCSA can't start vmware service etc. Error: Service crashed while starting
Great bug vmware!
Thanks to foxmr, I confirm that editing hosts file did the trick for me. everything works fine now
Thanks!! This worked for me.
Thanks a lot! I just wanted to reply that this worked for me as well.
I only added the 127.0.0.1 line to my /etc/hosts file and rebooted, that did the trick.
注意:如果登录vCenterServer 是页面出现“no healthy upstream”字样,则是vCenterServer服务没有完全启动,稍等一段时间,再次登录即可,或登录管理页面:https://vc_ip:548,找到“服务”,手动将没有自动启动的服务启动即可。导致vCenterServer服务没有完全启动的原因可能是域名服务器不能连接,或NTP时间配置不当等。
no healthy upstream vCenter 7.0 error:
Issue fixed:
after 3.5 hrs of quick search on various blogs/vmware communities/mylab work/organization cook books. found the solution.
Steps tried out:
1. VAMI <IP add:5480> - Stop-Start all Services.
2. Stopped temp Antivirus solution on local host also stopped Windows Defender Firewall.
3. VAMI <https://IP add:5480> click on TIME - change Time Zone to UTC also Time Sync - Mode - set it to Disabled.
4. upgraded VM hardware to version 18. <virtualhw.version = "18">
5. created a new host "A" record FLZ (forward lookup zone) on DNS Server for vcsa <192.168.40.110 vcsa.mylab.com>
6. created Pointer (PTR) record (Reverse lookup zone) on DNS Server for vcsa.<192.168.40.X vcsa.mylab.com>
7. restarted DNS server.
8. restarted vcsa 7.0.1 appliance (VM).
9. cleared web browser cache, tried multiple browsers thinking it can be a problem but,
didn't work all above steps <1 to 9>. (wasted my time though but found a way to fix it by editing " host file" on my local PC [DELL SERVER].
Added below entries on " host file"
Path - C:\Windows\System32\drivers\etc
###########vcenter###################
192.168.40.110 vcsa.mylab.com
192.168.40.100 dc01.mylab.com
192.168.40.110 https://vcsa.mylab.com:5480
192.168.40.110 https://192.168.40.110:5480
127.0.0.1 vcsa.mylab.com
rebooted the vCenter Server and then try to access it and it " Worked fine" for me. thanks.
regards,
vcloud3d
Hello, I have tested this in a lab environment; the error will be fixed when using an external NTP server during installation. And everything will run smoothly. Please check this:
感谢foxmr的分享,按照你的方法,我的环境已经正常工作,多谢多谢
I had similar issue and this is what I have done to fix it.
Problem: When I try to access vCenter Server URL, It gives an error "no healthy upstream"
I remember it was accessible few days ago and no configuration changes has been done for vCenter
Check: When I checked certificate, I saw that it was expired 2 days ago. (All certs were self signed)
Fix: I regenerated all certs by doing steps written at How to regenerate vSphere 6.x and 7.x certificates using self-signed VMCA 2112283: https://kb.vmware.com/s/article/2112283
Note: You may also have similar issue if your VMCA date/time is not correct. For this case, you can try to correct date/time, use NTP servers for VMCA etc.