Highlighted
Contributor
Contributor

vCenter 7 error - no healthy upstream

Hi All,

I'm working on brand new ESXi 7.0.0 deployment and have run into an issue when deploying vCenter 7.

When browsing to the vCenter UI I receive a "no healthy upstream" error.

I can access vCenter Server Management without issue.

I've done some googling and have been pointed towards certificate issues however my Leaf and Root certs are showing as valid.

All of the services which are set to Automatic start are running without any errors or warnings.

Is there a documented fix for this issue?

Tags (1)
30 Replies
Highlighted
Contributor
Contributor

I just came across this issue. This happened to be an issue with Chrome. I powered off the vcsa upgraded it to run hw for esxi 7 and changed the os to Photo - I don't think this had anything to do with it but I recommend everyone do this because it runs probably about 10x smoother... boot time is about 4 minutes less. But I then tried it again in chrome no go loaded up IE 11 and it went in just fine. I cleared out my cache and history from chrome and restored to defaults and it then allowed me back in. I can't say for certain if loading it in IE assisted anything, but have to put it in there because it was the steps I took.

0 Kudos
Highlighted
Enthusiast
Enthusiast

It's likely DNS. You need a forward A record and a PTR record before you deploy. I see lots of people missing the PTR.  If you were missing a PTR record, and you simply add one, it won't magically start working, even if you restart vCenter.

However if you need to add your PTR record after-the-fact, you can simply Edit the network settings in vCenter Management (no need to make changes) and that will force the network settings to refresh, and then it will work.

Highlighted
VMware Employee
VMware Employee

I'll add to some of the recent commentary. I recently had a power outage in my homelab. Upon starting up the hosts, my vCenter auto-started. However, the DNS server did not. vCenter was not happy and I had the no healthy upstream error. After a little investigation, I identified the issue with the VM hosting DNS being offline and brought it back online. Then, I restarted the VCSA using the VAMI. I still got the no healthy upstream error for a couple of minutes... and then, after a page refresh, the UI was there and everything was working well.

As I was troubleshooting the error (prior to realizing the DNS server was not running), I noticed that the vAPI Endpoint was showing Healthy - With Errors (or something similar). I don't see that any longer.

For those that may find this page in the future, make sure your environment supports:

  • FQDN resolution (ex: vcenter.domain.com)
  • Short resolution (ex: vcenter)
  • Reverse resolution (ex: if vCenter's IP is 192.168.86.221, then  221.86.168.192.in-addr.arpa should resolve to vcenter.domain.com)
  • IP-based routing between nodes in the network

In my instance, without DNS existing, vCenter lost its mind and couldn't find what it needed to. Bringing DNS back online enabled FQDN, Short, and Reverse resolution options!

0 Kudos
Highlighted
Enthusiast
Enthusiast

However if you need to add your PTR record after-the-fact, you can simply Edit the network settings in vCenter Management (no need to make changes) and that will force the network settings to refresh, and then it will work.

This was the fix for me.  Thank you for saving me a bunch of time... I was a couple google searches away from redeploying another version or something like that Smiley Happy

0 Kudos
Highlighted
Contributor
Contributor

Yeah this is definitely a DNS issue. You can flush your DNS on your local box first. If that didn't work, verify check out to verify that you have forward and reverse records. A and PTR records are required.

0 Kudos
Highlighted
Contributor
Contributor

This problem may be caused by DNS,try to create a DNS server and resolve your vCenter FQDN.If you don't have a DNS server,set the DNS of vCenter VM to 0.0.0.0,keep hostname as localhost。Good luck!

0 Kudos
Highlighted
Contributor
Contributor

Check the diskspace of the system. I my case the /storare/log volume was completely full. This cause the "no healthy upstream" error.

In particular my /storage/log/vmware/pod dir was full of pod-startup.log.xxxx files. I deleted them all, and the server started correctly.

0 Kudos
Highlighted
Contributor
Contributor

I faced the same error, you must make new deployment, synchronize your vCenter to reliable NTP server, and use DNS synchronized to the same NTP server. Reliable time synchronization fix the problem. Please check this   vCenter 7 deployment failing and "no healthy upstream" forever error - System Admin Labs

0 Kudos
Highlighted
Contributor
Contributor

Well, I ran into the same issue, but waited a bit and it started working.  I have done what others have stated by logging onto the vCenter management on port 5480 and checked on the performance.

CPU was pegged while I was getting the "no healthy upstream" message.  From the graph shown below, it took 5 minutes before the CPU settled down and I was able to log onto the vCenter on 443.  The error message could be more useful.  I don't know what a healthy upstream is LOL, but there you have it.  I didn't have to change the vHW version or change the OS to Photon. 

genevt_1-1605307100839.png

 

 

Just because you can, should you?
0 Kudos
Highlighted
Contributor
Contributor

0 Kudos
Highlighted
Contributor
Contributor

Great, that fixed my error too!

My DNS settings on the DNS server for the vCenter are correct, but the primary DNS was pointing to 127.0.0.1 on vCenter. I need to run /opt/vmware/share/vami/vami_config_net command on vcenter via SSH, select option 4, changed the primary DNS IP from 127.0.0.1 to a proper DNS server IP, then rebooted vCenter and Worked!!!!

0) Show Current Configuration (scroll with Shift-PgUp/PgDown)
1) Exit this program
2) Default Gateway
3) Hostname
4) DNS
5) Proxy Server
6) IP Address Allocation for eth0
Enter a menu number [0]: 4

0 Kudos