VMware Cloud Community
SSpiers
Contributor
Contributor

vCenter 7 error - no healthy upstream

Hi All,

I'm working on brand new ESXi 7.0.0 deployment and have run into an issue when deploying vCenter 7.

When browsing to the vCenter UI I receive a "no healthy upstream" error.

I can access vCenter Server Management without issue.

I've done some googling and have been pointed towards certificate issues however my Leaf and Root certs are showing as valid.

All of the services which are set to Automatic start are running without any errors or warnings.

Is there a documented fix for this issue?

75 Replies
ESXi_Since_3_5
Contributor
Contributor

I just came across this issue. This happened to be an issue with Chrome. I powered off the vcsa upgraded it to run hw for esxi 7 and changed the os to Photo - I don't think this had anything to do with it but I recommend everyone do this because it runs probably about 10x smoother... boot time is about 4 minutes less. But I then tried it again in chrome no go loaded up IE 11 and it went in just fine. I cleared out my cache and history from chrome and restored to defaults and it then allowed me back in. I can't say for certain if loading it in IE assisted anything, but have to put it in there because it was the steps I took.

Reply
0 Kudos
welder314
Enthusiast
Enthusiast

It's likely DNS. You need a forward A record and a PTR record before you deploy. I see lots of people missing the PTR.  If you were missing a PTR record, and you simply add one, it won't magically start working, even if you restart vCenter.

However if you need to add your PTR record after-the-fact, you can simply Edit the network settings in vCenter Management (no need to make changes) and that will force the network settings to refresh, and then it will work.

thevirtualbill
VMware Employee
VMware Employee

I'll add to some of the recent commentary. I recently had a power outage in my homelab. Upon starting up the hosts, my vCenter auto-started. However, the DNS server did not. vCenter was not happy and I had the no healthy upstream error. After a little investigation, I identified the issue with the VM hosting DNS being offline and brought it back online. Then, I restarted the VCSA using the VAMI. I still got the no healthy upstream error for a couple of minutes... and then, after a page refresh, the UI was there and everything was working well.

As I was troubleshooting the error (prior to realizing the DNS server was not running), I noticed that the vAPI Endpoint was showing Healthy - With Errors (or something similar). I don't see that any longer.

For those that may find this page in the future, make sure your environment supports:

  • FQDN resolution (ex: vcenter.domain.com)
  • Short resolution (ex: vcenter)
  • Reverse resolution (ex: if vCenter's IP is 192.168.86.221, then  221.86.168.192.in-addr.arpa should resolve to vcenter.domain.com)
  • IP-based routing between nodes in the network

In my instance, without DNS existing, vCenter lost its mind and couldn't find what it needed to. Bringing DNS back online enabled FQDN, Short, and Reverse resolution options!

Reply
0 Kudos
nary4484
Enthusiast
Enthusiast

However if you need to add your PTR record after-the-fact, you can simply Edit the network settings in vCenter Management (no need to make changes) and that will force the network settings to refresh, and then it will work.

This was the fix for me.  Thank you for saving me a bunch of time... I was a couple google searches away from redeploying another version or something like that Smiley Happy

Reply
0 Kudos
ESXi_Since_3_5
Contributor
Contributor

Yeah this is definitely a DNS issue. You can flush your DNS on your local box first. If that didn't work, verify check out to verify that you have forward and reverse records. A and PTR records are required.

Reply
0 Kudos
CPLASFV
Contributor
Contributor

This problem may be caused by DNS,try to create a DNS server and resolve your vCenter FQDN.If you don't have a DNS server,set the DNS of vCenter VM to 0.0.0.0,keep hostname as localhost。Good luck!

Reply
0 Kudos
ipasipas
Contributor
Contributor

Check the diskspace of the system. I my case the /storare/log volume was completely full. This cause the "no healthy upstream" error.

In particular my /storage/log/vmware/pod dir was full of pod-startup.log.xxxx files. I deleted them all, and the server started correctly.

Reply
0 Kudos
moheidb2030
Contributor
Contributor

I faced the same error, you must make new deployment, synchronize your vCenter to reliable NTP server, and use DNS synchronized to the same NTP server. Reliable time synchronization fix the problem. Please check this   vCenter 7 deployment failing and "no healthy upstream" forever error - System Admin Labs

Reply
0 Kudos
genevt
Contributor
Contributor

Well, I ran into the same issue, but waited a bit and it started working.  I have done what others have stated by logging onto the vCenter management on port 5480 and checked on the performance.

CPU was pegged while I was getting the "no healthy upstream" message.  From the graph shown below, it took 5 minutes before the CPU settled down and I was able to log onto the vCenter on 443.  The error message could be more useful.  I don't know what a healthy upstream is LOL, but there you have it.  I didn't have to change the vHW version or change the OS to Photon. 

genevt_1-1605307100839.png

 

 

Reply
0 Kudos
ojw_nc
Contributor
Contributor

MarcusZheng
Contributor
Contributor

Great, that fixed my error too!

My DNS settings on the DNS server for the vCenter are correct, but the primary DNS was pointing to 127.0.0.1 on vCenter. I need to run /opt/vmware/share/vami/vami_config_net command on vcenter via SSH, select option 4, changed the primary DNS IP from 127.0.0.1 to a proper DNS server IP, then rebooted vCenter and Worked!!!!

0) Show Current Configuration (scroll with Shift-PgUp/PgDown)
1) Exit this program
2) Default Gateway
3) Hostname
4) DNS
5) Proxy Server
6) IP Address Allocation for eth0
Enter a menu number [0]: 4

billdossett
Hot Shot
Hot Shot

@welder314 

bingo - forgot to add a new reverse zone for the new vlan subnet I'm building on so no PTR recs.  Created it and edited the network as you suggested which looked fine and then said can't connect to vcenter server - but it had finished and I can now login to my spanking new VCSA, thanks!

Bill Dossett
Reply
0 Kudos
cleston_tellis
Contributor
Contributor

added this record in the host file /etc/hosts/ to fix the issue...

127.0.0.1 (FQN name here vcsa.***.local) vcsa localhost

oparnes
Contributor
Contributor

What do you mean redeploy?

reinstall on the existing VM ?

or deploy a new one from fresh?

Reply
0 Kudos
Mohamed2233
Contributor
Contributor

Hello, I have tested this in a lab environment; the error will be fixed when using an external NTP server during installation. And everything will run smoothly. Please check this:

https://www.systemadminslabs.com/2020/11/22/vmware-vcenter-7-server-deployment-failing-and-no-health...

 

Reply
0 Kudos
HostaHost
Contributor
Contributor

Wow, I just caused a VCSA failover and this 'no healthy upstream' message by performing the incredibly dangerous operation of disconnecting the vcsa install DVD by switching from datastore ISO file to client device.  Seriously; 7.0 is so fragile that disconnecting a DVD causes a complete failure and HA failover... unreal.  Fortunately it did fail successfully so the interface came back a few minutes later.

Reply
0 Kudos
kronoz19
Contributor
Contributor

thank you, this option solved mi issues 

 

Nombre DNS:localhost

DNS: 127.0.0.1

 

Reply
0 Kudos
sean468
Contributor
Contributor

Thanks mate, this is exactly what fixed mine. 

Reply
0 Kudos
BinurajKR
Contributor
Contributor

This issue can be happened if you attempted to rename the hostname of vCenter server from console. if this is the case, add host and PTR record in DNS and then change the hostname from VAMI interface (under networking). this will reboot the appliance and then the issue will be solved. Hopes this helps.

Reply
0 Kudos
jc0ke
Contributor
Contributor

Reply
0 Kudos