ocsitspecialist
Contributor
Contributor

503 Service Unavailable (Failed to connect to endpoint:

Jump to solution

Error Message:

503 Service Unavailable (Failed to connect to endpoint: [class Vmacore::Http::LocalServiceSpec:0x000001d47454d440] _serverNamespace = /ui action = Allow _port = 5090)

error.png

Hello guys good evening, I was encountered this error on my host machine when accessing the vcenter server with vSphere Client (HTML5) - partial functionality, any solution or recommendation to solve my current issues on my vCenter Server 6.5 thank you guys and GOD BLESS US ALL...

1 Solution

Accepted Solutions
SAPTCO
Contributor
Contributor

HI,

This is Link VMware Knowledge Base  Solve My Issue Completely.

BR,,

View solution in original post

50 Replies
LWCSteve
Contributor
Contributor

I am also having this issue after rebooting VCenter Server. service-control shows the following services as stopped. I am having no luck starting them.

Stopped:

EsxAgentManager VMWareCAMService mbcs vapiEndpoint vmware-autodeploy-waiter vmw

are-imagebuilder vmware-network-coredump vsphere-ui vspherewebclientsvc

0 Kudos
Joseph80
Enthusiast
Enthusiast

Hi, you need to just restart vmware WebClient service from vcenter server and wait for 8 to 10 minutes.

pbraren
Hot Shot
Hot Shot

I've seen the same issue when trying to browse to VCSA:

https://vcsa.lab.local/

after a simple restart of the host/appliance, despite proper graceful shutdown/reboot.

The other symptoms include a blank inventory in vSphere Web Client and (HTML5) vSphere Client (pictured below), even though everything was working fine before the reboot.

Trying the suggestion about restarting the WebClient service:

service-control --stop vsphere-client

service-control --start vsphere-client

doesn't seem to have helped, even after waiting 15 minutes.

So still seeking:

a) a way to fix this, short of rebuilding the VCSA appliance (did that once, worked, then after reboot, problem came right back)

b) a way to prevent this issue from happening in the first place

Hope this information helps.

2016-11-27_9-10-41.jpg2016-11-27_9-10-28.jpg

TinkerTry.com
LWCSteve
Contributor
Contributor

Trying to manually start the web client service results in an error also. This is extremely frustrating.

0 Kudos
pbraren
Hot Shot
Hot Shot

I did use the handy "Create Support Bundle" button in VAMI

https://vcsa:5480/#/appliance/summary?locale=en

to download a vm-support.tgz

TinkerTry.com
0 Kudos
pbraren
Hot Shot
Hot Shot

I haven't worked too hard on this one yet, was frankly hoping somebody else had figured it out by now.

I wonder how prevalent this is. Got nothing on the twitters either:

Paul Braren on Twitter: "Starting to look at VCSA 6.5 "503 Service Unavailable (Failed to connect to...

This issue consistently bit me on 4 fresh builds, and once on upgrade from 6.0U2.

Using MAC address DHCP reservation with proper with forward and reverse lookup and FQDN, but happens even with hard-coded name.

So I haven't really defined the problem yet, nevermind found a solution.

TinkerTry.com
0 Kudos
pbraren
Hot Shot
Hot Shot

My initial impression was that this error seemed to be related to my shutdown technique that first time this issue hit me, which might be because I hadn't been sure to wait long enough for the VCSA appliance to complete its graceful shutdown. But if that were the case, it would be worrisome that it wouldn't recover.

This issue with VCSA is turned out to be repeatable, even with waiting longer for VCSA to reboot. I'll still want to try this VCSA deploy again with a fixed IP ESXi host and fixed IP VCSA appliance, using no names for anything, and see how that goes, to help rule out DNS/DHCP/etc.

Meanwhile, I noticed that this issue could be related to a Platform Services Controller issue, that is cropping up only after first reboot, for not-yet-determined reasons, something VCSA 6.0U2 wasn't apparently  prone to in my same exact network and physical hardware environment.

How did I determine PSC might be at play? The hint was right there, in my browser, when visiting the VCSA vSphere Web Client (where vcsa.lab.localhost is the name of my vcsa appliance):

https://vcsa.lab.local/vsphere-client/?csp

I notice this error, in the yellow area near the top:

Could not connect to one or more vCenter Server systems:

https://vcsa.lab.local:443/sdk

If I visit that URL, I get:

503 Service Unavailable (Failed to connect to endpoint: [N7Vmacore4Http16LocalServiceSpecE:0x00007f98680bcd40] _serverNamespace = /sdk action = Allow _port = 8085)

Looking for that error, I find this recently updated KB article that actually lists VCSA 6.5:

“Failed to verify the SSL certificate for one or more vCenter Server Systems” error in the vSphere W...

That said, the suggested fix doesn't seem to work, since I'm not seeing a "duplicate service endpoint" that the KB article suggests I look for.

2016-12-03_1-57-29.jpg

TinkerTry.com
ebrodje
Contributor
Contributor

Im also having this issue but i could not even start the webclient service it gave me a class error aswell.

0 Kudos
N3tw0rk99
Contributor
Contributor

I have the same error for the html5 interface.  On a fresh vcenter 6.5 install.  The html5 \ui interface has never worked on my system

0 Kudos
pbraren
Hot Shot
Hot Shot

There's enough people here that I feel more comfortable sharing this unlisted YouTube video I created,  showing my simple recreate procedure, now that I know I'm not alone with this strange Photon OS VCSA 6.5 behavior I'm seeing that wasn't there with SuSE-based VCSA 6.0U2:

https://youtu.be/gjls1PcxiWY

This is shared to hopefully help move this along toward resolution more rapidly, hoping to find what's different about my environment than the many successful VCSA deployments out there that handle restarts just fine.

I will hopefully have a chance to try this testing again with fixed IPs this weekend, to avoid DNS/DHCP implication, will see.

TinkerTry.com
0 Kudos
LWCSteve
Contributor
Contributor

We now have a support ticket open regarding this. As soon as we get a resolution, I will update here.

0 Kudos
LWCSteve
Contributor
Contributor

Our issue has now been resolved and our web client is now fully functional again.

It appears the following accounts need to be added to "Log on as a Batch job".

GPO Path: Computer Configuration -> Windows Settings ->Security Settings ->Local Policies -> User Rights Assignment ->Log on as a batch job

Accounts:

  cm

  content-library

  eam

  imagebuilder

  mbcs

  netdumper

  perfcharts

  rbd

  vapiEndpoint

  vmware-vpostgres

  vsan-health

  vsm

  vsphere-client

  vsphere-ui

gpo.png

I think the installer is supposed to add the accounts but if this policy is set by a GPO, it is unable to modify it locally.

I hope this helps some of you.

pbraren
Hot Shot
Hot Shot

I seem to have fixed my issue by simply typing in a System name in the dialogue pictured below, instead of leaving it blank, which relied on the DHCP reservation/DNS entry in my router that worked under 6.0U2.

I've now survived 5 ESXi reboots, using ESXi host reboot, which was set to shut down all VMs gracefully, and to restart VCSA automatically.

I will continue to test and reboot, but first, a flight to catch, with my server onboard, of course 😉

Here's how I was doing 6.0U2 or 6.5 installs, choosing DHCP, and leaving System name blank (but setting up a DHCP reservation for the VM's MAC address, and setting up a DNS name, by suspending the VCSA 6.5 appliance moments after the power up button appears, and grabbing the fresh  VM's MAC, then resuming)

2016-12-08_12-37-19.png

I don't use DDNS:

2016-12-08_12-37-50.png

but figured I'd try to put in a System name anyway:

2016-12-08_2-53-58.png

Five reboots later, I seem to be good, used to fail on first or second reboot/restart of VCSA.

TinkerTry.com
pbraren
Hot Shot
Hot Shot

3 reboots later, still good, feeling more and more like this issue is behind me now.

TinkerTry.com
0 Kudos
pbraren
Hot Shot
Hot Shot

Many reboots since, all is still working fine, this issue appears to now be completely resolved for me.

TinkerTry.com
0 Kudos
FSRWalt
Contributor
Contributor

LWCSteve, Adding those accounts to my GPO did the trick for me!

After applying the GPO I restarted the service with service-control --stop vsphere-client then a start - waited a few minutes, killed a chicken as a sacrifice to VMWare and was able to get into the Web Client.

(I think the sacrifice is optional, not sure if it was needed.)

Thanks so much!  I've been at this for over a week!

--Walt

0 Kudos
vmjoe
Enthusiast
Enthusiast

LWCSteve

Thanks for the solution! Works for me on 2012 R2 server with vCenter 6.5 (build 4602587).

I didn't use group policy - we have a group on the server where I had to add all the local users created by vCenter installation.

It's a shame that VMware is creating such rubbish... and from the error message in the browser it's impossible to even get a clue what's the problem. 😞

0 Kudos
lockdwn
Contributor
Contributor

Thank you!  I have been having issues with this for a while. 

0 Kudos
tdesai
Contributor
Contributor

Would you mind filing a PR for this so that I can have the appropriate team follow up on this ?

Kindly let me know the PR details once you have it

Thanks

Tushar Desai

0 Kudos