Error Message:

503 Service Unavailable (Failed to connect to endpoint: [class Vmacore::Http::LocalServiceSpec:0x000001d47454d440] _serverNamespace = /ui action = Allow _port = 5090)

Hello guys good evening, I was encountered this error on my host machine when accessing the vcenter server with vSphere Client (HTML5) - partial functionality, any solution or recommendation to solve my current issues on my vCenter Server 6.5 thank you guys and GOD BLESS US ALL...

I am also having this issue after rebooting VCenter Server. service-control shows the following services as stopped. I am having no luck starting them.

Stopped:

EsxAgentManager VMWareCAMService mbcs vapiEndpoint vmware-autodeploy-waiter vmw

are-imagebuilder vmware-network-coredump vsphere-ui vspherewebclientsvc

Below article helped me

Tech Kiranangal: VMware vCenter 6.0 : 503 Service Unavailable: Failed to connect to endpoint

Hi, you need to just restart vmware WebClient service from vcenter server and wait for 8 to 10 minutes.

I've seen the same issue when trying to browse to VCSA:

https://vcsa.lab.local/

after a simple restart of the host/appliance, despite proper graceful shutdown/reboot.

The other symptoms include a blank inventory in vSphere Web Client and (HTML5) vSphere Client (pictured below), even though everything was working fine before the reboot.

Trying the suggestion about restarting the WebClient service:

service-control --stop vsphere-client

service-control --start vsphere-client

doesn't seem to have helped, even after waiting 15 minutes.

So still seeking:

a) a way to fix this, short of rebuilding the VCSA appliance (did that once, worked, then after reboot, problem came right back)

b) a way to prevent this issue from happening in the first place

Hope this information helps.

I did use the handy "Create Support Bundle" button in VAMI

https://vcsa:5480/#/appliance/summary?locale=en

to download a vm-support.tgz

Have a look in  the datastore space as well, we found the similar issue and when checked the datastore have 0% of free space left.

After making some breathing space, the VC start responding.

Waiting 8 to 10 minutes seems to be working for me. First I was seeing the 503 Service Unavailable error, then the /vsphere-client/ endpoint got offline and finally I saw this:

After couple more minutes i was able to login in :smileyhappy:

Cheers,

Lucas

This worked for me, make sure to start the perf charts service after as well.

Weird that it takes some time after restarting the services but it appears to... just have patience.

It works for all Port related issues, thanks for suggestion.

Trying to manually start the web client service results in an error also. This is extremely frustrating.

I haven't worked too hard on this one yet, was frankly hoping somebody else had figured it out by now.

I wonder how prevalent this is. Got nothing on the twitters either:

Paul Braren on Twitter: "Starting to look at VCSA 6.5 "503 Service Unavailable (Failed to connect to endpoint:" browser …

This issue consistently bit me on 4 fresh builds, and once on upgrade from 6.0U2.

Using MAC address DHCP reservation with proper with forward and reverse lookup and FQDN, but happens even with hard-coded name.

So I haven't really defined the problem yet, nevermind found a solution.

My initial impression was that this error seemed to be related to my shutdown technique that first time this issue hit me, which might be because I hadn't been sure to wait long enough for the VCSA appliance to complete its graceful shutdown. But if that were the case, it would be worrisome that it wouldn't recover.

This issue with VCSA is turned out to be repeatable, even with waiting longer for VCSA to reboot. I'll still want to try this VCSA deploy again with a fixed IP ESXi host and fixed IP VCSA appliance, using no names for anything, and see how that goes, to help rule out DNS/DHCP/etc.

Meanwhile, I noticed that this issue could be related to a Platform Services Controller issue, that is cropping up only after first reboot, for not-yet-determined reasons, something VCSA 6.0U2 wasn't apparently  prone to in my same exact network and physical hardware environment.

How did I determine PSC might be at play? The hint was right there, in my browser, when visiting the VCSA vSphere Web Client (where vcsa.lab.localhost is the name of my vcsa appliance):

https://vcsa.lab.local/vsphere-client/?csp

I notice this error, in the yellow area near the top:

Could not connect to one or more vCenter Server systems:

https://vcsa.lab.local:443/sdk

If I visit that URL, I get:

503 Service Unavailable (Failed to connect to endpoint: [N7Vmacore4Http16LocalServiceSpecE:0x00007f98680bcd40] _serverNamespace = /sdk action = Allow _port = 8085)

Looking for that error, I find this recently updated KB article that actually lists VCSA 6.5:

“Failed to verify the SSL certificate for one or more vCenter Server Systems” error in the vSphere Web Client (2050273) …

That said, the suggested fix doesn't seem to work, since I'm not seeing a "duplicate service endpoint" that the KB article suggests I look for.

Im also having this issue but i could not even start the webclient service it gave me a class error aswell.

I have the same error for the html5 interface.  On a fresh vcenter 6.5 install.  The html5 \ui interface has never worked on my system

There's enough people here that I feel more comfortable sharing this unlisted YouTube video I created,  showing my simple recreate procedure, now that I know I'm not alone with this strange Photon OS VCSA 6.5 behavior I'm seeing that wasn't there with SuSE-based VCSA 6.0U2:

https://youtu.be/gjls1PcxiWY

This is shared to hopefully help move this along toward resolution more rapidly, hoping to find what's different about my environment than the many successful VCSA deployments out there that handle restarts just fine.

I will hopefully have a chance to try this testing again with fixed IPs this weekend, to avoid DNS/DHCP implication, will see.

We now have a support ticket open regarding this. As soon as we get a resolution, I will update here.

can you share details on this ticket ??

Our issue has now been resolved and our web client is now fully functional again.

It appears the following accounts need to be added to "Log on as a Batch job".

GPO Path: Computer Configuration -> Windows Settings ->Security Settings ->Local Policies -> User Rights Assignment ->Log on as a batch job

Accounts:

  cm

  content-library

  eam

  imagebuilder

  mbcs

  netdumper

  perfcharts

  rbd

  vapiEndpoint

  vmware-vpostgres

  vsan-health

  vsm

  vsphere-client

  vsphere-ui

I think the installer is supposed to add the accounts but if this policy is set by a GPO, it is unable to modify it locally.

I hope this helps some of you.

I seem to have fixed my issue by simply typing in a System name in the dialogue pictured below, instead of leaving it blank, which relied on the DHCP reservation/DNS entry in my router that worked under 6.0U2.

I've now survived 5 ESXi reboots, using ESXi host reboot, which was set to shut down all VMs gracefully, and to restart VCSA automatically.

I will continue to test and reboot, but first, a flight to catch, with my server onboard, of course ;-)

Here's how I was doing 6.0U2 or 6.5 installs, choosing DHCP, and leaving System name blank (but setting up a DHCP reservation for the VM's MAC address, and setting up a DNS name, by suspending the VCSA 6.5 appliance moments after the power up button appears, and grabbing the fresh  VM's MAC, then resuming)

I don't use DDNS:

but figured I'd try to put in a System name anyway:

Five reboots later, I seem to be good, used to fail on first or second reboot/restart of VCSA.

3 reboots later, still good, feeling more and more like this issue is behind me now.

Many reboots since, all is still working fine, this issue appears to now be completely resolved for me.

I had this same issue on a 6.0 u2 windows machine last week after a restart.  If you so choose the installer creates many virtual accounts NT Service\VMware-thingx..etc.  That it uses to log on different services.   On the windows machine I like to use the service-control --start --all command in the \bin of the vcenter folder.  It starts all services or with --stop stops all services in the correct order.  I had mine fail on any vmware service that wasn't using system as checked in services.msc.

I attempted to use "local service" as I have two vcenters and this was working on anohter, but had the same result.  When I looked at the user rights "logon as a service" no longer had local service in the use rights.  TIP: when you make a GPO to give domain accounts specific user rights, be sure to add the defaults back in.  The gpo does not add the accounts to those user rights it replace whatever values are already there with the ones you set. 

After setting each of those services to the domain account....all is working properly.  Leave the services using SYSTEM account to logon alone.

Would you mind filing a PR for this so that I can have the appropriate team follow up on this ?

Kindly let me know the PR details once you have it

Thanks

Tushar Desai

LWCSteve, Adding those accounts to my GPO did the trick for me!

After applying the GPO I restarted the service with service-control --stop vsphere-client then a start - waited a few minutes, killed a chicken as a sacrifice to VMWare and was able to get into the Web Client.

(I think the sacrifice is optional, not sure if it was needed.)

Thanks so much!  I've been at this for over a week!

--Walt

LWCSteve‌

Thanks for the solution! Works for me on 2012 R2 server with vCenter 6.5 (build 4602587).

I didn't use group policy - we have a group on the server where I had to add all the local users created by vCenter installation.

It's a shame that VMware is creating such rubbish... and from the error message in the browser it's impossible to even get a clue what's the problem. :-(

Thank you!  I have been having issues with this for a while. 

What kind of sorcery is that? Good Lord! You deserve every cookie in the world for this! Not even VMware support could figure it out for me. If only I found this a couple of weeks earlier - would have saved me days of struggle with upgrade to 6.5! You rock! :smileycool:

It really help bro, thank you

Worked like a charm.

one thing to check, where is webclient service in Windows based vCenter 6.5.

Hello

I fixed same issue , but my customer infrastructure was like this. 

Basically this issue occurred because of your DB full or DNS have wrong entry.

2 PSC

1 VC ( with vPostgreSQL db).

Giving here step by step solution.

run hostname -f on vc and psc virtual machine  if you got this error ,

Unknown host

1. Log in to the console of vCenter Appliance as the root user.
2. To change directory to /etc , run this command:
   
   cd /etc
   
   
3. Open the hosts file using a text editor.
4. Add the hostname in vCenter_IP vCenter Server Appliance FQDN vcenter short name format.
   
   For example, 10.10.10.10 vcenter.vmware.com vcenter
5. Save and close the file.
6. Restart the vmware-vpxd service by running this command:

service vmware-vpxd restart

1. Issue was in DNS setting on PSC , login by root on GUI change DNS according to infrastructure , you can check by it nslookup from you virtual center server.

2. IP address of your PSC should be resolved on both ways ( forward lookup and reverse lookup)

3. Then try to restart vpxd service

4. service vmware-vpxd restart

May be it would failed as service initialized failed if yes

Shut down your PSC and restart

Shut down your VC and restart

Check vpxd status service vmware-vpxd status ( If not running)

Go to PSC/VM vm by putty and restart these service.

/etc/init.d/vmware-stsd restart

/etc/init.d/vmware-sts-idmd restart

service vmware-cm restart

then go to vc vm by putty

and restart

service vmware-vpxd restart

Now you should be connect by client.

If you have still issue send me log file.

/var/log/vmware/vpx/vpxd.log

Hi,

After looking at all the issues in this thread, I still had the 503 problem.

I'm running the Photon-based VCSA Appliance 6.5.  I did the Photon update, and upgraded to the latest version, 6.5.0.5300.

After a reboot, that did not solve my issue.  I again logged into the management appliance, and looked at the settings.  Following the helpful video on the DNS/DHCP reservation, which did not apply to me, I began to wonder if network settings where the issue. 

In my settings, I found that IPv4 was enabled, but IPv6 was turned off.  On a hunch, I enabled IPv6, and a few minutes later the /ui and vSphere 6.5 web client started working again. Great! 

On a reboot, I found Photon did not keep any of the IPv6 settings, and again, the 503 error was thrown, no matter how long I waited.  I logged into the appliance via the console, and tried to set IPv6 settings manually.  This did not work.  There's some modification to the configuration to prevent setting IPv6 up on the command line. 

Upon deeper digging, I found the following settings in /etc/sysctl.conf on the machine,

          #Disabling SLAAC/Link Local addresses

          net.ipv6.conf.eth0.disable_ipv6=1

Note to VMware staff: This does not disable link local, it turns off IPv6 completely on the eth0 interface!

This appeared two times.  I then went and changed the settings to,

          net.ipv6.conf.eth0.disable_ipv6=0

And ran,

          sysctl -a

          sysctl -p

And the settings where changed.  After a few minutes, the web based GUI started to work again.

Next, I rebooted the machine, and it worked just fine.  Multiple reboots, it works just fine.

I found that after I changed this setting, I could change IPv6 settings in the Photon Management application.

You will notice that you will get 404 errors or cannot connect errors when you try the web GUI very early in the startup when things are working as they should.  That's a good sign, since, the 503 error will come up almost the same second the Photon Login console shows up.  After a few minutes, you will see your VM's get settings taken from the vSphere appliance, as it should, which you will then see the classic Web Initializing message in your browser.  Again, a good sign! If you login to the Photon Management application, you need to see "Good", not "Unknown".  Once "Good" displays for all the statuses, you should start seeing the 404 error, and then eventually the web gui will work. If you continue to see 503 errors and good, after waiting 10 or 15 minutes, then something else is wrong...

It does seem indeed that this problem is related to network settings.  Either bad hostname in /etc/hosts or some kind of issue with how IP's are being stored.  I hope someone from VMware sees this and decides to dig in, as it is very frustrating to fix.  You need good Linux development (not just admin skills) to debug this kind of issue.  That's not something most VMware admins will have.

HTH

I had the same issue (503), nothing was wrong with DNS or DB related.  The Problem popped up right after I updated my first host to 6.5.  Changing these IPv6 settings corrected the issue immediately.

Had the same problem, changing the net.ipv6.conf.eth0.disable_ipv6 from 1 to 0 also fixed the problem for me!

Thanks, this problem was driving me nuts.

We have VCSA 6.5 (Embedded PSC) and 503 error appear after each reboot or it got busy with multiple API calls:

503 Service Unavailable (Failed to connect to endpoint: [N7Vmacore4Http20NamedPipeServiceSpecE:0x00007f111401ed30] _serverNamespace = / action = Allow _pipeName =/var/run/vmware/vpxd-webserver-pipe)

Hi guys! I have the same problem since a few month and I can't find the solution.

My vcenter server is in:

OS. Windows Server 2012 R2 Standard

DB: Microsoft SQL Server Enterprise 11.0.2100.60

VCenter Version: Versión 6.5.0 Compilación 5318154 (18 APRIL 2017)

Also I had the same problem in oldest versions (4602587 and 4944578).

As a temporaly solution I delete de profile folders from C:\ProgramData\VMware\vCenterServer\data\vSphere Web Client\SerenityDB\serenity, after delete the profile folders works.

But I need to fix it definitly because crash frecuently.

I checked the GPO and not all user are add, and I cound't add it. hi users I couldn't add because doesn´t exist in my server.

  content-library

  perfcharts

  vmware-vpostgres

  vsan-health

The IPv6 is disabled in my Vcenter Server,

The DNS is correct in my DC ( register A and PTR)

The host file it's empty.

Someone can help me?

Regards.

Patricio.

Hello,

My setup is:

vCenter Server: Server 2016 x64 with Desktop Experience

vCenter DB Server: Server 2016 x64 with Desktop Experience and SQL Server Standard x64 w/SP1

I get the 503 Service Unavailable when I select vSphere Client (HTML5) - Partial functionality

I was able to get the HTML5 version working with a workaround:

1. Edit the following file: "C:\ProgramData\VMware\vCenterServer\cfg\vmware-vmon\svcCfgfiles\vsphere-ui.json"
2. Go just below “StartCommandArgs”: [
3. Insert a new line and add the following line: "-Dos.name=Windows Server 2012",
   1. Include the ,
   2. Be mindful of case sensitive
4. Stop vsphere-ui: C:\Program Files\VMware\vCenter Server\vmon\vmon-cli.exe --stop vsphere-ui
5. Start vsphere-ui: C:\Program Files\VMware\vCenter Server\vmon\vmon-cli.exe --start vsphere-ui

I'm not sure why Server 2016 isn't supported in the vsphere-ui but its supported in the Supported host operating systems for VMware vCenter Server installation

https://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2091273

-Ted

getting the 503 service unavailable error message on a vCenter 6.0 U2 deployment. Refer to the attachment. Logged a call to VMware and after many hours on a webex and logs been sent they suggested to rebuild but the issue could occur again and I don't want to rebuild it every time this happens.

any ideas?

Personally I think this will be a never ending story as long as this dreaded web-client is written in its current form, namely as a collection of hundreds of python scripts.

If you look at the amount of configuration information, which is there to get things going, where only 1/1000 is the data specific to your installation and the vast majority is made up of things, which are just needed, that the whole system does not crash or hang, you can see easily, that this is not a robust setup. It needs just one piece of configuration data getting wrong to havoc the whole system. As there is so much configuration data to look after, it is almost impossible to find the wrong value on your own - and as you see - even Vmware technicians themselves are unable to locate the wrong data in a reasonable timeframe.

I just had a similar problem after an unclean shutdown - and in the end resorted to restore the vCenter from a tape backup. And before restoring I made a copy of the (nonfunctional) configuration - and it turned out to be quite a big lot of information.

i find it hard to believe that the only option is to rebuild the appliance and there is no fix for this issue (for mine anyway). Not the best customer service from VMware at all.

Issue:- 503 Service Unavailable (Failed to connect to endpoint: [N7Vmacore4Http20NamedPipeServiceSpecE:0x0000559b1531ef80] _serverNamespace = / action = Allow _pipeName =/var/run/vmware/vpxd-webserver-pipe)

Troubleshooting steps:-

I followed below KBs and checked below logs and removed duplicate keys.

VMware Knowledge Base

https://anthonyspiteri.net/quick-fix-vcsa-503-service-unavailable-error/

# cat /var/log/vmware/vsphere-client/logs/vsphere_client_virgo.log |grep error

#cd /var/log/vmware/vpxd/

/var/log/vmware/vpxd/ # cat vpxd.log

/var/log/vmware/vpxd/ # grep -r already * | grep device_key

Note down the duplicate key and delete those duplicate keys.

# /opt/vmware/vpostgres/current/bin/psql -d VCDB -U postgres

# Delete from vc.vpx_vm_virtual_device where id='**' and device_key='***' ;

Try it and above issue would get resolve.

HI ,

I faced same issue after disabling the smb1 on my network  . i have to configure the smbv2 and restart the vcenter it worked .

503 Service Unavailable (Failed to connect to endpoint: [N7Vmacore4Http20NamedPipeServiceSpecE:0x0000558d2d78be90] _serverNamespace = / action = Allow _pipeName =/var/run/vmware/vpxd-webserver-pipe)

After scratching my head for a couple of hours and Googling for possible solutions. It turned out to simply be that my DNS server was down.

I am running into the same issue but the vpxd log is showing a different error.

cat /var/log/vmware/vpxd/vpxd.log | grep -i error

2018-05-24T18:01:56.545Z error vpxd[7FE19ACE7800] [Originator@6876 sub=MoOptionMgr] Unable to read from '/etc/motd':N7Vmacore23FileIONotFoundExceptionE(FileIO error: Could not find file  : /etc/motd)

2018-05-24T18:01:56.549Z error vpxd[7FE19ACE7800] [Originator@6876 sub=MoOptionMgr] [OptionMgr] Skipping bad entry config.vpxd.enableDebugBrowse from DB. Resetting to default.Exception: vmodl.fault.InvalidArgument

2018-05-24T18:01:59.021Z error vpxd[7FE19ACE7800] [Originator@6876 sub=DbBulkLoader] [VpxdDbBulkLoader::Load] Failed to load tableDef 41 from database: Column DVS_ID does not exist in table VPX_DVS_NRP

Before this happened, i had VC VM running on local datastore and the datastore ran out of space. So swap couldnt expand. I reserved all the memory and was able to start the VC VM. But now i am getting the same 503 error

I don't want to rebuild this VC since i have other applications connected to it.

help...!!!

MK

503 Service Unavailable (Failed to connect to endpoint: [class Vmacore::Http::NamedPipeServiceSpec:0x000000061c9e0f80] _serverNamespace = / action = Allow _pipeName =\\.\pipe\vmware-vpxd-webserver-pipe)

Hello All,

If no one has found an answer to this yet, I have a solution that worked for me and was pretty easy actually.

Log in to VCSA using SSH and modify /etc/hosts. Make you enable SSH first if you don't know how to do that please research. I have the answer but I'm sure most of you can figure that out.

Type ---- > shell.set --enabled TRUE

Type ----- > shell

Type ----- >vi /etc/hosts or you can "cat" the file first to look at it but you will need to edit more than likely.

I ADDED A LINE HERE THAT SAID THE FOLLOWING WITHIN THE "HOST":

I inserted this right below the 127.0.0.1 line .......

192.168.x.x <INSERT FQDN>.LOCAL

SAVED FILE AND REBOOTED VCSA

Once I rebooted VCSA and the servers started up completely, took a few minutes, I was able to access VCSA just fine using the FQDN name URL.

My initial error message was the following:

503 Service Unavailable (Failed to connect to endpoint: [N7Vmacore4Http16LocalServiceSpecE:0x00007fe854035a70] _serverNamespace = /vsphere-client action = Allow _port = 9090)

http://www.settlersoman.com/wp-content/uploads/2016/07/How-to-change-vCenter-Appliance-VCSA-6.x-IP-Address-7.png

Hope this helps someone. 

I have the same case.And you are right.Thanks.

recently upgraded from v6.0.0.30400 build 74641010 to v6.0.0.30800 Build Number 9448190.

Getting same error as above, but ours is space related.

login as: root

VMware vCenter Server Appliance 6.0.0.30800

Type: vCenter Server with an embedded Platform Services Controller

root@192.168.2.53's password:

vcenterdr:~ # df -h

Filesystem                            Size  Used Avail Use% Mounted on

/dev/sda3                              11G   11G     0 100% /

udev                                  4.0G  164K  4.0G   1% /dev

tmpfs                                 4.0G   32K  4.0G   1% /dev/shm

/dev/sda1                             128M   39M   83M  32% /boot

/dev/mapper/core_vg-core               26G  1.4G   23G   6% /storage/core

/dev/mapper/log_vg-log                 21G   13G  7.5G  63% /storage/log

/dev/mapper/db_vg-db                   12G  1.2G   11G  10% /storage/db

/dev/mapper/dblog_vg-dblog            5.9G  363M  5.3G   7% /storage/dblog

/dev/mapper/seat_vg-seat               12G  2.3G  9.0G  21% /storage/seat

/dev/mapper/netdump_vg-netdump        2.0G   18M  1.9G   1% /storage/netdump

/dev/mapper/autodeploy_vg-autodeploy   13G  154M   12G   2% /storage/autodeploy

/dev/mapper/invsvc_vg-invsvc          5.9G  222M  5.4G   4% /storage/invsvc

vcenterdr:~ #

Cant seem to increase the disks....

Hi all, I have the same problem, vcenter server 6.7 on win server 2012 on esxi 6.5 platform.

I did:

C:\ProgramData\VMware\vCenterServer\bin

service-control --stop vsphere-client

service-control --start vsphere-client

then

service-control --start vsphere-ui

All works perfectly.

Thansk

Rico

HI,

This is Link VMware Knowledge Base  Solve My Issue Completely.

BR,,

While my error is a little different from the OP, mine was this:

"503 Service Unavailable (Failed to connect to endpoint: [N7Vmacore4Http16LocalServiceSpecE:0x000055f28255ebb0] _serverNamespace = /ui action = Allow _port = 5090)"

I checked the audit.log and it was not hugh and took up all the space.   Then I went on to check the health of the VCSA by going  to

https://vcsa:5480

Everything was fine.  So I attempted to update and then I got an error saying my root password expired.  Change it before retrying (something like).

Once I changed the root password of the vcsa, my web-gui came back immediately.

I wish this is some kind of notification that my root password expired.  The kicker is that I was able to login as root using expired password.  Once login, no message to prompt me to change password. 

Hope this helps someone.