503 Service Unavailable (Failed to connect to endpoint:

[ocsitspecialist]

Error Message:

503 Service Unavailable (Failed to connect to endpoint: [class Vmacore::Http::LocalServiceSpec:0x000001d47454d440] _serverNamespace = /ui action = Allow _port = 5090)

Hello guys good evening, I was encountered this error on my host machine when accessing the vcenter server with vSphere Client (HTML5) - partial functionality, any solution or recommendation to solve my current issues on my vCenter Server 6.5 thank you guys and GOD BLESS US ALL...

[tdesai]

can you share details on this ticket ??

[RajeevVCP4]

Hello

I fixed same issue , but my customer infrastructure was like this. 

Basically this issue occurred because of your DB full or DNS have wrong entry.

2 PSC

1 VC ( with vPostgreSQL db).

Giving here step by step solution.

run hostname -f on vc and psc virtual machine  if you got this error ,

Unknown host

1. Log in to the console of vCenter Appliance as the root user.
2. To change directory to /etc , run this command:
   
   cd /etc
   
   
3. Open the hosts file using a text editor.
4. Add the hostname in vCenter_IP vCenter Server Appliance FQDN vcenter short name format.
   
   For example, 10.10.10.10 vcenter.vmware.com vcenter
5. Save and close the file.
6. Restart the vmware-vpxd service by running this command:

service vmware-vpxd restart

1. Issue was in DNS setting on PSC , login by root on GUI change DNS according to infrastructure , you can check by it nslookup from you virtual center server.

2. IP address of your PSC should be resolved on both ways ( forward lookup and reverse lookup)

3. Then try to restart vpxd service

4. service vmware-vpxd restart

May be it would failed as service initialized failed if yes

Shut down your PSC and restart

Shut down your VC and restart

Check vpxd status service vmware-vpxd status ( If not running)

Go to PSC/VM vm by putty and restart these service.

/etc/init.d/vmware-stsd restart

/etc/init.d/vmware-sts-idmd restart

service vmware-cm restart

then go to vc vm by putty

and restart

service vmware-vpxd restart

Now you should be connect by client.

If you have still issue send me log file.

/var/log/vmware/vpx/vpxd.log

[TechNut]

Hi,

After looking at all the issues in this thread, I still had the 503 problem.

I'm running the Photon-based VCSA Appliance 6.5.  I did the Photon update, and upgraded to the latest version, 6.5.0.5300.

After a reboot, that did not solve my issue.  I again logged into the management appliance, and looked at the settings.  Following the helpful video on the DNS/DHCP reservation, which did not apply to me, I began to wonder if network settings where the issue. 

In my settings, I found that IPv4 was enabled, but IPv6 was turned off.  On a hunch, I enabled IPv6, and a few minutes later the /ui and vSphere 6.5 web client started working again. Great! 

On a reboot, I found Photon did not keep any of the IPv6 settings, and again, the 503 error was thrown, no matter how long I waited.  I logged into the appliance via the console, and tried to set IPv6 settings manually.  This did not work.  There's some modification to the configuration to prevent setting IPv6 up on the command line. 

Upon deeper digging, I found the following settings in /etc/sysctl.conf on the machine,

          #Disabling SLAAC/Link Local addresses

          net.ipv6.conf.eth0.disable_ipv6=1

Note to VMware staff: This does not disable link local, it turns off IPv6 completely on the eth0 interface!

This appeared two times.  I then went and changed the settings to,

          net.ipv6.conf.eth0.disable_ipv6=0

And ran,

          sysctl -a

          sysctl -p

And the settings where changed.  After a few minutes, the web based GUI started to work again.

Next, I rebooted the machine, and it worked just fine.  Multiple reboots, it works just fine.

I found that after I changed this setting, I could change IPv6 settings in the Photon Management application.

You will notice that you will get 404 errors or cannot connect errors when you try the web GUI very early in the startup when things are working as they should.  That's a good sign, since, the 503 error will come up almost the same second the Photon Login console shows up.  After a few minutes, you will see your VM's get settings taken from the vSphere appliance, as it should, which you will then see the classic Web Initializing message in your browser.  Again, a good sign! If you login to the Photon Management application, you need to see "Good", not "Unknown".  Once "Good" displays for all the statuses, you should start seeing the 404 error, and then eventually the web gui will work. If you continue to see 503 errors and good, after waiting 10 or 15 minutes, then something else is wrong...

It does seem indeed that this problem is related to network settings.  Either bad hostname in /etc/hosts or some kind of issue with how IP's are being stored.  I hope someone from VMware sees this and decides to dig in, as it is very frustrating to fix.  You need good Linux development (not just admin skills) to debug this kind of issue.  That's not something most VMware admins will have.

HTH

[macralf]

I had the same issue (503), nothing was wrong with DNS or DB related.  The Problem popped up right after I updated my first host to 6.5.  Changing these IPv6 settings corrected the issue immediately.

[Richie_NL_]

Had the same problem, changing the net.ipv6.conf.eth0.disable_ipv6 from 1 to 0 also fixed the problem for me!

Thanks, this problem was driving me nuts.

[AaronRose]

I had this same issue on a 6.0 u2 windows machine last week after a restart.  If you so choose the installer creates many virtual accounts NT Service\VMware-thingx..etc.  That it uses to log on different services.   On the windows machine I like to use the service-control --start --all command in the \bin of the vcenter folder.  It starts all services or with --stop stops all services in the correct order.  I had mine fail on any vmware service that wasn't using system as checked in services.msc.

I attempted to use "local service" as I have two vcenters and this was working on anohter, but had the same result.  When I looked at the user rights "logon as a service" no longer had local service in the use rights.  TIP: when you make a GPO to give domain accounts specific user rights, be sure to add the defaults back in.  The gpo does not add the accounts to those user rights it replace whatever values are already there with the ones you set. 

After setting each of those services to the domain account....all is working properly.  Leave the services using SYSTEM account to logon alone.

[MihailsA]

What kind of sorcery is that? Good Lord! You deserve every cookie in the world for this! Not even VMware support could figure it out for me. If only I found this a couple of weeks earlier - would have saved me days of struggle with upgrade to 6.5! You rock! :smileycool:

[patomanna]

Hi guys! I have the same problem since a few month and I can't find the solution.

My vcenter server is in:

OS. Windows Server 2012 R2 Standard

DB: Microsoft SQL Server Enterprise 11.0.2100.60

VCenter Version: Versión 6.5.0 Compilación 5318154 (18 APRIL 2017)

Also I had the same problem in oldest versions (4602587 and 4944578).

As a temporaly solution I delete de profile folders from C:\ProgramData\VMware\vCenterServer\data\vSphere Web Client\SerenityDB\serenity, after delete the profile folders works.

But I need to fix it definitly because crash frecuently.

I checked the GPO and not all user are add, and I cound't add it. hi users I couldn't add because doesn´t exist in my server.

  content-library

  perfcharts

  vmware-vpostgres

  vsan-health

The IPv6 is disabled in my Vcenter Server,

The DNS is correct in my DC ( register A and PTR)

The host file it's empty.

Someone can help me?

Regards.

Patricio.

[aladin98]

It really help bro, thank you

[GreenerRabbit]

Hello,

My setup is:

vCenter Server: Server 2016 x64 with Desktop Experience

vCenter DB Server: Server 2016 x64 with Desktop Experience and SQL Server Standard x64 w/SP1

I get the 503 Service Unavailable when I select vSphere Client (HTML5) - Partial functionality

I was able to get the HTML5 version working with a workaround:

1. Edit the following file: "C:\ProgramData\VMware\vCenterServer\cfg\vmware-vmon\svcCfgfiles\vsphere-ui.json"
2. Go just below “StartCommandArgs”: [
3. Insert a new line and add the following line: "-Dos.name=Windows Server 2012",
   1. Include the ,
   2. Be mindful of case sensitive
4. Stop vsphere-ui: C:\Program Files\VMware\vCenter Server\vmon\vmon-cli.exe --stop vsphere-ui
5. Start vsphere-ui: C:\Program Files\VMware\vCenter Server\vmon\vmon-cli.exe --start vsphere-ui

I'm not sure why Server 2016 isn't supported in the vsphere-ui but its supported in the Supported host operating systems for VMware vCenter Server installation

https://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=20912...

-Ted

[Sateesh_vCloud]

We have VCSA 6.5 (Embedded PSC) and 503 error appear after each reboot or it got busy with multiple API calls:

503 Service Unavailable (Failed to connect to endpoint: [N7Vmacore4Http20NamedPipeServiceSpecE:0x00007f111401ed30] _serverNamespace = / action = Allow _pipeName =/var/run/vmware/vpxd-webserver-pipe)

[lucasrosa]

Waiting 8 to 10 minutes seems to be working for me. First I was seeing the 503 Service Unavailable error, then the /vsphere-client/ endpoint got offline and finally I saw this:

After couple more minutes i was able to login in

Cheers,

Lucas

[vijayrana968]

Worked like a charm.

one thing to check, where is webclient service in Windows based vCenter 6.5.

[abugeja]

getting the 503 service unavailable error message on a vCenter 6.0 U2 deployment. Refer to the attachment. Logged a call to VMware and after many hours on a webex and logs been sent they suggested to rebuild but the issue could occur again and I don't want to rebuild it every time this happens.

any ideas?

[pwolf]

Personally I think this will be a never ending story as long as this dreaded web-client is written in its current form, namely as a collection of hundreds of python scripts.

If you look at the amount of configuration information, which is there to get things going, where only 1/1000 is the data specific to your installation and the vast majority is made up of things, which are just needed, that the whole system does not crash or hang, you can see easily, that this is not a robust setup. It needs just one piece of configuration data getting wrong to havoc the whole system. As there is so much configuration data to look after, it is almost impossible to find the wrong value on your own - and as you see - even Vmware technicians themselves are unable to locate the wrong data in a reasonable timeframe.

I just had a similar problem after an unclean shutdown - and in the end resorted to restore the vCenter from a tape backup. And before restoring I made a copy of the (nonfunctional) configuration - and it turned out to be quite a big lot of information.

[abugeja]

i find it hard to believe that the only option is to rebuild the appliance and there is no fix for this issue (for mine anyway). Not the best customer service from VMware at all.

[sarikrizvi]

Issue:- 503 Service Unavailable (Failed to connect to endpoint: [N7Vmacore4Http20NamedPipeServiceSpecE:0x0000559b1531ef80] _serverNamespace = / action = Allow _pipeName =/var/run/vmware/vpxd-webserver-pipe)

Troubleshooting steps:-

I followed below KBs and checked below logs and removed duplicate keys.

VMware Knowledge Base

https://anthonyspiteri.net/quick-fix-vcsa-503-service-unavailable-error/

# cat /var/log/vmware/vsphere-client/logs/vsphere_client_virgo.log |grep error

#cd /var/log/vmware/vpxd/

/var/log/vmware/vpxd/ # cat vpxd.log

/var/log/vmware/vpxd/ # grep -r already * | grep device_key

Note down the duplicate key and delete those duplicate keys.

# /opt/vmware/vpostgres/current/bin/psql -d VCDB -U postgres

# Delete from vc.vpx_vm_virtual_device where id='**' and device_key='***' ;

Try it and above issue would get resolve.

[devino]

This worked for me, make sure to start the perf charts service after as well.

Weird that it takes some time after restarting the services but it appears to... just have patience.

[RAJ_RAJ]

HI ,

I faced same issue after disabling the smb1 on my network  . i have to configure the smbv2 and restart the vcenter it worked .

[blocktrev]

503 Service Unavailable (Failed to connect to endpoint: [N7Vmacore4Http20NamedPipeServiceSpecE:0x0000558d2d78be90] _serverNamespace = / action = Allow _pipeName =/var/run/vmware/vpxd-webserver-pipe)

After scratching my head for a couple of hours and Googling for possible solutions. It turned out to simply be that my DNS server was down.