We have VCSA 6.5 (Embedded PSC) and 503 error appear after each reboot or it got busy with multiple API calls:
503 Service Unavailable (Failed to connect to endpoint: [N7Vmacore4Http20NamedPipeServiceSpecE:0x00007f111401ed30] _serverNamespace = / action = Allow _pipeName =/var/run/vmware/vpxd-webserver-pipe)
getting the 503 service unavailable error message on a vCenter 6.0 U2 deployment. Refer to the attachment. Logged a call to VMware and after many hours on a webex and logs been sent they suggested to rebuild but the issue could occur again and I don't want to rebuild it every time this happens.
Personally I think this will be a never ending story as long as this dreaded web-client is written in its current form, namely as a collection of hundreds of python scripts.
If you look at the amount of configuration information, which is there to get things going, where only 1/1000 is the data specific to your installation and the vast majority is made up of things, which are just needed, that the whole system does not crash or hang, you can see easily, that this is not a robust setup. It needs just one piece of configuration data getting wrong to havoc the whole system. As there is so much configuration data to look after, it is almost impossible to find the wrong value on your own - and as you see - even Vmware technicians themselves are unable to locate the wrong data in a reasonable timeframe.
I just had a similar problem after an unclean shutdown - and in the end resorted to restore the vCenter from a tape backup. And before restoring I made a copy of the (nonfunctional) configuration - and it turned out to be quite a big lot of information.
i find it hard to believe that the only option is to rebuild the appliance and there is no fix for this issue (for mine anyway). Not the best customer service from VMware at all.
Issue:- 503 Service Unavailable (Failed to connect to endpoint: [N7Vmacore4Http20NamedPipeServiceSpecE:0x0000559b1531ef80] _serverNamespace = / action = Allow _pipeName =/var/run/vmware/vpxd-webserver-pipe)
I followed below KBs and checked below logs and removed duplicate keys.
# cat /var/log/vmware/vsphere-client/logs/vsphere_client_virgo.log |grep error
/var/log/vmware/vpxd/ # cat vpxd.log
/var/log/vmware/vpxd/ # grep -r already * | grep device_key
Note down the duplicate key and delete those duplicate keys.
# /opt/vmware/vpostgres/current/bin/psql -d VCDB -U postgres
# Delete from vc.vpx_vm_virtual_device where id='**' and device_key='***' ;
Try it and above issue would get resolve.
This worked for me, make sure to start the perf charts service after as well.
Weird that it takes some time after restarting the services but it appears to... just have patience.
I faced same issue after disabling the smb1 on my network . i have to configure the smbv2 and restart the vcenter it worked .RAJESH RADHAKRISHNAN
VCA -DCV/WM/Cloud,VCP 5 - DCV/DT/CLOUD, ,VCP6-DCV, EMCISA,EMCSA,MCTS,MCPS,BCFA
Mark my post as "helpful" or "correct" if I've helped resolve or answered your query!
1 person found this helpful
503 Service Unavailable (Failed to connect to endpoint: [N7Vmacore4Http20NamedPipeServiceSpecE:0x0000558d2d78be90] _serverNamespace = / action = Allow _pipeName =/var/run/vmware/vpxd-webserver-pipe)
After scratching my head for a couple of hours and Googling for possible solutions. It turned out to simply be that my DNS server was down.
I am running into the same issue but the vpxd log is showing a different error.
cat /var/log/vmware/vpxd/vpxd.log | grep -i error
2018-05-24T18:01:56.545Z error vpxd[7FE19ACE7800] [Originator@6876 sub=MoOptionMgr] Unable to read from '/etc/motd':N7Vmacore23FileIONotFoundExceptionE(FileIO error: Could not find file : /etc/motd)
2018-05-24T18:01:56.549Z error vpxd[7FE19ACE7800] [Originator@6876 sub=MoOptionMgr] [OptionMgr] Skipping bad entry config.vpxd.enableDebugBrowse from DB. Resetting to default.Exception: vmodl.fault.InvalidArgument
2018-05-24T18:01:59.021Z error vpxd[7FE19ACE7800] [Originator@6876 sub=DbBulkLoader] [VpxdDbBulkLoader::Load] Failed to load tableDef 41 from database: Column DVS_ID does not exist in table VPX_DVS_NRP
Before this happened, i had VC VM running on local datastore and the datastore ran out of space. So swap couldnt expand. I reserved all the memory and was able to start the VC VM. But now i am getting the same 503 error
I don't want to rebuild this VC since i have other applications connected to it.
503 Service Unavailable (Failed to connect to endpoint: [class Vmacore::Http::NamedPipeServiceSpec:0x000000061c9e0f80] _serverNamespace = / action = Allow _pipeName =\\.\pipe\vmware-vpxd-webserver-pipe)
This is what was in the Windows Event Log:
Log Name: Application
Source: VMware VirtualCenter Server
Date: 08/07/2018 10:06:19
Event ID: 1000
Task Category: None
Failed to report event. Reason: Data error (cyclic redundancy check)
Apparently there is a lot of services depending on the connection to this service, producing similar error but with completely different causes. This time we ran out of space on the SQL database logs filesystem. Clearing down that filesystem allowed us to make the vsphere operational again. We didn't want to rebuild the whole ESX cluster.
Last time it was the (already mentioned before in this thread) lack of permissions for local user to log on as service (our group policy overwrote the permissions that were done during the installation of vSphere. Not sure if the error number was exactly the same, but the message was strikingly similar.
If no one has found an answer to this yet, I have a solution that worked for me and was pretty easy actually.
Log in to VCSA using SSH and modify /etc/hosts. Make you enable SSH first if you don't know how to do that please research. I have the answer but I'm sure most of you can figure that out.
Type ---- > shell.set --enabled TRUE
Type ----- > shell
Type ----- >vi /etc/hosts or you can "cat" the file first to look at it but you will need to edit more than likely.
I ADDED A LINE HERE THAT SAID THE FOLLOWING WITHIN THE "HOST":
I inserted this right below the 127.0.0.1 line .......
192.168.x.x <INSERT FQDN>.LOCAL
SAVED FILE AND REBOOTED VCSA
Once I rebooted VCSA and the servers started up completely, took a few minutes, I was able to access VCSA just fine using the FQDN name URL.
My initial error message was the following:
503 Service Unavailable (Failed to connect to endpoint: [N7Vmacore4Http16LocalServiceSpecE:0x00007fe854035a70] _serverNamespace = /vsphere-client action = Allow _port = 9090)
Hope this helps someone.
I have the same case.And you are right.Thanks.
recently upgraded from v220.127.116.11400 build 74641010 to v18.104.22.168800 Build Number 9448190.
Getting same error as above, but ours is space related.
login as: root
VMware vCenter Server Appliance 22.214.171.124800
Type: vCenter Server with an embedded Platform Services Controller
vcenterdr:~ # df -h
Filesystem Size Used Avail Use% Mounted on
/dev/sda3 11G 11G 0 100% /
udev 4.0G 164K 4.0G 1% /dev
tmpfs 4.0G 32K 4.0G 1% /dev/shm
/dev/sda1 128M 39M 83M 32% /boot
/dev/mapper/core_vg-core 26G 1.4G 23G 6% /storage/core
/dev/mapper/log_vg-log 21G 13G 7.5G 63% /storage/log
/dev/mapper/db_vg-db 12G 1.2G 11G 10% /storage/db
/dev/mapper/dblog_vg-dblog 5.9G 363M 5.3G 7% /storage/dblog
/dev/mapper/seat_vg-seat 12G 2.3G 9.0G 21% /storage/seat
/dev/mapper/netdump_vg-netdump 2.0G 18M 1.9G 1% /storage/netdump
/dev/mapper/autodeploy_vg-autodeploy 13G 154M 12G 2% /storage/autodeploy
/dev/mapper/invsvc_vg-invsvc 5.9G 222M 5.4G 4% /storage/invsvc
Cant seem to increase the disks....