1 2 3 4 Previous Next 47 Replies Latest reply on May 1, 2019 4:58 AM by parvezaliusmani Go to original post
      • 30. Re: 503 Service Unavailable (Failed to connect to endpoint:
        Sateesh_vCloud Hot Shot

        We have VCSA 6.5 (Embedded PSC) and 503 error appear after each reboot or it got busy with multiple API calls:

         

        503 Service Unavailable (Failed to connect to endpoint: [N7Vmacore4Http20NamedPipeServiceSpecE:0x00007f111401ed30] _serverNamespace = / action = Allow _pipeName =/var/run/vmware/vpxd-webserver-pipe)

        • 31. Re: 503 Service Unavailable (Failed to connect to endpoint:
          lucasrosa Lurker

          Waiting 8 to 10 minutes seems to be working for me. First I was seeing the 503 Service Unavailable error, then the /vsphere-client/ endpoint got offline and finally I saw this:

          After couple more minutes i was able to login in

           

          Cheers,

          Lucas

          • 32. Re: 503 Service Unavailable (Failed to connect to endpoint:
            vijayrana968 Master
            vExpert

            Worked like a charm.

             

            one thing to check, where is webclient service in Windows based vCenter 6.5.

            • 33. Re: 503 Service Unavailable (Failed to connect to endpoint:
              abugeja Enthusiast

              getting the 503 service unavailable error message on a vCenter 6.0 U2 deployment. Refer to the attachment. Logged a call to VMware and after many hours on a webex and logs been sent they suggested to rebuild but the issue could occur again and I don't want to rebuild it every time this happens.

               

              any ideas?

              503.PNG

              • 34. Re: 503 Service Unavailable (Failed to connect to endpoint:
                pwolf Enthusiast

                Personally I think this will be a never ending story as long as this dreaded web-client is written in its current form, namely as a collection of hundreds of python scripts.

                 

                If you look at the amount of configuration information, which is there to get things going, where only 1/1000 is the data specific to your installation and the vast majority is made up of things, which are just needed, that the whole system does not crash or hang, you can see easily, that this is not a robust setup. It needs just one piece of configuration data getting wrong to havoc the whole system. As there is so much configuration data to look after, it is almost impossible to find the wrong value on your own - and as you see - even Vmware technicians themselves are unable to locate the wrong data in a reasonable timeframe.

                 

                I just had a similar problem after an unclean shutdown - and in the end resorted to restore the vCenter from a tape backup. And before restoring I made a copy of the (nonfunctional) configuration - and it turned out to be quite a big lot of information.

                • 35. Re: 503 Service Unavailable (Failed to connect to endpoint:
                  abugeja Enthusiast

                  i find it hard to believe that the only option is to rebuild the appliance and there is no fix for this issue (for mine anyway). Not the best customer service from VMware at all.

                  • 36. Re: 503 Service Unavailable (Failed to connect to endpoint:
                    sarikrizvi Enthusiast

                    Issue:- 503 Service Unavailable (Failed to connect to endpoint: [N7Vmacore4Http20NamedPipeServiceSpecE:0x0000559b1531ef80] _serverNamespace = / action = Allow _pipeName =/var/run/vmware/vpxd-webserver-pipe)

                     

                    Troubleshooting steps:-

                     

                    I followed below KBs and checked below logs and removed duplicate keys.


                    VMware Knowledge Base

                    https://anthonyspiteri.net/quick-fix-vcsa-503-service-unavailable-error/

                     

                    # cat /var/log/vmware/vsphere-client/logs/vsphere_client_virgo.log |grep error

                    #cd /var/log/vmware/vpxd/


                    /var/log/vmware/vpxd/ # cat vpxd.log

                    /var/log/vmware/vpxd/ # grep -r already * | grep device_key

                     

                    Note down the duplicate key and delete those duplicate keys.

                     

                    # /opt/vmware/vpostgres/current/bin/psql -d VCDB -U postgres

                     

                    # Delete from vc.vpx_vm_virtual_device where id='**' and device_key='***' ;

                     

                    Try it and above issue would get resolve.

                    • 37. Re: 503 Service Unavailable (Failed to connect to endpoint:
                      devino Lurker

                      This worked for me, make sure to start the perf charts service after as well.

                       

                      Weird that it takes some time after restarting the services but it appears to... just have patience.

                      • 38. Re: 503 Service Unavailable (Failed to connect to endpoint:
                        RAJ_RAJ Expert
                        vExpert

                        HI ,

                         

                        I faced same issue after disabling the smb1 on my network  . i have to configure the smbv2 and restart the vcenter it worked .

                        RAJESH RADHAKRISHNAN
                        VCA -DCV/WM/Cloud,VCP 5 - DCV/DT/CLOUD, ,VCP6-DCV, EMCISA,EMCSA,MCTS,MCPS,BCFA
                        https://ae.linkedin.com/in/rajesh-radhakrishnan-76269335
                        Mark my post as "helpful" or "correct" if I've helped resolve or answered your query!
                        • 39. Re: 503 Service Unavailable (Failed to connect to endpoint:
                          blocktrev Novice

                          503 Service Unavailable (Failed to connect to endpoint: [N7Vmacore4Http20NamedPipeServiceSpecE:0x0000558d2d78be90] _serverNamespace = / action = Allow _pipeName =/var/run/vmware/vpxd-webserver-pipe)

                           

                          After scratching my head for a couple of hours and Googling for possible solutions. It turned out to simply be that my DNS server was down.

                          1 person found this helpful
                          • 40. Re: 503 Service Unavailable (Failed to connect to endpoint:
                            Mokshir Novice
                            vExpert

                            I am running into the same issue but the vpxd log is showing a different error.

                             

                            cat /var/log/vmware/vpxd/vpxd.log | grep -i error

                             

                            2018-05-24T18:01:56.545Z error vpxd[7FE19ACE7800] [Originator@6876 sub=MoOptionMgr] Unable to read from '/etc/motd':N7Vmacore23FileIONotFoundExceptionE(FileIO error: Could not find file  : /etc/motd)

                            2018-05-24T18:01:56.549Z error vpxd[7FE19ACE7800] [Originator@6876 sub=MoOptionMgr] [OptionMgr] Skipping bad entry config.vpxd.enableDebugBrowse from DB. Resetting to default.Exception: vmodl.fault.InvalidArgument

                            2018-05-24T18:01:59.021Z error vpxd[7FE19ACE7800] [Originator@6876 sub=DbBulkLoader] [VpxdDbBulkLoader::Load] Failed to load tableDef 41 from database: Column DVS_ID does not exist in table VPX_DVS_NRP

                             

                            Before this happened, i had VC VM running on local datastore and the datastore ran out of space. So swap couldnt expand. I reserved all the memory and was able to start the VC VM. But now i am getting the same 503 error

                             

                            I don't want to rebuild this VC since i have other applications connected to it.

                             

                            help...!!!

                             

                            MK

                            • 41. Re: 503 Service Unavailable (Failed to connect to endpoint:
                              pikmaster Novice

                              503 Service Unavailable (Failed to connect to endpoint: [class Vmacore::Http::NamedPipeServiceSpec:0x000000061c9e0f80] _serverNamespace = / action = Allow _pipeName =\\.\pipe\vmware-vpxd-webserver-pipe)

                               


                              This is what was in the Windows Event Log:

                               

                              Log Name:      Application

                              Source: VMware VirtualCenter Server

                              Date: 08/07/2018 10:06:19

                              Event ID:      1000

                              Task Category: None

                              Level: Warning

                              Keywords: Classic

                              User: N/A

                              Computer: extdubesxvc501.eu.dsext.susq.com

                              Description:

                              Failed to report event. Reason: Data error (cyclic redundancy check)

                               

                              Apparently there is a lot of services depending on the connection to this service, producing similar error but with completely different causes. This time we ran out of space on the SQL database logs filesystem. Clearing down that filesystem allowed us to make the vsphere operational again. We didn't want to rebuild the whole ESX cluster.

                               

                              Last time it was the (already mentioned before in this thread) lack of permissions for local user to log on as service (our group policy overwrote the permissions that were done during the installation of vSphere. Not sure if the error number was exactly the same, but the message was strikingly similar.

                               

                              • 42. Re: 503 Service Unavailable (Failed to connect to endpoint:
                                ew0506 Novice

                                Hello All,

                                 

                                If no one has found an answer to this yet, I have a solution that worked for me and was pretty easy actually.

                                 

                                Log in to VCSA using SSH and modify /etc/hosts. Make you enable SSH first if you don't know how to do that please research. I have the answer but I'm sure most of you can figure that out.

                                 

                                Type ---- > shell.set --enabled TRUE

                                 

                                Type ----- > shell

                                 

                                Type ----- >vi /etc/hosts or you can "cat" the file first to look at it but you will need to edit more than likely.

                                 

                                I ADDED A LINE HERE THAT SAID THE FOLLOWING WITHIN THE "HOST":

                                 

                                I inserted this right below the 127.0.0.1 line .......

                                 

                                192.168.x.x <INSERT FQDN>.LOCAL

                                 

                                SAVED FILE AND REBOOTED VCSA

                                 

                                Once I rebooted VCSA and the servers started up completely, took a few minutes, I was able to access VCSA just fine using the FQDN name URL.

                                 

                                My initial error message was the following:

                                 

                                 

                                503 Service Unavailable (Failed to connect to endpoint: [N7Vmacore4Http16LocalServiceSpecE:0x00007fe854035a70] _serverNamespace = /vsphere-client action = Allow _port = 9090)

                                 

                                http://www.settlersoman.com/wp-content/uploads/2016/07/How-to-change-vCenter-Appliance-VCSA-6.x-IP-Address-7.png

                                 

                                Hope this helps someone. 

                                • 43. Re: 503 Service Unavailable (Failed to connect to endpoint:
                                  wellschen90 Lurker

                                  I have the same case.And you are right.Thanks.

                                  • 44. Re: 503 Service Unavailable (Failed to connect to endpoint:
                                    chi201110141 Novice

                                    recently upgraded from v6.0.0.30400 build 74641010 to v6.0.0.30800 Build Number 9448190.

                                    Getting same error as above, but ours is space related.

                                    login as: root

                                     

                                    VMware vCenter Server Appliance 6.0.0.30800

                                     

                                    Type: vCenter Server with an embedded Platform Services Controller

                                    root@192.168.2.53's password:

                                     

                                    vcenterdr:~ # df -h

                                    Filesystem                            Size  Used Avail Use% Mounted on

                                    /dev/sda3                              11G   11G     0 100% /

                                    udev                                  4.0G  164K  4.0G   1% /dev

                                    tmpfs                                 4.0G   32K  4.0G   1% /dev/shm

                                    /dev/sda1                             128M   39M   83M  32% /boot

                                    /dev/mapper/core_vg-core               26G  1.4G   23G   6% /storage/core

                                    /dev/mapper/log_vg-log                 21G   13G  7.5G  63% /storage/log

                                    /dev/mapper/db_vg-db                   12G  1.2G   11G  10% /storage/db

                                    /dev/mapper/dblog_vg-dblog            5.9G  363M  5.3G   7% /storage/dblog

                                    /dev/mapper/seat_vg-seat               12G  2.3G  9.0G  21% /storage/seat

                                    /dev/mapper/netdump_vg-netdump        2.0G   18M  1.9G   1% /storage/netdump

                                    /dev/mapper/autodeploy_vg-autodeploy   13G  154M   12G   2% /storage/autodeploy

                                    /dev/mapper/invsvc_vg-invsvc          5.9G  222M  5.4G   4% /storage/invsvc

                                    vcenterdr:~ #

                                     

                                    Cant seem to increase the disks....