12 Replies Latest reply on Jun 29, 2017 8:25 AM by ChrisDixon777

    503 error on vCSA 6.5.0a running on ESXi 6.5.0a

    sjk1987 Lurker

      Hello all well I been having problems with my vCSA for some time now. and i cant found anything on google what will fix my problem. The problem is after rebooting my ESXi my vCSA web GUI will not loud when i go to the main site i get this Error:

       

      503 Service Unavailable (Failed to connect to endpoint: [N7Vmacore4Http20NamedPipeServiceSpecE:0x00007f0ec80aaf10] _serverNamespace = / action = Allow _pipeName =/var/run/vmware/vpxd-webserver-pipe)

       

      But if i go to "https://********/vsphere-client" i can login but i get a error on the top of the screen (see below) and none of the VM's or ESXi popup.

       

      Could not connect to one or more vCenter Server systems:

      https://********:443/sdk

       

      Only way i can fix this is to delete the vCSA and reinstall it. I also tried upgrading to the newest build 6.5.0a on the ESXi host and vCSA and still after a reboot vCSA comes up with at 503 error (i was running just 6.5). I can get in to SSH and the web management on port 5480 and all the health statuses are saying "Good". I can not get in to sFTP to copy out files like the logs files. I have not change the IP of the ESXi or vCSA. On my network i am using a UBNT ERLite‑3 router and I have the MAC and IP set in the DHCP table. From anyware on the network i can ping it by name and IP and if i do a NSLOOKUP it comes back correctly. I also shutdown the vCSA and try to ping in and i get no replay (just checking to make sure there is no duplicate IP's). Below are the service running and stopped on vCSA.

       

      Running:

      applmgmt lwsmd vmafdd vmcad vmdird vmdnsd vmonapi vmware-cis-license vmware-cm vmware-content-library vmware-eam vmware-perfcharts vmware-psc-client vmware-rhttpproxy vmware-sca vmware-statsmonitor vmware-sts-idmd vmware-stsd vmware-vapi-endpoint vmware-vmon vmware-vpostgres vmware-vpxd-svcs vmware-vsm vsphere-client vsphere-ui

      Stopped:

      pschealth vmcam vmware-imagebuilder vmware-mbcs vmware-netdumper vmware-rbd-watchdog vmware-sps vmware-updatemgr vmware-vcha vmware-vpxd vmware-vsan-health


      I have found that some people having the same problem and they were able to just restart the "vsphere-client" and then start it but that does not work for me. I also tried when i installed vCSA with DHCP instead of a static IP.. vCSA got the right IP but after I reboot the ESXi I still got the 503 error. "pbraren" from HERE had some good luck with this but it did not work for me .

       

      If anyone have any suggestions and need more info please let me know.

       

      Thanks,

       

      Steve

        • 1. Re: 503 error on vCSA 6.5.0a running on ESXi 6.5.0a
          misan Novice

          Exactly the same issue for me.  I upgraded my VCSA 6.5 to 6.5a, which went fine and I could access the web GUI without any issues.  Only after upgrading ESXi 6.5 to 6.5a did I receive the following message from the web GUI

          Could not connect to one or more vCenter Server systems:

          https://%2A%2A%2A%2A%2A%2A%2A%2A/sdkhttps://********:443/sdk

           

          Clearly rebuilding my VCSA will resolve this issue, but surely there is a better solution..

           

          I've synchronised both ESXi host and VCSA with correct NTP configuration.

          DNS lookups perform fine.

          • 2. Re: 503 error on vCSA 6.5.0a running on ESXi 6.5.0a
            KAI_IT Lurker

            Same Issue *BUMP*

            • 3. Re: 503 error on vCSA 6.5.0a running on ESXi 6.5.0a
              vwic Novice

              Same issue here!

              The first time it happened last week, I had to restore to a copy more than a week old (using Veeam)!

              Then it happened again yesterday. I don't want to have to restore it again, so have just raised a Sev 2 case with VMware - will see what they find; which means, in the meantime, I can't use vCenter.

              • 4. Re: 503 error on vCSA 6.5.0a running on ESXi 6.5.0a
                vwic Novice

                So, I opened a case with VMware and it's a known issue for vCenter 6.5.0a. The below fix directly solved the error I was seeing.

                 

                Information attached for reference only. Please be aware that this may NOT be applicable to your system and may bork it completely!

                 

                 

                Symptoms


                On the webclient, the web page shows the error: 503 Service Unavailable (Failed to connect to endpoint). . .

                PowerCLI cannot connect to vCenter.


                vpxd on vCenter Server 6.5 intermittently crashes with the following error stack in the vpxd log file:

                 

                2016-11-28T02:09:36.418Z error vpxd[7F27BD4A9700] [Originator@6876 sub=Default opID=HB-host-83@226512-71ec21d0] [VdbStatement] SQLError was thrown: "ODBC error: (23505) - ERROR: duplicate key value violates unique constraint "pk_vpx_vm_virtual_device";

                --> Error while executing the query" is returned when executing SQL statement "INSERT INTO VPX_VM_VIRTUAL_DEVICE (ID, DEVICE_KEY, CONN_AL_GU_CONTROL_FLAG, CONN_CONNECTED_FLAG, CONN_START_CONNECTED_FLAG, CONN_STATUS, DEVICE_INFO_SUMMARY, DEVICE_INFO_LABEL, DEVICE_CONTROLLER_KEY, DEVICE_UNIT_NUMBER, DEVICE_TYPE, DEVICE_TEXT) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)"

                 

                Note:

                • The VMware vCenter Server 6.5 vpxd logs are located in the %ALLUSERSPROFILE%\VMWare\vCenterServer\logs\vmware-vpx folder.
                • The VMware vCenter Server Appliance vpxd 6.5 logs are located in the /var/log/vmware/vmware-vpx folder.

                 

                Resolution

                Caution: Before making database modifications, ensure there is a working backup of the database or snapshot.

                 

                If the embedded postgres database is being used, locate the duplicate key in the postgresql logs:

                 

                postgresql.log

                UTC 583a9c81.5814 127750 VCDB vc ERROR:  duplicate key value violates unique constraint "pk_vpx_vm_virtual_device"

                UTC 583a9c81.5814 127750 VCDB vc DETAIL:  Key (id, device_key)=(361, 3002) already exists.

                 

                Note: The postgresql.log  can be found in var/log/vmware/vpostgres

                 

                To connect to the embedded postgres database:


                /opt/vmware/vpostgres/current/bin/psql -d VCDB -U postgres

                 

                To remove the duplicate key run the below command:

                (modify the id and device_key according to your vpxd log below!):

                 

                DELETE FROM vc.vpx_vm_virtual_device where id='361' and device_key='3002';

                 

                3 people found this helpful
                • 5. Re: 503 error on vCSA 6.5.0a running on ESXi 6.5.0a
                  gcallari Lurker

                  This procedure worked fine for me on two different installations. Thanks!

                  • 6. Re: 503 error on vCSA 6.5.0a running on ESXi 6.5.0a
                    iamKhris Lurker

                    Chiming in as this fixed our issue today as well!!  Thanks!!!

                    • 7. Re: 503 error on vCSA 6.5.0a running on ESXi 6.5.0a
                      cjastone Lurker

                      Thanks so much!  This procedure worked flawlessly for me after the machine experienced a hard shutdown!

                      • 8. Re: 503 error on vCSA 6.5.0a running on ESXi 6.5.0a
                        brundon Lurker

                        This solution worked great.  Thanks for posting!

                        • 9. Re: 503 error on vCSA 6.5.0a running on ESXi 6.5.0a
                          vojd Lurker

                          vwic 's

                           

                          2017-05-24 08:19:19.166 UTC 592541cf.1346 2991399 VCDB vc ERROR:  duplicate key value violates unique constraint "pk_vpx_vm_virtual_device"

                          2017-05-24 08:19:19.166 UTC 592541cf.1346 2991399 VCDB vc DETAIL:  Key (id, device_key)=(40, 4000) already exists.    <== here the id and device_key will be different for others

                           

                          So, I've done snapshot of an entire vCenter appliance (just in case) and deleted all of duplicated key ( in my case there were 2 of them) and both of them were Alladin USB HASP key dongles, used by some application for Licening purposes. After I've rebooted vCSA and after couple minutes, my vCenter has alive.

                           

                          Thanks one more time to vwic and I want to tell to VMware, that they should do something with this error, as I've spent 2 days for searching solution to this issue and it seems that there are a lot of users faced with it. I repeat, that I've changed nothing in my configuration, more over, I've done upgrade just from very previous version, to the next, via vCSA's management panel. So it's not the problem of changind FQDN, or IP, or upgrade from very old version and etc. I even don't have another vCenter instance, I have only one. So, it pure problem of upgrade procedure, which is used by vCSA 6.5

                          • 10. Re: 503 error on vCSA 6.5.0a running on ESXi 6.5.0a
                            Raaltink Lurker

                            We had the same error.

                            The problem was that we had a VM with a USB dongle connected.  When we unregister the VM and disconnected the USB dongle, Vcenter would start again.

                            It is a bug in Vcenter 6.5a en fixed in Vcenter 6.5b.

                            Based on the wonderful people that pointed me to /var/log/vmware/vpxd/vpxd.log - There was an error "duplicate key value violates unique constraint “pk_vpx_vm_virtual_device”" -

                            After some googling it appears that its a bug in vCenter 6.5 related to VMs with USB Devices.

                             

                            This issue has been patched with vCenter 6.5b, just search for “pk_vpx_vm_virtual_device“. and you will find the details.

                            • 11. Re: 503 error on vCSA 6.5.0a running on ESXi 6.5.0a
                              Fitou13 Lurker

                              This solution worked great.  Thanks for posting!

                              • 12. Re: 503 error on vCSA 6.5.0a running on ESXi 6.5.0a
                                ChrisDixon777 Lurker

                                I upgraded vCSA from 6.0 to 6.5GA which worked fine.

                                The 503 issue started when I then upgraded to the latest vCSA patch (6.5.0.5600 Build Number 5705665) using the Appliance Management Web UI

                                This (Postgres) fix worked for me.

                                • 13. Re: 503 error on vCSA 6.5.0a running on ESXi 6.5.0a
                                  roconnor Enthusiast

                                  Had same issue but not same cause.. see VCSA 6.5 Web GUI Error  and look for the reply from Barra87 which I've copied below

                                   

                                  On VCSA Shell:

                                  service-control --status vmware-vpxd shows "stopped"

                                  service-control --start vmware-vpxd starts the service starts for a couple of seconds and stops again

                                   

                                  VMware Support  figured out a problem with a single VM

                                  There was a problem with a network adapter on that VM (DB entry NULL)

                                   

                                  Solution that worked for us:

                                   

                                  Find VM

                                  /opt/vmware/vpostgres/current/bin/psql -d VCDB -U postgres (Login to postgres)

                                  select v.id, v.local_file_name, h.IP_ADDRESS FROM VPX_VM v, VPX_NIC n, VPX_HOST h WHERE v.id=n.entity_id and v.host_id=h.id and n.network_name=''; (get affected VM)

                                  select * from vpx_entity where id = '<YOURID>'; (get affected VM Name)

                                   

                                  the ip is the host, so nslookup to get the name

                                   

                                  Fix VM

                                  Connect to the host, the vm should have a mesage

                                  This VMis attached to a network portgroup that doesn't exist, Edit this VM and attach it to a different network

                                   

                                  Create a local vSwitch and port group

                                  (we shut the vm down, as it will lose connectivity)

                                  add the vm to the local portgroup if it hasn't assigned itself to it

                                   

                                   

                                  Restart vpxd

                                  service-control --stop --all

                                  service-control --start --all

                                   

                                  You should now be able to login to your vCenter.

                                   

                                  Another optionis in case of "Could not connect to.....:" just Find VM unregister it from vCenter and Reregister.

                                   

                                  thanks to Barra87

                                  (and guys if you are going to post on communities and expect others to read and help you out, make the post as Answered and give out some points, community sprit please)