12 Replies Latest reply on Mar 27, 2019 3:32 AM by Bloodmaker

    vcenter service 6.5 stopped and is not starting

    ehsan7715 Lurker

      hello my vcenter server appliance 6.5 suddenly stop working and vcenter main webpage https://vcenter_name/ show this error to me :

       

      503 Service Unavailable (Failed to connect to endpoint: [N7Vmacore4Http20NamedPipeServiceSpecE:0x00007f4b700dc900] _serverNamespace = / action = Allow _pipeName =/var/run/vmware/vpxd-webserver-pipe)

       

      and i can access server through ssh and see that vpxd service is stopped and can not start

       

      root@vcenter [ ~ ]# service-control --start vmware-vpxd

      Perform start operation. vmon_profile=None, svc_names=['vmware-vpxd'], include_coreossvcs=False, include_leafossvcs=False

      2016-12-24T06:47:18.538Z   Service vpxd state STOPPED

      Error executing start on service vpxd. Details {

          "resolution": null,

          "detail": [

              {

                  "args": [

                      "vpxd"

                  ],

                  "id": "install.ciscommon.service.failstart",

                  "localized": "An error occurred while starting service 'vpxd'",

                  "translatable": "An error occurred while starting service '%(0)s'"

              }

          ],

          "componentKey": null,

          "problemId": null

      }

      Service-control failed. Error {

          "resolution": null,

          "detail": [

              {

                  "args": [

                      "vpxd"

                  ],

                  "id": "install.ciscommon.service.failstart",

                  "localized": "An error occurred while starting service 'vpxd'",

                  "translatable": "An error occurred while starting service '%(0)s'"

              }

          ],

          "componentKey": null,

          "problemId": null

      }

       

      how can i solve this problem

        • 1. Re: vcenter service 6.5 stopped and is not starting
          stevegrass Lurker

          Issue has now been resolved and web client is now fully functional again.

           

          It appears the following accounts need to be added to "Log on as a Batch job".

           

          GPO Path: Computer Configuration -> Windows Settings ->Security Settings ->Local Policies -> User Rights Assignment ->Log on as a batch job

           

          Accounts:

            cm

            content-library

            eam

            imagebuilder

            mbcs !

            netdumper

            perfcharts

            rbd

            vapiEndpoint

            vmware-vpostgres

            vsan-health

            vsm

            vsphere-client

            vsphere-ui

           

          gpo.png

          Reboot vCenter.

          • 2. Re: vcenter service 6.5 stopped and is not starting
            DianaTT Lurker

            Hi,

             

            Does anyone know how I can do the same on a Linux VCSA?

            Thanks

            • 3. Re: vcenter service 6.5 stopped and is not starting
              edera1218 Lurker

              Hi Diana,

              Have you figured out how to make the changes above on an vCenter appliance yet? If so, can you post it?

               

              Thanks!


              Dee

              • 4. Re: vcenter service 6.5 stopped and is not starting
                Brian_007 Novice

                I just had this issue and apparently the PSC was having issue.  I tried starting the VC several times but the required services weren't restarting.

                 

                After opening a support case we had to restart the PSC, wait until all the services  restarted (usually several minutes after you lose ping during restart).

                Then restart the VC and wait until all the services restart and retest the web client.

                 

                This is on a Linux VCSA for reference.

                • 5. Re: vcenter service 6.5 stopped and is not starting
                  DianaTT Lurker

                  Hi Edera1218,

                   

                   

                  After digging through logs I realised the vmware-vpxd wasn't starting and came up with errors similer to the ones in this article:

                   

                  https://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2149010

                  The SSO pass reset reconnected it to the PSC and the web client is working but now the whole networking side needs to be reconfigured,

                   

                  Brian_007 : We did try to play with the PSC but it would only bring the vcenter up for a short while

                   

                   

                  I hope that helps,

                  • 6. Re: vcenter service 6.5 stopped and is not starting
                    Brian_007 Novice

                    What generated this issue for me is that the PSC was running out of disk space because of Java Core dumps in /storage/core

                    Reference here for steps to see this: Increasing the disk space for the VMware vCenter Server Appliance in vSphere 6.0 (2126276) | VMware KB

                    After you delete some of the dumps you can restart your PSC, wait 5 minutes until all the services restart and then restart your VC.  After about 5 minutes to allow for the services to restart then everything was looking good.

                     

                    Now why Java Core is dumping is an open case for me and I'm waiting for a patch release so the above is just a workaround until i get a patch.

                    • 7. Re: vcenter service 6.5 stopped and is not starting
                      JLawrence919 Lurker

                      I had the same issue when i tried to start the service on my windows vSphere 6.5 server...

                      root@vcenter [ ~ ]# service-control --start vmware-vpxd

                      Perform start operation. vmon_profile=None, svc_names=['vmware-vpxd'], include_coreossvcs=False, include_leafossvcs=False

                      2016-12-24T06:47:18.538Z   Service vpxd state STOPPED

                      Error executing start on service vpxd. Details {

                          "resolution": null,

                          "detail": [

                              {

                                  "args": [

                                      "vpxd"

                                  ],

                                  "id": "install.ciscommon.service.failstart",

                                  "localized": "An error occurred while starting service 'vpxd'",

                                  "translatable": "An error occurred while starting service '%(0)s'"

                              }

                          ],

                          "componentKey": null,

                          "problemId": null

                      }

                      Service-control failed. Error {

                          "resolution": null,

                          "detail": [

                              {

                                  "args": [

                                      "vpxd"

                                  ],

                                  "id": "install.ciscommon.service.failstart",

                                  "localized": "An error occurred while starting service 'vpxd'",

                                  "translatable": "An error occurred while starting service '%(0)s'"

                              }

                          ],

                          "componentKey": null,

                          "problemId": null

                      }

                       

                      I think my issue was that our GPO policy is extremely restrictive with user accounts, so I had to set a password for all the vmware accounts. I did notice that when i started the service from the vpshere CLI, a security event would show an incorrect password event or the account was locked out.

                       

                      before i found this fix, the only way that i could get my vsphere server to work was to reinstall every week... for some reason it would work for a few days and then die...

                       

                      NOTE: Changing the GPO policy for "Log on as a batch job" did NOT work for me and I removed the policy.

                       

                      1. Go to computer management, Local Users and Groups, Users
                      2. Change the password of the vmware service accounts and make sure they are not locked out
                      3. change the password for the vmware service account via the vsphere CLI
                      4. bring up an administrator powershell console and go to: C:\Program Files\VMware\vCenter Server\vmon\
                      5. run .\vmon.exe –p <service account> (Example .\vmon.exe –p vsphere-client)
                        1. When prompted for new password, enter new password
                        2. NOTE: <run the command again to change the password – I found that running it twice makes sure it works>
                      6. Now start the associated vsphere service (from C:\Program Files\VMware\vCenter Server\bin)
                        1. run: .\service-control –-start vpshere-client
                      • 8. Re: vcenter service 6.5 stopped and is not starting
                        sholliday_aus Lurker

                        Had this same error at a customer site.

                         

                        They had unsupported HAProxy load-balancing the 2 x PSCs and there was an issue with the HAProxy.

                        Once they fixed HAProxy I was able to start the vmware-vpxd-svcs and everything came good.

                         

                        If using external PSCs with load-balancing ensure you can get to your VIP before starting VCSA.

                        • 9. Re: vcenter service 6.5 stopped and is not starting
                          lazorlash Lurker

                          I just had this EXACT issue (Windows 2012 vCenter) and your post was right on the money...

                           

                          We have restrictive GPOs in place aswell and I noticed all the local vsphere accounts had set to "change on next logon" suggesting that they had all expired.

                           

                          Nothing in the logs suggesting failed logins etc...

                           

                          service-control showed the following services failing to start:

                           

                          Stopped:

                          EsxAgentManager VMWareCAMService VServiceManager content-library mbcs vPostgres

                          vapiEndpoint vimPBSM vmsyslogcollector vmware-autodeploy-waiter vmware-imagebui

                          lder vmware-network-coredump vmware-perfcharts vpxd vpxd-svcs vsan-health vspher

                          e-ui vspherewebclientsvc

                           

                          If you look at the local service accounts;

                           

                          cm

                          content-library

                          eam

                          imagebuilder

                          mbcs

                          netdumper

                          perfcharts

                          rbd

                          vapiEndpoint

                          vmware-vpostgres

                          vsan-health

                          vsm

                          vsphere-client

                          vsphere-ui

                           

                          The failed services list is suspiciously close to the names of these accounts so if you're seeing the same list of stopped/failed services, it's probably the local accounts!

                           

                          Thanks for the help here... saved me a lot of hassle.

                          • 10. Re: vcenter service 6.5 stopped and is not starting
                            emitchell365 Lurker

                            Great post. Changing the password worked for me as well.

                            • 11. Re: vcenter service 6.5 stopped and is not starting
                              TMQ Lurker

                              Hi guy,

                               

                              I facing some issue when i enable the Bypass Traverse Checking which cause the vCenter web access unable to launch.

                              Anyone got any solution to it.

                              I had added those service account under log on as a batch. which also unable to resolve it.

                               

                              Anyone?

                              • 12. Re: vcenter service 6.5 stopped and is not starting
                                Bloodmaker Lurker

                                please check your DNS server is working or not

                                检查你的DNS服务器是否正常

                                I had the same problem.Soved:be sure the DNS is reachble.

                                我有一样的问题,解决:确保DNS可达。