7 Replies Latest reply on Jun 19, 2018 1:41 AM by Finikiez

    vCenter 6.0 Issues

    Viceman256 Lurker

      I'll preface this with the fact that my VMWare knowledge is very limited. I know how to use vSphere and connect to a host to make simple changes. The issue is, for one of our clients we have a vCenter set up and we have Veeam backing it up. Backups started to fail with authentication errors, so I tried connecting to vCenter via vSphere and received login errors as well. I rebooted the vCenter, and now I receive 503 errors if I try to login via vSphere or the web.

       

      After searching online, I saw documentation saying this could be caused by the storage being full. I tried allocating more storage from the host, but the host is being managed by the vCenter I guess so it won't let me. I was able to access it by it's IP:5480 but it logged me out immediately every time. Connected to it via shell by editing the boot sequence and accessing shell, then resetting the root pw and setting it to never expire. I could then login and it alerted me storage was low there too. I enabled SSH and connected to it and checked it using df -h. It was using 100% storage, and it appeared most was coming from the storage/logs. I cleared it out using info from KBs and now I'm at 53% usage. I also ran an update from the 5480 page and now it says there are no more, the current version is 6.0.0.30500.

       

      I saw a forum post mentioning that someone fixed it by upgrading it to 6.5. Well I downloaded the installer and attempted this, but it won't connect to it to upgrade it.

       

      Also just tried this but it didn't work:

      Unable to Connect to VCSA - Unknown Connection Error (503 Server Unavailable)

       

      I still have the same errors, but I can access it via SSH. I confirmed the IP and the loopback are listed in the hosts file. I've restarted all the services I could find.. I've attached some screenshots and I also pulled the vm-support package but it's over 12 GB and I'm not sure how to get anyone access to it as this would take two weeks to upload on their network. I also have no idea how to read these so any insight on what I need to check here would help too! I feel like it might be a simple fix, but I've read through so many articles and forum posts and I have no idea what I'm missing here.

       

      Thank you ahead of time!

       

      - Vince

        • 1. Re: vCenter 6.0 Issues
          Finikiez Master
          vExpert

          Hello!

           

          Can you show the following command output from CLI

           

          service-control --status all

           

          Also you may hit this issue vCenter 6.5 - vcenter appliance stops working out of the blue, AGAIN!!

          • 2. Re: vCenter 6.0 Issues
            Viceman256 Lurker

            Hi Finikiez,

             

            Regarding that post, if I try to browse to that folder they posted about, it says it doesn't exist:

            bbfc-vcenter:/etc # cd /etc/systemd/system

            bash: cd: /etc/systemd/system: No such file or directory

            bbfc-vcenter:/etc #

             

            It appears systemd doesn't exist, and if I try to run systemctl, it says "Command not found".

             

            When I run that command you posted from SSH, it displays: "Unknown services: all". If I run service-control --status it displays:

             

            "service-control --status

            INFO:root:Service: vmware-rhttpproxy, Action: status

            Service: vmware-rhttpproxy, Action: status

            2018-06-14T18:17:47.244Z   Running command: ['/sbin/service', u'vmware-rhttpprox                                                                             y', 'status']

            2018-06-14T18:17:47.279Z   Done running command

            INFO:root:Service: vmware-cis-license, Action: status

            Service: vmware-cis-license, Action: status

            2018-06-14T18:17:47.280Z   Running command: ['/sbin/service', u'vmware-cis-licen                                                                             se', 'status']

            2018-06-14T18:17:47.418Z   Done running command

            INFO:root:Service: vmware-psc-client, Action: status

            Service: vmware-psc-client, Action: status

            2018-06-14T18:17:47.418Z   Running command: ['/sbin/service', u'vmware-psc-clien                                                                             t', 'status']

            2018-06-14T18:17:47.449Z   Done running command

            INFO:root:Service: vmware-sca, Action: status

            Service: vmware-sca, Action: status

            2018-06-14T18:17:47.450Z   Running command: ['/sbin/service', u'vmware-sca', 'st                                                                             atus']

            2018-06-14T18:17:47.540Z   Done running command

            INFO:root:Service: vmware-netdumper, Action: status

            Service: vmware-netdumper, Action: status

            2018-06-14T18:17:47.540Z   Running command: ['/sbin/service', u'vmware-netdumper                                                                             ', 'status']

            2018-06-14T18:17:47.585Z   Done running command

            INFO:root:Service: vmware-syslog, Action: status

            Service: vmware-syslog, Action: status

            2018-06-14T18:17:47.585Z   Running command: ['/sbin/service', u'vmware-syslog',                                                                              'status']

            2018-06-14T18:17:47.637Z   Done running command

            INFO:root:Service: vmware-syslog-health, Action: status

            Service: vmware-syslog-health, Action: status

            2018-06-14T18:17:47.638Z   Running command: ['/sbin/service', u'vmware-syslog-he                                                                             alth', 'status']

            2018-06-14T18:17:47.750Z   Done running command

            INFO:root:Service: vmware-vapi-endpoint, Action: status

            Service: vmware-vapi-endpoint, Action: status

            2018-06-14T18:17:47.750Z   Running command: ['/sbin/service', u'vmware-vapi-endp                                                                             oint', 'status']

            2018-06-14T18:17:47.798Z   Done running command

            INFO:root:Service: vmware-vpostgres, Action: status

            Service: vmware-vpostgres, Action: status

            2018-06-14T18:17:47.799Z   Running command: ['/sbin/service', u'vmware-vpostgres                                                                             ', 'status']

            2018-06-14T18:17:47.924Z   Done running command

            INFO:root:Service: vmware-invsvc, Action: status

            Service: vmware-invsvc, Action: status

            2018-06-14T18:17:47.925Z   Running command: ['/sbin/service', u'vmware-invsvc',                                                                              'status']

            2018-06-14T18:17:48.025Z   Done running command

            INFO:root:Service: vmware-mbcs, Action: status

            Service: vmware-mbcs, Action: status

            2018-06-14T18:17:48.025Z   Running command: ['/sbin/service', u'vmware-mbcs', 's                                                                             tatus']

            2018-06-14T18:17:48.123Z   Done running command

            INFO:root:Service: vmware-vpxd, Action: status

            Service: vmware-vpxd, Action: status

            2018-06-14T18:17:48.124Z   Running command: ['/sbin/service', u'vmware-vpxd', 's                                                                             tatus']

            2018-06-14T18:17:48.509Z   Done running command

            INFO:root:Service: vmware-eam, Action: status

            Service: vmware-eam, Action: status

            2018-06-14T18:17:48.509Z   Running command: ['/sbin/service', u'vmware-eam', 'st                                                                             atus']

            2018-06-14T18:17:48.575Z   Done running command

            INFO:root:Service: vmware-rbd-watchdog, Action: status

            Service: vmware-rbd-watchdog, Action: status

            2018-06-14T18:17:48.575Z   Running command: ['/sbin/service', u'vmware-rbd-watch                                                                             dog', 'status']

            2018-06-14T18:17:48.617Z   Done running command

            INFO:root:Service: vmware-sps, Action: status

            Service: vmware-sps, Action: status

            2018-06-14T18:17:48.617Z   Running command: ['/sbin/service', u'vmware-sps', 'st                                                                             atus']

            2018-06-14T18:17:48.747Z   Done running command

            INFO:root:Service: vmware-vdcs, Action: status

            Service: vmware-vdcs, Action: status

            2018-06-14T18:17:48.748Z   Running command: ['/sbin/service', u'vmware-vdcs', 's                                                                             tatus']

            2018-06-14T18:17:48.838Z   Done running command

            INFO:root:Service: vmware-vpx-workflow, Action: status

            Service: vmware-vpx-workflow, Action: status

            2018-06-14T18:17:48.838Z   Running command: ['/sbin/service', u'vmware-vpx-workf                                                                             low', 'status']

            2018-06-14T18:17:48.894Z   Done running command

            INFO:root:Service: vmware-vsan-health, Action: status

            Service: vmware-vsan-health, Action: status

            2018-06-14T18:17:48.894Z   Running command: ['/sbin/service', u'vmware-vsan-heal                                                                             th', 'status']

            2018-06-14T18:17:48.902Z   Done running command

            INFO:root:Service: vmware-vsm, Action: status

            Service: vmware-vsm, Action: status

            2018-06-14T18:17:48.903Z   Running command: ['/sbin/service', u'vmware-vsm', 'st                                                                             atus']

            2018-06-14T18:17:48.953Z   Done running command

            INFO:root:Service: vsphere-client, Action: status

            Service: vsphere-client, Action: status

            2018-06-14T18:17:48.953Z   Running command: ['/sbin/service', u'vsphere-client',                                                                              'status']

            2018-06-14T18:17:49.045Z   Done running command

            INFO:root:Service: vmware-perfcharts, Action: status

            Service: vmware-perfcharts, Action: status

            2018-06-14T18:17:49.045Z   Running command: ['/sbin/service', u'vmware-perfchart                                                                             s', 'status']

            2018-06-14T18:17:49.106Z   Done running command

            INFO:root:Service: vmware-vws, Action: status

            Service: vmware-vws, Action: status

            2018-06-14T18:17:49.107Z   Running command: ['/sbin/service', u'vmware-vws', 'st                                                                             atus']

            2018-06-14T18:17:49.168Z   Done running command

            INFO:root:Running:

            vmware-cis-license (VMware License Service) vmware-perfcharts (VMware Performan                                                                             ce Charts) vmware-psc-client (VMware Platform Services Controller Client) vmware                                                                             -rhttpproxy (VMware HTTP Reverse Proxy) vmware-sps (VMware vSphere Profile-Drive                                                                             n Storage Service) vmware-syslog (VMware Common Logging Service) vmware-syslog-h                                                                             ealth (VMware Syslog Health Service) vmware-vpostgres (VMware Postgres) vmware-v                                                                             san-health (VMware VSAN Health Service) vmware-vws (VMware System and Hardware H                                                                             ealth Manager)

            Running:

            vmware-cis-license (VMware License Service) vmware-perfcharts (VMware Performan                                                                             ce Charts) vmware-psc-client (VMware Platform Services Controller Client) vmware                                                                             -rhttpproxy (VMware HTTP Reverse Proxy) vmware-sps (VMware vSphere Profile-Drive                                                                             n Storage Service) vmware-syslog (VMware Common Logging Service) vmware-syslog-h                                                                             ealth (VMware Syslog Health Service) vmware-vpostgres (VMware Postgres) vmware-v                                                                             san-health (VMware VSAN Health Service) vmware-vws (VMware System and Hardware H                                                                             ealth Manager)

            INFO:root:Stopped:

            vmware-eam (VMware ESX Agent Manager) vmware-invsvc (VMware Inventory Service)                                                                              vmware-mbcs (VMware Message Bus Configuration Service) vmware-netdumper (VMware                                                                              vSphere ESXi Dump Collector) vmware-rbd-watchdog (VMware vSphere Auto Deploy Wai                                                                             ter) vmware-sca (VMware Service Control Agent) vmware-vapi-endpoint (VMware vAPI                                                                              Endpoint) vmware-vdcs (VMware Content Library Service) vmware-vpx-workflow (VMw                                                                             are vCenter Workflow Manager) vmware-vpxd (VMware vCenter Server) vmware-vsm (VM                                                                             ware vService Manager) vsphere-client ()

            Stopped:

            vmware-eam (VMware ESX Agent Manager) vmware-invsvc (VMware Inventory Service)                                                                              vmware-mbcs (VMware Message Bus Configuration Service) vmware-netdumper (VMware                                                                              vSphere ESXi Dump Collector) vmware-rbd-watchdog (VMware vSphere Auto Deploy Wai                                                                             ter) vmware-sca (VMware Service Control Agent) vmware-vapi-endpoint (VMware vAPI                                                                              Endpoint) vmware-vdcs (VMware Content Library Service) vmware-vpx-workflow (VMw                                                                             are vCenter Workflow Manager) vmware-vpxd (VMware vCenter Server) vmware-vsm (VM                                                                             ware vService Manager) vsphere-client ()"

            • 3. Re: vCenter 6.0 Issues
              Finikiez Master
              vExpert

              I missed that you use vcsa 6.0

               

              You can't connect to VC because main services are stopped vmware-invsvc, vmware-vapi-endpoint, vmware-vpxd and vsphere-client

               

              Try to start them manually and see what happened.

              If they can't be started manually then you need to check the logs in /var/log/vmware directory

              1 person found this helpful
              • 4. Re: vCenter 6.0 Issues
                Viceman256 Lurker

                I can start the vmware-vapi-endpoint and vsphere-client. I tried the vmware-vpxd then it got stuck for several minutes and then I cancelled the start, but when I check the status it says running.  When I try to start vmware-invsvs I get the errors:

                 

                bbfc-vcenter:~ # service vmware-invsvc start

                Starting VMware Inventory Service...

                Waiting for VMware Inventory Service..............

                WARNING: VMware Inventory Service may have failed to start.

                bbfc-vcenter:~ # service vmware-invsvc status

                VMware Inventory Service is not running.

                 

                I'm not sure which logs to check in that directory.

                • 5. Re: vCenter 6.0 Issues
                  Finikiez Master
                  vExpert

                  Sorry, notifications are broken and I can't reply in time.

                   

                  Can you go to /var/log/vmware/invsvc and check latest ds.log

                  There could inventory sevice database corruption. However it's better to check the logs.

                  • 6. Re: vCenter 6.0 Issues
                    Viceman256 Lurker

                    Hi ,

                     

                    When I browse to that directory, it actually has no ds.log file there.

                     

                    Thanks,

                     

                    Vince

                    • 7. Re: vCenter 6.0 Issues
                      Finikiez Master
                      vExpert

                      I don't have VC 6.0 installed at the moment. Can you show a schreenshot? or can you check if there is a Logs directory or something on that path?