1 2 Previous Next 19 Replies Latest reply on Oct 1, 2018 8:05 PM by AllBlack

    vCenter 6.5 - vCenter Appliance stops working

    KDA27 Novice

      Hi, this issue started 2 days ago when I attempted to update from to "VC-6.5.0U2b-Appliance-FP" (6.5.0.21000 Build Number 8815520) in the vSphere Appliance Management. It stopped at 60% and prompted with an error code about "not being able to contact the server...". I then restarted the Appliance and from there it went down hill.

      I figured out if I ran the command "Service vami-lighttp start" and  it came up again, but after some time I couldn't authenticate etc.. seems as some services aren't running.

       

      I did find this discussion in a different thread and noticed it was the exact same issue I have -> vCenter 6.5 - vcenter appliance stops working out of the blue, AGAIN!!

       

      I will post the same output and hopefully figure this out. (I did notice those 4 commands, but haven't used those yet since I don't quite understand what they do)

       

      service-control --status
      Running:
       applmgmt lwsmd pschealth vmafdd vmcad vmdird vmdnsd vmonapi vmware-cis-license vmware-cm vmware-content-library vmware-eam vmware-perfcharts vmware-psc-client vmware-rhttpproxy vmware-sca vmware-sps vmware-sts-idmd vmware-stsd vmware-updatemgr vmware-vapi-endpoint vmware-vmon vmware-vpostgres vmware-vpxd vmware-vpxd-svcs vmware-vsan-health vmware-vsm vsphere-client vsphere-ui
      Stopped:
       vmcam vmware-imagebuilder vmware-mbcs vmware-netdumper vmware-rbd-watchdog vmware-statsmonitor vmware-vcha

       

      service-control --start vmware-vpxd-svcs
      Perform start operation. vmon_profile=None, svc_names=['vmware-vpxd-svcs'], include_coreossvcs=False, include_leafossvcs=False
      2018-07-18T17:45:49.534Z   Service vpxd-svcs state STARTED
      Successfully started service vpxd-svcs

       

       

       

      systemctl list-unit-files | grep vmware
      vmware-bigsister.service               static
      vmware-cis-license.service             masked
      vmware-cm.service                      masked
      vmware-content-library.service         masked
      vmware-eam.service                     masked
      vmware-firewall.service                enabled
      vmware-imagebuilder.service            masked
      vmware-mbcs.service                    masked
      vmware-netdump.service                 masked
      vmware-perfcharts.service              masked
      vmware-pschealth.service               masked
      vmware-rbd-watchdog.service            masked
      vmware-rhttpproxy.service              masked
      vmware-sca.service                     masked
      vmware-sps.service                     masked
      vmware-statsmonitor.service            masked
      vmware-updatemgr.service               masked
      vmware-vapi.service                    masked
      vmware-vcha.service                    masked
      vmware-vmon.service                    enabled
      vmware-vmonapi.service                 masked
      vmware-vpostgres.service               masked
      vmware-vpxd-svcs.service               masked
      vmware-vpxd.service                    masked
      vmware-vsan-health.service             masked
      vmware-vsm.service                     masked
      vmware-bigsister.timer                 disabled

       

       

       

       

      After running the "service-control --start vmware-vpxd-svcs" the Appliance Manager came up again.

       

       

      systemctl list-unit-files | grep masked
      applmgmt.service                       masked
      vmcam.service                          masked
      vmware-cis-license.service             masked
      vmware-cm.service                      masked
      vmware-content-library.service         masked
      vmware-eam.service                     masked
      vmware-imagebuilder.service            masked
      vmware-mbcs.service                    masked
      vmware-netdump.service                 masked
      vmware-perfcharts.service              masked
      vmware-pschealth.service               masked
      vmware-rbd-watchdog.service            masked
      vmware-rhttpproxy.service              masked
      vmware-sca.service                     masked
      vmware-sps.service                     masked
      vmware-statsmonitor.service            masked
      vmware-updatemgr.service               masked
      vmware-vapi.service                    masked
      vmware-vcha.service                    masked
      vmware-vmonapi.service                 masked
      vmware-vpostgres.service               masked
      vmware-vpxd-svcs.service               masked
      vmware-vpxd.service                    masked
      vmware-vsan-health.service             masked
      vmware-vsm.service                     masked
      vsphere-client.service                 masked
      vsphere-ui.service                     masked
      ctrl-alt-del.target                    masked
      
      

       

       

       

      /etc/systemd/system ]# ls -lisa
      total 108
      180710 4 drwxr-xr-x 24 root root 4096 Jul 17 17:01 .
      180226 4 drwxr-xr-x  7 root root 4096 May 31 15:04 ..
      183069 0 lrwxrwxrwx  1 root root    9 Jun  6  2017 applmgmt.service -> /dev/null
      183110 4 drwxr-xr-x  2 root root 4096 Jun  6  2017 applmgmt.service.d
      182577 0 lrwxrwxrwx  1 root root   40 Feb  3  2017 default.target -> /usr/lib/systemd/system/runlevel3.target
      180712 4 drwxr-xr-x  2 root root 4096 May 31 15:03 getty.target.wants
      183131 4 drwxr-xr-x  2 root root 4096 Jun  6  2017 halt.target.wants
      188806 4 drwxr-xr-x  2 root root 4096 Feb  3  2017 local-fs.target.wants
      182743 4 drwxr-xr-x  2 root root 4096 Jul 17 23:17 lwsmd.service.d
      180714 4 drwxr-xr-x  2 root root 4096 Jul 17 19:58 multi-user.target.wants
      180718 4 drwxr-xr-x  2 root root 4096 May 31 15:03 network-online.target.wants
      183133 4 drwxr-xr-x  2 root root 4096 Jun  6  2017 poweroff.target.wants
      183129 4 drwxr-xr-x  2 root root 4096 Jun  6  2017 reboot.target.wants
      182530 4 -rw-r--r--  1 root root  268 Jun  7  2016 sendmail.service
      183127 4 drwxr-xr-x  2 root root 4096 Jun  6  2017 shutdown.target.wants
      182483 4 -rw-r--r--  1 root root  476 Mar  1 19:03 snmpd.service
      180720 4 drwxr-xr-x  2 root root 4096 May 31 15:03 sockets.target.wants
      180722 4 drwxr-xr-x  2 root root 4096 May 31 15:03 sysinit.target.wants
      182042 0 lrwxrwxrwx  1 root root   39 Feb  3  2017 syslog.service -> /usr/lib/systemd/system/rsyslog.service
      182708 4 -r-xr-xr-x  1 root root  470 Jun 13 09:50 vcha-hacheck.service
      232102 4 drwxr-xr-x  2 root root 4096 Jul 17 23:17 vmafdd.service.d
      320650 4 drwxr-xr-x  2 root root 4096 Jul 17 23:17 vmcad.service.d
      183086 0 lrwxrwxrwx  1 root root    9 Jun  6  2017 vmcam.service -> /dev/null
      182854 4 drwxr-xr-x  2 root root 4096 May  7 12:12 vmcam.service.d
      320579 4 drwxr-xr-x  2 root root 4096 Jul 17 23:17 vmdird.service.d
      320789 4 drwxr-xr-x  2 root root 4096 May  7 12:12 vmdnsd.service.d
      182138 4 drwxr-xr-x  2 root root 4096 Feb  3  2017 vmtoolsd.service.requires
      182905 0 lrwxrwxrwx  1 root root    9 Jun  6  2017 vmware-cis-license.service -> /dev/null
      182904 0 lrwxrwxrwx  1 root root    9 Jun  6  2017 vmware-cm.service -> /dev/null
      183082 0 lrwxrwxrwx  1 root root    9 Jun  6  2017 vmware-content-library.service -> /dev/null
      183083 0 lrwxrwxrwx  1 root root    9 Jun  6  2017 vmware-eam.service -> /dev/null
      183104 0 lrwxrwxrwx  1 root root    9 Jun  6  2017 vmware-imagebuilder.service -> /dev/null
      183076 0 lrwxrwxrwx  1 root root    9 Jun  6  2017 vmware-mbcs.service -> /dev/null
      183072 0 lrwxrwxrwx  1 root root    9 Jun  6  2017 vmware-netdump.service -> /dev/null
      183105 0 lrwxrwxrwx  1 root root    9 Jun  6  2017 vmware-perfcharts.service -> /dev/null
      182906 4 drwxr-xr-x  2 root root 4096 Jul 17 23:18 vmware-psc-client.service.d
      183073 0 lrwxrwxrwx  1 root root    9 Jun  6  2017 vmware-pschealth.service -> /dev/null
      183084 0 lrwxrwxrwx  1 root root    9 Jun  6  2017 vmware-rbd-watchdog.service -> /dev/null
      182903 0 lrwxrwxrwx  1 root root    9 Jun  6  2017 vmware-rhttpproxy.service -> /dev/null
      182916 0 lrwxrwxrwx  1 root root    9 Jun  6  2017 vmware-sca.service -> /dev/null
      183087 0 lrwxrwxrwx  1 root root    9 Jun  6  2017 vmware-sps.service -> /dev/null
      183074 0 lrwxrwxrwx  1 root root    9 Jun  6  2017 vmware-statsmonitor.service -> /dev/null
      183118 4 drwxr-xr-x  2 root root 4096 Jul 17 23:18 vmware-stsd.service.d
      183122 4 drwxr-xr-x  2 root root 4096 Jul 17 23:17 vmware-sts-idmd.service.d
      183093 0 lrwxrwxrwx  1 root root    9 Jun  6  2017 vmware-updatemgr.service -> /dev/null
      183060 0 lrwxrwxrwx  1 root root    9 Jun  6  2017 vmware-vapi.service -> /dev/null
      182878 0 lrwxrwxrwx  1 root root    9 Jun  6  2017 vmware-vcha.service -> /dev/null
      183067 0 lrwxrwxrwx  1 root root    9 Jun  6  2017 vmware-vmonapi.service -> /dev/null
      183114 4 drwxr-xr-x  2 root root 4096 Jul 17 23:18 vmware-vmon.service.d
      183075 0 lrwxrwxrwx  1 root root    9 Jun  6  2017 vmware-vpostgres.service -> /dev/null
      183081 0 lrwxrwxrwx  1 root root    9 Jun  6  2017 vmware-vpxd.service -> /dev/null
      182872 0 lrwxrwxrwx  1 root root    9 Jun  6  2017 vmware-vpxd-svcs.service -> /dev/null
      183094 0 lrwxrwxrwx  1 root root    9 Jun  6  2017 vmware-vsan-health.service -> /dev/null
      183103 0 lrwxrwxrwx  1 root root    9 Jun  6  2017 vmware-vsm.service -> /dev/null
      183077 0 lrwxrwxrwx  1 root root    9 Jun  6  2017 vsphere-client.service -> /dev/null
      183080 0 lrwxrwxrwx  1 root root    9 Jun  6  2017 vsphere-ui.service -> /dev/null
        • 1. Re: vCenter 6.5 - vCenter Appliance stops working
          KDA27 Novice

          I ran these two commands:

           

          systemctl daemon-reload
          

           

           

           

          service-control --start --all
          Perform start operation. vmon_profile=ALL, svc_names=None, include_coreossvcs=True, include_leafossvcs=True
          2018-07-18T18:28:11.440Z   Running command: ['/usr/bin/systemctl', 'is-enabled', u'lwsmd']
          2018-07-18T18:28:11.444Z   Done running command
          2018-07-18T18:28:11.448Z   Service lwsmd does not seem to be registered with vMon. If this is unexpected please make sure your service config is a valid json. Also check vmon logs for warnings.
          2018-07-18T18:28:11.448Z   Running command: ['/sbin/service', u'lwsmd', 'status']
          2018-07-18T18:28:11.489Z   Done running command
          Successfully started service lwsmd
          2018-07-18T18:28:11.494Z   Running command: ['/usr/bin/systemctl', 'is-enabled', u'vmafdd']
          2018-07-18T18:28:11.500Z   Done running command
          2018-07-18T18:28:11.509Z   Service vmafdd does not seem to be registered with vMon. If this is unexpected please make sure your service config is a valid json. Also check vmon logs for warnings.
          2018-07-18T18:28:11.509Z   Running command: ['/sbin/service', u'vmafdd', 'status']
          2018-07-18T18:28:11.537Z   Done running command
          Successfully started service vmafdd
          2018-07-18T18:28:11.541Z   Running command: ['/usr/bin/systemctl', 'is-enabled', u'vmdird']
          2018-07-18T18:28:11.550Z   Done running command
          2018-07-18T18:28:11.557Z   Service vmdird does not seem to be registered with vMon. If this is unexpected please make sure your service config is a valid json. Also check vmon logs for warnings.
          2018-07-18T18:28:11.557Z   Running command: ['/sbin/service', u'vmdird', 'status']
          2018-07-18T18:28:11.590Z   Done running command
          Successfully started service vmdird
          2018-07-18T18:28:11.594Z   Running command: ['/usr/bin/systemctl', 'is-enabled', u'vmcad']
          2018-07-18T18:28:11.600Z   Done running command
          2018-07-18T18:28:11.603Z   Service vmcad does not seem to be registered with vMon. If this is unexpected please make sure your service config is a valid json. Also check vmon logs for warnings.
          2018-07-18T18:28:11.603Z   Running command: ['/sbin/service', u'vmcad', 'status']
          2018-07-18T18:28:11.628Z   Done running command
          Successfully started service vmcad
          2018-07-18T18:28:11.632Z   Running command: ['/usr/bin/systemctl', 'is-enabled', u'vmware-sts-idmd']
          2018-07-18T18:28:11.638Z   Done running command
          2018-07-18T18:28:11.641Z   Service vmware-sts-idmd does not seem to be registered with vMon. If this is unexpected please make sure your service config is a valid json. Also check vmon logs for warnings.
          2018-07-18T18:28:11.641Z   Running command: ['/sbin/service', u'vmware-sts-idmd', 'status']
          2018-07-18T18:28:11.695Z   Done running command
          Successfully started service vmware-sts-idmd
          2018-07-18T18:28:11.700Z   Running command: ['/usr/bin/systemctl', 'is-enabled', u'vmware-stsd']
          2018-07-18T18:28:11.713Z   Done running command
          2018-07-18T18:28:11.717Z   Service vmware-stsd does not seem to be registered with vMon. If this is unexpected please make sure your service config is a valid json. Also check vmon logs for warnings.
          2018-07-18T18:28:11.717Z   Running command: ['/sbin/service', u'vmware-stsd', 'status']
          2018-07-18T18:28:11.743Z   Done running command
          Successfully started service vmware-stsd
          2018-07-18T18:28:11.747Z   Running command: ['/usr/bin/systemctl', 'is-enabled', u'vmdnsd']
          2018-07-18T18:28:11.753Z   Done running command
          2018-07-18T18:28:11.757Z   Service vmdnsd does not seem to be registered with vMon. If this is unexpected please make sure your service config is a valid json. Also check vmon logs for warnings.
          2018-07-18T18:28:11.757Z   Running command: ['/sbin/service', u'vmdnsd', 'status']
          2018-07-18T18:28:11.783Z   Done running command
          Successfully started service vmdnsd
          2018-07-18T18:28:11.787Z   Running command: ['/usr/bin/systemctl', 'is-enabled', u'vmware-psc-client']
          2018-07-18T18:28:11.793Z   Done running command
          2018-07-18T18:28:11.797Z   Service vmware-psc-client does not seem to be registered with vMon. If this is unexpected please make sure your service config is a valid json. Also check vmon logs for warnings.
          2018-07-18T18:28:11.797Z   Running command: ['/sbin/service', u'vmware-psc-client', 'status']
          2018-07-18T18:28:11.823Z   Done running command
          Successfully started service vmware-psc-client
          Service-control failed. Error Failed to start vmon services.vmon-cli RC=5, stderr=Failed to start statsmonitor services. Error: Operation not allowed in current service state
          
          • 2. Re: vCenter 6.5 - vCenter Appliance stops working
            daphnissov Guru
            vExpertCommunity Warriors

            And guessing here that you didn't snapshot your vCSA before updating, correct?

            • 3. Re: vCenter 6.5 - vCenter Appliance stops working
              KDA27 Novice

              Hi daphnissov,

               

              Yes, that is correct. I did read in a forum about reverting after a failed update.
              I'm not sure if that would have helped in this case, but I should have made at least at few snapshots with different versions.

               

              Kind Regards,
              Kaspar Danielsen

              • 4. Re: vCenter 6.5 - vCenter Appliance stops working
                daphnissov Guru
                vExpertCommunity Warriors

                It certainly would have helped in this case (you would simply revert to that snapshot), but seeing as how you didn't take a snapshot the point is moot.

                • 5. Re: vCenter 6.5 - vCenter Appliance stops working
                  KDA27 Novice

                  Hi daphnissov,

                   

                  I'm aware of the possibility to revert back after a failed installation.. the strange thing is that I've never really had any issues up until now. The version I'm running could of cause be bugged.

                   

                  The strange part is, I actually got the Appliance Management up an running by starting all services and running the two different commands in this thread.

                   

                  I still struggle accessing the client interface - I do get the message that it's initializing while it's restarting and after that... nothing. (It does respond on IP)

                   

                  This actually happened after I uploaded a new ISO for v.6.7. The Appliance Management does become un-responsive as well, after attempting to update the Appliance.. maaan!

                   

                  Kind Regards,
                  Kaspar Danielsen

                  • 6. Re: vCenter 6.5 - vCenter Appliance stops working
                    KDA27 Novice

                    I did look into this one -> VCSA 6.5 fails to start File System Check and Network Service - Nolabnoparty

                     

                    I still cannot figure out how to open up the log's within /var/…. etc.

                     

                    Kind Regards,
                    Kaspar Danielsen

                    • 7. Re: vCenter 6.5 - vCenter Appliance stops working
                      KDA27 Novice

                      I found this one too - mine just stops at 60%(the step "Pre install Scripts") -> https://vdr.one/updating-vcsa-using-vami-from-6-0u1-to-6-0u2-stuck-at-70/

                       

                      Kind Regards,
                      Kaspar Danielsen

                      • 8. Re: vCenter 6.5 - vCenter Appliance stops working
                        KDA27 Novice

                        Hi,

                         

                        I managed to get the vSphere Client and the Appliance Manager to run at the same time, by shutting down a few services and running some of them in order + starting all services (don't ask how I managed to get it running).

                         

                        I'm launching a update on the Appliance Manager -> Goes straight to 60% -> Hangs for 1-2min -> Error: Cannot reach the appliance server. The server could be down or it's a network issue. You can try again after resolving this issue.

                         

                        service-control --status
                        StartPending:
                         vmware-content-library vsphere-client
                        Running:
                         applmgmt lwsmd pschealth vmafdd vmcad vmdird vmdnsd vmonapi vmware-cis-license vmware-cm vmware-eam vmware-psc-client vmware-rhttpproxy vmware-sca vmware-sps vmware-statsmonitor vmware-sts-idmd vmware-stsd vmware-updatemgr vmware-vapi-endpoint vmware-vmon vmware-vpostgres vmware-vpxd vmware-vpxd-svcs vmware-vsan-health vmware-vsm vsphere-ui
                        Stopped:
                         vmcam vmware-imagebuilder vmware-mbcs vmware-netdumper vmware-perfcharts vmware-rbd-watchdog vmware-vcha
                        

                         

                        How do I troubleshoot this?

                         

                        Kind Regards,
                        Kaspar Danielsen

                        • 9. Re: vCenter 6.5 - vCenter Appliance stops working
                          sk84 Expert
                          vExpert

                          A broken or incomplete update is often very difficult to debug, as some files have probably already been replaced and some configuration changes may have been made. It's hard to find out what exactly no longer fits together.

                           

                           

                          It'll probably go faster if you recreate the vCenter from scratch. If you are not using a dvSwitch, you can simple disconnect the ESXi hosts with the running VMs from the existing vCenter and add them to the new vCenter.

                           

                          Best regards,

                          Sebastian

                          • 10. Re: vCenter 6.5 - vCenter Appliance stops working
                            KDA27 Novice

                            Hi sk84,

                             

                            Thank you for your feedback in this thread.

                             

                            I'm not able to deploy a new  vSphere Appliance and make the necessary configurations.

                             

                            I've been looking more into, what's going wrong at right now it looks like this known issue -> vcsa 6.5 update - VAMI not working (SOLVED)

                             

                            I can't figure out how to read the log.. hmm

                             

                            /opt/vmware/etc/lighttpd/lighttpd.conf
                            bash: /opt/vmware/etc/lighttpd/lighttpd.conf: Permission denied
                            

                             

                             

                            Kind Regards,

                            Kaspar Danielsen

                            • 11. Re: vCenter 6.5 - vCenter Appliance stops working
                              sk84 Expert
                              vExpert

                              Hi Kaspar,

                               

                              Files with the extension ".conf" are configuration files and not log files. Log files have the ending ".log" and are usually located in the path "/var/logs/vmware/".

                               

                              You can use linux commands to read the files (or edit them):

                               

                              "less" is for reading, for example.

                              "vi" is for editing.

                               

                              If you need help with these 2 tools, I recommend to read the linux man pages:

                              less Man Page - Bash - SS64.com

                              The vi Editor - A quick reference guide. | SS64.com

                               

                              Best regards,

                              Sebastian

                              • 12. Re: vCenter 6.5 - vCenter Appliance stops working
                                KDA27 Novice

                                Hi sk84,

                                 

                                It's much appreciated!

                                 

                                I'll give it a shot tomorrow - it's getting past midnight already.

                                 

                                Thank you.

                                 

                                Kind Regards,
                                Kaspar Danielsen

                                • 13. Re: vCenter 6.5 - vCenter Appliance stops working
                                  daphnissov Guru
                                  vExpertCommunity Warriors

                                  At this point, why aren't you opening a SR with VMware? If you can't deploy a new vCSA, as you said, and you didn't snapshot it, this would seem to be your next best option.

                                  • 14. Re: vCenter 6.5 - vCenter Appliance stops working
                                    KDA27 Novice

                                    Hi daphnissov,

                                     

                                    I've created a case and have to pay at least 300$ for each incident (at the best).

                                     

                                    I have a Dell consultant for this task and he's on vacation at the moment.

                                     

                                    Kind Regards,
                                    Kaspar Danielsen

                                    1 2 Previous Next