vCenter 6.5 - vCenter Appliance stops working

[KDA27]

Hi, this issue started 2 days ago when I attempted to update from 6.5.0.20100 to "VC-6.5.0U2b-Appliance-FP" (6.5.0.21000 Build Number 8815520) in the vSphere Appliance Management. It stopped at 60% and prompted with an error code about "not being able to contact the server...". I then restarted the Appliance and from there it went down hill.

I figured out if I ran the command "Service vami-lighttp start" and  it came up again, but after some time I couldn't authenticate etc.. seems as some services aren't running.

I did find this discussion in a different thread and noticed it was the exact same issue I have -> vCenter 6.5 - vcenter appliance stops working out of the blue, AGAIN!!

I will post the same output and hopefully figure this out. (I did notice those 4 commands, but haven't used those yet since I don't quite understand what they do)

service-control --status
Running:
 applmgmt lwsmd pschealth vmafdd vmcad vmdird vmdnsd vmonapi vmware-cis-license vmware-cm vmware-content-library vmware-eam vmware-perfcharts vmware-psc-client vmware-rhttpproxy vmware-sca vmware-sps vmware-sts-idmd vmware-stsd vmware-updatemgr vmware-vapi-endpoint vmware-vmon vmware-vpostgres vmware-vpxd vmware-vpxd-svcs vmware-vsan-health vmware-vsm vsphere-client vsphere-ui
Stopped:
 vmcam vmware-imagebuilder vmware-mbcs vmware-netdumper vmware-rbd-watchdog vmware-statsmonitor vmware-vcha

service-control --start vmware-vpxd-svcs
Perform start operation. vmon_profile=None, svc_names=['vmware-vpxd-svcs'], include_coreossvcs=False, include_leafossvcs=False
2018-07-18T17:45:49.534Z   Service vpxd-svcs state STARTED
Successfully started service vpxd-svcs

systemctl list-unit-files | grep vmware
vmware-bigsister.service               static
vmware-cis-license.service             masked
vmware-cm.service                      masked
vmware-content-library.service         masked
vmware-eam.service                     masked
vmware-firewall.service                enabled
vmware-imagebuilder.service            masked
vmware-mbcs.service                    masked
vmware-netdump.service                 masked
vmware-perfcharts.service              masked
vmware-pschealth.service               masked
vmware-rbd-watchdog.service            masked
vmware-rhttpproxy.service              masked
vmware-sca.service                     masked
vmware-sps.service                     masked
vmware-statsmonitor.service            masked
vmware-updatemgr.service               masked
vmware-vapi.service                    masked
vmware-vcha.service                    masked
vmware-vmon.service                    enabled
vmware-vmonapi.service                 masked
vmware-vpostgres.service               masked
vmware-vpxd-svcs.service               masked
vmware-vpxd.service                    masked
vmware-vsan-health.service             masked
vmware-vsm.service                     masked
vmware-bigsister.timer                 disabled

After running the "service-control --start vmware-vpxd-svcs" the Appliance Manager came up again.

systemctl list-unit-files | grep masked
applmgmt.service                       masked
vmcam.service                          masked
vmware-cis-license.service             masked
vmware-cm.service                      masked
vmware-content-library.service         masked
vmware-eam.service                     masked
vmware-imagebuilder.service            masked
vmware-mbcs.service                    masked
vmware-netdump.service                 masked
vmware-perfcharts.service              masked
vmware-pschealth.service               masked
vmware-rbd-watchdog.service            masked
vmware-rhttpproxy.service              masked
vmware-sca.service                     masked
vmware-sps.service                     masked
vmware-statsmonitor.service            masked
vmware-updatemgr.service               masked
vmware-vapi.service                    masked
vmware-vcha.service                    masked
vmware-vmonapi.service                 masked
vmware-vpostgres.service               masked
vmware-vpxd-svcs.service               masked
vmware-vpxd.service                    masked
vmware-vsan-health.service             masked
vmware-vsm.service                     masked
vsphere-client.service                 masked
vsphere-ui.service                     masked
ctrl-alt-del.target                    masked

/etc/systemd/system ]# ls -lisa
total 108
180710 4 drwxr-xr-x 24 root root 4096 Jul 17 17:01 .
180226 4 drwxr-xr-x  7 root root 4096 May 31 15:04 ..
183069 0 lrwxrwxrwx  1 root root    9 Jun  6  2017 applmgmt.service -> /dev/null
183110 4 drwxr-xr-x  2 root root 4096 Jun  6  2017 applmgmt.service.d
182577 0 lrwxrwxrwx  1 root root   40 Feb  3  2017 default.target -> /usr/lib/systemd/system/runlevel3.target
180712 4 drwxr-xr-x  2 root root 4096 May 31 15:03 getty.target.wants
183131 4 drwxr-xr-x  2 root root 4096 Jun  6  2017 halt.target.wants
188806 4 drwxr-xr-x  2 root root 4096 Feb  3  2017 local-fs.target.wants
182743 4 drwxr-xr-x  2 root root 4096 Jul 17 23:17 lwsmd.service.d
180714 4 drwxr-xr-x  2 root root 4096 Jul 17 19:58 multi-user.target.wants
180718 4 drwxr-xr-x  2 root root 4096 May 31 15:03 network-online.target.wants
183133 4 drwxr-xr-x  2 root root 4096 Jun  6  2017 poweroff.target.wants
183129 4 drwxr-xr-x  2 root root 4096 Jun  6  2017 reboot.target.wants
182530 4 -rw-r--r--  1 root root  268 Jun  7  2016 sendmail.service
183127 4 drwxr-xr-x  2 root root 4096 Jun  6  2017 shutdown.target.wants
182483 4 -rw-r--r--  1 root root  476 Mar  1 19:03 snmpd.service
180720 4 drwxr-xr-x  2 root root 4096 May 31 15:03 sockets.target.wants
180722 4 drwxr-xr-x  2 root root 4096 May 31 15:03 sysinit.target.wants
182042 0 lrwxrwxrwx  1 root root   39 Feb  3  2017 syslog.service -> /usr/lib/systemd/system/rsyslog.service
182708 4 -r-xr-xr-x  1 root root  470 Jun 13 09:50 vcha-hacheck.service
232102 4 drwxr-xr-x  2 root root 4096 Jul 17 23:17 vmafdd.service.d
320650 4 drwxr-xr-x  2 root root 4096 Jul 17 23:17 vmcad.service.d
183086 0 lrwxrwxrwx  1 root root    9 Jun  6  2017 vmcam.service -> /dev/null
182854 4 drwxr-xr-x  2 root root 4096 May  7 12:12 vmcam.service.d
320579 4 drwxr-xr-x  2 root root 4096 Jul 17 23:17 vmdird.service.d
320789 4 drwxr-xr-x  2 root root 4096 May  7 12:12 vmdnsd.service.d
182138 4 drwxr-xr-x  2 root root 4096 Feb  3  2017 vmtoolsd.service.requires
182905 0 lrwxrwxrwx  1 root root    9 Jun  6  2017 vmware-cis-license.service -> /dev/null
182904 0 lrwxrwxrwx  1 root root    9 Jun  6  2017 vmware-cm.service -> /dev/null
183082 0 lrwxrwxrwx  1 root root    9 Jun  6  2017 vmware-content-library.service -> /dev/null
183083 0 lrwxrwxrwx  1 root root    9 Jun  6  2017 vmware-eam.service -> /dev/null
183104 0 lrwxrwxrwx  1 root root    9 Jun  6  2017 vmware-imagebuilder.service -> /dev/null
183076 0 lrwxrwxrwx  1 root root    9 Jun  6  2017 vmware-mbcs.service -> /dev/null
183072 0 lrwxrwxrwx  1 root root    9 Jun  6  2017 vmware-netdump.service -> /dev/null
183105 0 lrwxrwxrwx  1 root root    9 Jun  6  2017 vmware-perfcharts.service -> /dev/null
182906 4 drwxr-xr-x  2 root root 4096 Jul 17 23:18 vmware-psc-client.service.d
183073 0 lrwxrwxrwx  1 root root    9 Jun  6  2017 vmware-pschealth.service -> /dev/null
183084 0 lrwxrwxrwx  1 root root    9 Jun  6  2017 vmware-rbd-watchdog.service -> /dev/null
182903 0 lrwxrwxrwx  1 root root    9 Jun  6  2017 vmware-rhttpproxy.service -> /dev/null
182916 0 lrwxrwxrwx  1 root root    9 Jun  6  2017 vmware-sca.service -> /dev/null
183087 0 lrwxrwxrwx  1 root root    9 Jun  6  2017 vmware-sps.service -> /dev/null
183074 0 lrwxrwxrwx  1 root root    9 Jun  6  2017 vmware-statsmonitor.service -> /dev/null
183118 4 drwxr-xr-x  2 root root 4096 Jul 17 23:18 vmware-stsd.service.d
183122 4 drwxr-xr-x  2 root root 4096 Jul 17 23:17 vmware-sts-idmd.service.d
183093 0 lrwxrwxrwx  1 root root    9 Jun  6  2017 vmware-updatemgr.service -> /dev/null
183060 0 lrwxrwxrwx  1 root root    9 Jun  6  2017 vmware-vapi.service -> /dev/null
182878 0 lrwxrwxrwx  1 root root    9 Jun  6  2017 vmware-vcha.service -> /dev/null
183067 0 lrwxrwxrwx  1 root root    9 Jun  6  2017 vmware-vmonapi.service -> /dev/null
183114 4 drwxr-xr-x  2 root root 4096 Jul 17 23:18 vmware-vmon.service.d
183075 0 lrwxrwxrwx  1 root root    9 Jun  6  2017 vmware-vpostgres.service -> /dev/null
183081 0 lrwxrwxrwx  1 root root    9 Jun  6  2017 vmware-vpxd.service -> /dev/null
182872 0 lrwxrwxrwx  1 root root    9 Jun  6  2017 vmware-vpxd-svcs.service -> /dev/null
183094 0 lrwxrwxrwx  1 root root    9 Jun  6  2017 vmware-vsan-health.service -> /dev/null
183103 0 lrwxrwxrwx  1 root root    9 Jun  6  2017 vmware-vsm.service -> /dev/null
183077 0 lrwxrwxrwx  1 root root    9 Jun  6  2017 vsphere-client.service -> /dev/null
183080 0 lrwxrwxrwx  1 root root    9 Jun  6  2017 vsphere-ui.service -> /dev/null

[KDA27]

I ran these two commands:

systemctl daemon-reload

service-control --start --all
Perform start operation. vmon_profile=ALL, svc_names=None, include_coreossvcs=True, include_leafossvcs=True
2018-07-18T18:28:11.440Z   Running command: ['/usr/bin/systemctl', 'is-enabled', u'lwsmd']
2018-07-18T18:28:11.444Z   Done running command
2018-07-18T18:28:11.448Z   Service lwsmd does not seem to be registered with vMon. If this is unexpected please make sure your service config is a valid json. Also check vmon logs for warnings.
2018-07-18T18:28:11.448Z   Running command: ['/sbin/service', u'lwsmd', 'status']
2018-07-18T18:28:11.489Z   Done running command
Successfully started service lwsmd
2018-07-18T18:28:11.494Z   Running command: ['/usr/bin/systemctl', 'is-enabled', u'vmafdd']
2018-07-18T18:28:11.500Z   Done running command
2018-07-18T18:28:11.509Z   Service vmafdd does not seem to be registered with vMon. If this is unexpected please make sure your service config is a valid json. Also check vmon logs for warnings.
2018-07-18T18:28:11.509Z   Running command: ['/sbin/service', u'vmafdd', 'status']
2018-07-18T18:28:11.537Z   Done running command
Successfully started service vmafdd
2018-07-18T18:28:11.541Z   Running command: ['/usr/bin/systemctl', 'is-enabled', u'vmdird']
2018-07-18T18:28:11.550Z   Done running command
2018-07-18T18:28:11.557Z   Service vmdird does not seem to be registered with vMon. If this is unexpected please make sure your service config is a valid json. Also check vmon logs for warnings.
2018-07-18T18:28:11.557Z   Running command: ['/sbin/service', u'vmdird', 'status']
2018-07-18T18:28:11.590Z   Done running command
Successfully started service vmdird
2018-07-18T18:28:11.594Z   Running command: ['/usr/bin/systemctl', 'is-enabled', u'vmcad']
2018-07-18T18:28:11.600Z   Done running command
2018-07-18T18:28:11.603Z   Service vmcad does not seem to be registered with vMon. If this is unexpected please make sure your service config is a valid json. Also check vmon logs for warnings.
2018-07-18T18:28:11.603Z   Running command: ['/sbin/service', u'vmcad', 'status']
2018-07-18T18:28:11.628Z   Done running command
Successfully started service vmcad
2018-07-18T18:28:11.632Z   Running command: ['/usr/bin/systemctl', 'is-enabled', u'vmware-sts-idmd']
2018-07-18T18:28:11.638Z   Done running command
2018-07-18T18:28:11.641Z   Service vmware-sts-idmd does not seem to be registered with vMon. If this is unexpected please make sure your service config is a valid json. Also check vmon logs for warnings.
2018-07-18T18:28:11.641Z   Running command: ['/sbin/service', u'vmware-sts-idmd', 'status']
2018-07-18T18:28:11.695Z   Done running command
Successfully started service vmware-sts-idmd
2018-07-18T18:28:11.700Z   Running command: ['/usr/bin/systemctl', 'is-enabled', u'vmware-stsd']
2018-07-18T18:28:11.713Z   Done running command
2018-07-18T18:28:11.717Z   Service vmware-stsd does not seem to be registered with vMon. If this is unexpected please make sure your service config is a valid json. Also check vmon logs for warnings.
2018-07-18T18:28:11.717Z   Running command: ['/sbin/service', u'vmware-stsd', 'status']
2018-07-18T18:28:11.743Z   Done running command
Successfully started service vmware-stsd
2018-07-18T18:28:11.747Z   Running command: ['/usr/bin/systemctl', 'is-enabled', u'vmdnsd']
2018-07-18T18:28:11.753Z   Done running command
2018-07-18T18:28:11.757Z   Service vmdnsd does not seem to be registered with vMon. If this is unexpected please make sure your service config is a valid json. Also check vmon logs for warnings.
2018-07-18T18:28:11.757Z   Running command: ['/sbin/service', u'vmdnsd', 'status']
2018-07-18T18:28:11.783Z   Done running command
Successfully started service vmdnsd
2018-07-18T18:28:11.787Z   Running command: ['/usr/bin/systemctl', 'is-enabled', u'vmware-psc-client']
2018-07-18T18:28:11.793Z   Done running command
2018-07-18T18:28:11.797Z   Service vmware-psc-client does not seem to be registered with vMon. If this is unexpected please make sure your service config is a valid json. Also check vmon logs for warnings.
2018-07-18T18:28:11.797Z   Running command: ['/sbin/service', u'vmware-psc-client', 'status']
2018-07-18T18:28:11.823Z   Done running command
Successfully started service vmware-psc-client
Service-control failed. Error Failed to start vmon services.vmon-cli RC=5, stderr=Failed to start statsmonitor services. Error: Operation not allowed in current service state

[daphnissov]

And guessing here that you didn't snapshot your vCSA before updating, correct?

[KDA27]

Hi daphnissov,

Yes, that is correct. I did read in a forum about reverting after a failed update.
I'm not sure if that would have helped in this case, but I should have made at least at few snapshots with different versions.

Kind Regards,
Kaspar Danielsen

[daphnissov]

It certainly would have helped in this case (you would simply revert to that snapshot), but seeing as how you didn't take a snapshot the point is moot.

[KDA27]

Hi daphnissov,

I'm aware of the possibility to revert back after a failed installation.. the strange thing is that I've never really had any issues up until now. The version I'm running could of cause be bugged.

The strange part is, I actually got the Appliance Management up an running by starting all services and running the two different commands in this thread.

I still struggle accessing the client interface - I do get the message that it's initializing while it's restarting and after that... nothing. (It does respond on IP)

This actually happened after I uploaded a new ISO for v.6.7. The Appliance Management does become un-responsive as well, after attempting to update the Appliance.. maaan!

Kind Regards,
Kaspar Danielsen

[KDA27]

I did look into this one -> VCSA 6.5 fails to start File System Check and Network Service - Nolabnoparty

I still cannot figure out how to open up the log's within /var/…. etc.

Kind Regards,
Kaspar Danielsen

[KDA27]

I found this one too - mine just stops at 60%(the step "Pre install Scripts") -> https://vdr.one/updating-vcsa-using-vami-from-6-0u1-to-6-0u2-stuck-at-70/

Kind Regards,
Kaspar Danielsen

[KDA27]

Hi,

I managed to get the vSphere Client and the Appliance Manager to run at the same time, by shutting down a few services and running some of them in order + starting all services (don't ask how I managed to get it running).

I'm launching a update on the Appliance Manager -> Goes straight to 60% -> Hangs for 1-2min -> Error: Cannot reach the appliance server. The server could be down or it's a network issue. You can try again after resolving this issue.

service-control --status
StartPending:
 vmware-content-library vsphere-client
Running:
 applmgmt lwsmd pschealth vmafdd vmcad vmdird vmdnsd vmonapi vmware-cis-license vmware-cm vmware-eam vmware-psc-client vmware-rhttpproxy vmware-sca vmware-sps vmware-statsmonitor vmware-sts-idmd vmware-stsd vmware-updatemgr vmware-vapi-endpoint vmware-vmon vmware-vpostgres vmware-vpxd vmware-vpxd-svcs vmware-vsan-health vmware-vsm vsphere-ui
Stopped:
 vmcam vmware-imagebuilder vmware-mbcs vmware-netdumper vmware-perfcharts vmware-rbd-watchdog vmware-vcha

How do I troubleshoot this?

Kind Regards,
Kaspar Danielsen

[sk84]

A broken or incomplete update is often very difficult to debug, as some files have probably already been replaced and some configuration changes may have been made. It's hard to find out what exactly no longer fits together.

It'll probably go faster if you recreate the vCenter from scratch. If you are not using a dvSwitch, you can simple disconnect the ESXi hosts with the running VMs from the existing vCenter and add them to the new vCenter.

Best regards,

Sebastian

[KDA27]

Hi sk84,

Thank you for your feedback in this thread.

I'm not able to deploy a new  vSphere Appliance and make the necessary configurations.

I've been looking more into, what's going wrong at right now it looks like this known issue -> vcsa 6.5 update - VAMI not working (SOLVED)

I can't figure out how to read the log.. hmm

/opt/vmware/etc/lighttpd/lighttpd.conf
bash: /opt/vmware/etc/lighttpd/lighttpd.conf: Permission denied

Kind Regards,

Kaspar Danielsen

[sk84]

Hi Kaspar,

Files with the extension ".conf" are configuration files and not log files. Log files have the ending ".log" and are usually located in the path "/var/logs/vmware/".

You can use linux commands to read the files (or edit them):

"less" is for reading, for example.

"vi" is for editing.

If you need help with these 2 tools, I recommend to read the linux man pages:

less Man Page - Bash - SS64.com

The vi Editor - A quick reference guide. | SS64.com

Best regards,

Sebastian

[KDA27]

Hi sk84,

It's much appreciated!

I'll give it a shot tomorrow - it's getting past midnight already.

Thank you.

Kind Regards,
Kaspar Danielsen

[daphnissov]

At this point, why aren't you opening a SR with VMware? If you can't deploy a new vCSA, as you said, and you didn't snapshot it, this would seem to be your next best option.

[KDA27]

Hi daphnissov,

I've created a case and have to pay at least 300$ for each incident (at the best).

I have a Dell consultant for this task and he's on vacation at the moment.

Kind Regards,
Kaspar Danielsen

[KDA27]

Hi sk84,

I did manage to open up the .conf file by using "vi".

When I finish the edit op the file and press ESC - how do I save the changes?

Kind Regards,
Kaspar Danielsen

[KDA27]

Hi,

I managed to change and save the log. I did disable IPv6 but no change so far.

I have access to the Client for administrating the VM's etc. but no access to the Appliance Management through the port number FQDM:5480.

Even if I run the few commands to start the services, it's not accessible.. only looks like I can have one running at the same time hmm.

Kind Regards,
Kaspar Danielsen

[gjbrown]

I am having the same issue.  I can tell you if you start the  vami-lighttp service vami will work again,  service vami-lighttp start   .. however I still can't update my PSC.  everytime it crashes out with the same error, lost network connection, even if I try update via iso.

Gb

[atokhi]

Greetings. Any luck? I'm in your shoes...

[AllBlack]

Having the same issue in my lab environment while upgrading to vcsa 6.5 U2C via VAMI
No issue with PSC upgrade but vCSA failed with symptoms described in other replies.

I also find that vCenter indicates it is the latest version but VAMI does not.

Did not take a snapshot as it was my lab.

VMware telling me to rebuild. I do have vDS so this would be nightmare and I do not find this a good enough answer.
Imagine this happening in my huge production environment. Would not want to rebuild! Will definitely take a snapshot as I always do except this time 0-olll

I am going to wait until a new version of vCenter comes out (i.e. 6.5 U3 ) and try again. Will be interesting to see if I get same issue in production.

PS: Also tried updating from ISO and installer. Same issue.