Hi, this issue started 2 days ago when I attempted to update from to "VC-6.5.0U2b-Appliance-FP" (6.5.0.21000 Build Number 8815520) in the vSphere Appliance Management. It stopped at 60% and prompted with an error code about "not being able to contact the server...". I then restarted the Appliance and from there it went down hill.
I figured out if I ran the command "Service vami-lighttp start" and it came up again, but after some time I couldn't authenticate etc.. seems as some services aren't running.
I did find this discussion in a different thread and noticed it was the exact same issue I have -> vCenter 6.5 - vcenter appliance stops working out of the blue, AGAIN!!
I will post the same output and hopefully figure this out. (I did notice those 4 commands, but haven't used those yet since I don't quite understand what they do)
service-control --status
Running:
applmgmt lwsmd pschealth vmafdd vmcad vmdird vmdnsd vmonapi vmware-cis-license vmware-cm vmware-content-library vmware-eam vmware-perfcharts vmware-psc-client vmware-rhttpproxy vmware-sca vmware-sps vmware-sts-idmd vmware-stsd vmware-updatemgr vmware-vapi-endpoint vmware-vmon vmware-vpostgres vmware-vpxd vmware-vpxd-svcs vmware-vsan-health vmware-vsm vsphere-client vsphere-ui
Stopped:
vmcam vmware-imagebuilder vmware-mbcs vmware-netdumper vmware-rbd-watchdog vmware-statsmonitor vmware-vcha
service-control --start vmware-vpxd-svcs
Perform start operation. vmon_profile=None, svc_names=['vmware-vpxd-svcs'], include_coreossvcs=False, include_leafossvcs=False
2018-07-18T17:45:49.534Z Service vpxd-svcs state STARTED
Successfully started service vpxd-svcs
systemctl list-unit-files | grep vmware
vmware-bigsister.service static
vmware-cis-license.service masked
vmware-cm.service masked
vmware-content-library.service masked
vmware-eam.service masked
vmware-firewall.service enabled
vmware-imagebuilder.service masked
vmware-mbcs.service masked
vmware-netdump.service masked
vmware-perfcharts.service masked
vmware-pschealth.service masked
vmware-rbd-watchdog.service masked
vmware-rhttpproxy.service masked
vmware-sca.service masked
vmware-sps.service masked
vmware-statsmonitor.service masked
vmware-updatemgr.service masked
vmware-vapi.service masked
vmware-vcha.service masked
vmware-vmon.service enabled
vmware-vmonapi.service masked
vmware-vpostgres.service masked
vmware-vpxd-svcs.service masked
vmware-vpxd.service masked
vmware-vsan-health.service masked
vmware-vsm.service masked
vmware-bigsister.timer disabled
After running the "service-control --start vmware-vpxd-svcs" the Appliance Manager came up again.
systemctl list-unit-files | grep masked
applmgmt.service masked
vmcam.service masked
vmware-cis-license.service masked
vmware-cm.service masked
vmware-content-library.service masked
vmware-eam.service masked
vmware-imagebuilder.service masked
vmware-mbcs.service masked
vmware-netdump.service masked
vmware-perfcharts.service masked
vmware-pschealth.service masked
vmware-rbd-watchdog.service masked
vmware-rhttpproxy.service masked
vmware-sca.service masked
vmware-sps.service masked
vmware-statsmonitor.service masked
vmware-updatemgr.service masked
vmware-vapi.service masked
vmware-vcha.service masked
vmware-vmonapi.service masked
vmware-vpostgres.service masked
vmware-vpxd-svcs.service masked
vmware-vpxd.service masked
vmware-vsan-health.service masked
vmware-vsm.service masked
vsphere-client.service masked
vsphere-ui.service masked
ctrl-alt-del.target masked
/etc/systemd/system ]# ls -lisa
total 108
180710 4 drwxr-xr-x 24 root root 4096 Jul 17 17:01 .
180226 4 drwxr-xr-x 7 root root 4096 May 31 15:04 ..
183069 0 lrwxrwxrwx 1 root root 9 Jun 6 2017 applmgmt.service -> /dev/null
183110 4 drwxr-xr-x 2 root root 4096 Jun 6 2017 applmgmt.service.d
182577 0 lrwxrwxrwx 1 root root 40 Feb 3 2017 default.target -> /usr/lib/systemd/system/runlevel3.target
180712 4 drwxr-xr-x 2 root root 4096 May 31 15:03 getty.target.wants
183131 4 drwxr-xr-x 2 root root 4096 Jun 6 2017 halt.target.wants
188806 4 drwxr-xr-x 2 root root 4096 Feb 3 2017 local-fs.target.wants
182743 4 drwxr-xr-x 2 root root 4096 Jul 17 23:17 lwsmd.service.d
180714 4 drwxr-xr-x 2 root root 4096 Jul 17 19:58 multi-user.target.wants
180718 4 drwxr-xr-x 2 root root 4096 May 31 15:03 network-online.target.wants
183133 4 drwxr-xr-x 2 root root 4096 Jun 6 2017 poweroff.target.wants
183129 4 drwxr-xr-x 2 root root 4096 Jun 6 2017 reboot.target.wants
182530 4 -rw-r--r-- 1 root root 268 Jun 7 2016 sendmail.service
183127 4 drwxr-xr-x 2 root root 4096 Jun 6 2017 shutdown.target.wants
182483 4 -rw-r--r-- 1 root root 476 Mar 1 19:03 snmpd.service
180720 4 drwxr-xr-x 2 root root 4096 May 31 15:03 sockets.target.wants
180722 4 drwxr-xr-x 2 root root 4096 May 31 15:03 sysinit.target.wants
182042 0 lrwxrwxrwx 1 root root 39 Feb 3 2017 syslog.service -> /usr/lib/systemd/system/rsyslog.service
182708 4 -r-xr-xr-x 1 root root 470 Jun 13 09:50 vcha-hacheck.service
232102 4 drwxr-xr-x 2 root root 4096 Jul 17 23:17 vmafdd.service.d
320650 4 drwxr-xr-x 2 root root 4096 Jul 17 23:17 vmcad.service.d
183086 0 lrwxrwxrwx 1 root root 9 Jun 6 2017 vmcam.service -> /dev/null
182854 4 drwxr-xr-x 2 root root 4096 May 7 12:12 vmcam.service.d
320579 4 drwxr-xr-x 2 root root 4096 Jul 17 23:17 vmdird.service.d
320789 4 drwxr-xr-x 2 root root 4096 May 7 12:12 vmdnsd.service.d
182138 4 drwxr-xr-x 2 root root 4096 Feb 3 2017 vmtoolsd.service.requires
182905 0 lrwxrwxrwx 1 root root 9 Jun 6 2017 vmware-cis-license.service -> /dev/null
182904 0 lrwxrwxrwx 1 root root 9 Jun 6 2017 vmware-cm.service -> /dev/null
183082 0 lrwxrwxrwx 1 root root 9 Jun 6 2017 vmware-content-library.service -> /dev/null
183083 0 lrwxrwxrwx 1 root root 9 Jun 6 2017 vmware-eam.service -> /dev/null
183104 0 lrwxrwxrwx 1 root root 9 Jun 6 2017 vmware-imagebuilder.service -> /dev/null
183076 0 lrwxrwxrwx 1 root root 9 Jun 6 2017 vmware-mbcs.service -> /dev/null
183072 0 lrwxrwxrwx 1 root root 9 Jun 6 2017 vmware-netdump.service -> /dev/null
183105 0 lrwxrwxrwx 1 root root 9 Jun 6 2017 vmware-perfcharts.service -> /dev/null
182906 4 drwxr-xr-x 2 root root 4096 Jul 17 23:18 vmware-psc-client.service.d
183073 0 lrwxrwxrwx 1 root root 9 Jun 6 2017 vmware-pschealth.service -> /dev/null
183084 0 lrwxrwxrwx 1 root root 9 Jun 6 2017 vmware-rbd-watchdog.service -> /dev/null
182903 0 lrwxrwxrwx 1 root root 9 Jun 6 2017 vmware-rhttpproxy.service -> /dev/null
182916 0 lrwxrwxrwx 1 root root 9 Jun 6 2017 vmware-sca.service -> /dev/null
183087 0 lrwxrwxrwx 1 root root 9 Jun 6 2017 vmware-sps.service -> /dev/null
183074 0 lrwxrwxrwx 1 root root 9 Jun 6 2017 vmware-statsmonitor.service -> /dev/null
183118 4 drwxr-xr-x 2 root root 4096 Jul 17 23:18 vmware-stsd.service.d
183122 4 drwxr-xr-x 2 root root 4096 Jul 17 23:17 vmware-sts-idmd.service.d
183093 0 lrwxrwxrwx 1 root root 9 Jun 6 2017 vmware-updatemgr.service -> /dev/null
183060 0 lrwxrwxrwx 1 root root 9 Jun 6 2017 vmware-vapi.service -> /dev/null
182878 0 lrwxrwxrwx 1 root root 9 Jun 6 2017 vmware-vcha.service -> /dev/null
183067 0 lrwxrwxrwx 1 root root 9 Jun 6 2017 vmware-vmonapi.service -> /dev/null
183114 4 drwxr-xr-x 2 root root 4096 Jul 17 23:18 vmware-vmon.service.d
183075 0 lrwxrwxrwx 1 root root 9 Jun 6 2017 vmware-vpostgres.service -> /dev/null
183081 0 lrwxrwxrwx 1 root root 9 Jun 6 2017 vmware-vpxd.service -> /dev/null
182872 0 lrwxrwxrwx 1 root root 9 Jun 6 2017 vmware-vpxd-svcs.service -> /dev/null
183094 0 lrwxrwxrwx 1 root root 9 Jun 6 2017 vmware-vsan-health.service -> /dev/null
183103 0 lrwxrwxrwx 1 root root 9 Jun 6 2017 vmware-vsm.service -> /dev/null
183077 0 lrwxrwxrwx 1 root root 9 Jun 6 2017 vsphere-client.service -> /dev/null
183080 0 lrwxrwxrwx 1 root root 9 Jun 6 2017 vsphere-ui.service -> /dev/null
I ran these two commands:
systemctl daemon-reload
service-control --start --all
Perform start operation. vmon_profile=ALL, svc_names=None, include_coreossvcs=True, include_leafossvcs=True
2018-07-18T18:28:11.440Z Running command: ['/usr/bin/systemctl', 'is-enabled', u'lwsmd']
2018-07-18T18:28:11.444Z Done running command
2018-07-18T18:28:11.448Z Service lwsmd does not seem to be registered with vMon. If this is unexpected please make sure your service config is a valid json. Also check vmon logs for warnings.
2018-07-18T18:28:11.448Z Running command: ['/sbin/service', u'lwsmd', 'status']
2018-07-18T18:28:11.489Z Done running command
Successfully started service lwsmd
2018-07-18T18:28:11.494Z Running command: ['/usr/bin/systemctl', 'is-enabled', u'vmafdd']
2018-07-18T18:28:11.500Z Done running command
2018-07-18T18:28:11.509Z Service vmafdd does not seem to be registered with vMon. If this is unexpected please make sure your service config is a valid json. Also check vmon logs for warnings.
2018-07-18T18:28:11.509Z Running command: ['/sbin/service', u'vmafdd', 'status']
2018-07-18T18:28:11.537Z Done running command
Successfully started service vmafdd
2018-07-18T18:28:11.541Z Running command: ['/usr/bin/systemctl', 'is-enabled', u'vmdird']
2018-07-18T18:28:11.550Z Done running command
2018-07-18T18:28:11.557Z Service vmdird does not seem to be registered with vMon. If this is unexpected please make sure your service config is a valid json. Also check vmon logs for warnings.
2018-07-18T18:28:11.557Z Running command: ['/sbin/service', u'vmdird', 'status']
2018-07-18T18:28:11.590Z Done running command
Successfully started service vmdird
2018-07-18T18:28:11.594Z Running command: ['/usr/bin/systemctl', 'is-enabled', u'vmcad']
2018-07-18T18:28:11.600Z Done running command
2018-07-18T18:28:11.603Z Service vmcad does not seem to be registered with vMon. If this is unexpected please make sure your service config is a valid json. Also check vmon logs for warnings.
2018-07-18T18:28:11.603Z Running command: ['/sbin/service', u'vmcad', 'status']
2018-07-18T18:28:11.628Z Done running command
Successfully started service vmcad
2018-07-18T18:28:11.632Z Running command: ['/usr/bin/systemctl', 'is-enabled', u'vmware-sts-idmd']
2018-07-18T18:28:11.638Z Done running command
2018-07-18T18:28:11.641Z Service vmware-sts-idmd does not seem to be registered with vMon. If this is unexpected please make sure your service config is a valid json. Also check vmon logs for warnings.
2018-07-18T18:28:11.641Z Running command: ['/sbin/service', u'vmware-sts-idmd', 'status']
2018-07-18T18:28:11.695Z Done running command
Successfully started service vmware-sts-idmd
2018-07-18T18:28:11.700Z Running command: ['/usr/bin/systemctl', 'is-enabled', u'vmware-stsd']
2018-07-18T18:28:11.713Z Done running command
2018-07-18T18:28:11.717Z Service vmware-stsd does not seem to be registered with vMon. If this is unexpected please make sure your service config is a valid json. Also check vmon logs for warnings.
2018-07-18T18:28:11.717Z Running command: ['/sbin/service', u'vmware-stsd', 'status']
2018-07-18T18:28:11.743Z Done running command
Successfully started service vmware-stsd
2018-07-18T18:28:11.747Z Running command: ['/usr/bin/systemctl', 'is-enabled', u'vmdnsd']
2018-07-18T18:28:11.753Z Done running command
2018-07-18T18:28:11.757Z Service vmdnsd does not seem to be registered with vMon. If this is unexpected please make sure your service config is a valid json. Also check vmon logs for warnings.
2018-07-18T18:28:11.757Z Running command: ['/sbin/service', u'vmdnsd', 'status']
2018-07-18T18:28:11.783Z Done running command
Successfully started service vmdnsd
2018-07-18T18:28:11.787Z Running command: ['/usr/bin/systemctl', 'is-enabled', u'vmware-psc-client']
2018-07-18T18:28:11.793Z Done running command
2018-07-18T18:28:11.797Z Service vmware-psc-client does not seem to be registered with vMon. If this is unexpected please make sure your service config is a valid json. Also check vmon logs for warnings.
2018-07-18T18:28:11.797Z Running command: ['/sbin/service', u'vmware-psc-client', 'status']
2018-07-18T18:28:11.823Z Done running command
Successfully started service vmware-psc-client
Service-control failed. Error Failed to start vmon services.vmon-cli RC=5, stderr=Failed to start statsmonitor services. Error: Operation not allowed in current service state
And guessing here that you didn't snapshot your vCSA before updating, correct?
Hi daphnissov,
Yes, that is correct. I did read in a forum about reverting after a failed update.
I'm not sure if that would have helped in this case, but I should have made at least at few snapshots with different versions.
Kind Regards,
Kaspar Danielsen
It certainly would have helped in this case (you would simply revert to that snapshot), but seeing as how you didn't take a snapshot the point is moot.
Hi daphnissov,
I'm aware of the possibility to revert back after a failed installation.. the strange thing is that I've never really had any issues up until now. The version I'm running could of cause be bugged.
The strange part is, I actually got the Appliance Management up an running by starting all services and running the two different commands in this thread.
I still struggle accessing the client interface - I do get the message that it's initializing while it's restarting and after that... nothing. (It does respond on IP)
This actually happened after I uploaded a new ISO for v.6.7. The Appliance Management does become un-responsive as well, after attempting to update the Appliance.. maaan!
Kind Regards,
Kaspar Danielsen
I did look into this one -> VCSA 6.5 fails to start File System Check and Network Service - Nolabnoparty
I still cannot figure out how to open up the log's within /var/…. etc.
Kind Regards,
Kaspar Danielsen
I found this one too - mine just stops at 60%(the step "Pre install Scripts") -> https://vdr.one/updating-vcsa-using-vami-from-6-0u1-to-6-0u2-stuck-at-70/
Kind Regards,
Kaspar Danielsen
Hi,
I managed to get the vSphere Client and the Appliance Manager to run at the same time, by shutting down a few services and running some of them in order + starting all services (don't ask how I managed to get it running).
I'm launching a update on the Appliance Manager -> Goes straight to 60% -> Hangs for 1-2min -> Error: Cannot reach the appliance server. The server could be down or it's a network issue. You can try again after resolving this issue.
service-control --status
StartPending:
vmware-content-library vsphere-client
Running:
applmgmt lwsmd pschealth vmafdd vmcad vmdird vmdnsd vmonapi vmware-cis-license vmware-cm vmware-eam vmware-psc-client vmware-rhttpproxy vmware-sca vmware-sps vmware-statsmonitor vmware-sts-idmd vmware-stsd vmware-updatemgr vmware-vapi-endpoint vmware-vmon vmware-vpostgres vmware-vpxd vmware-vpxd-svcs vmware-vsan-health vmware-vsm vsphere-ui
Stopped:
vmcam vmware-imagebuilder vmware-mbcs vmware-netdumper vmware-perfcharts vmware-rbd-watchdog vmware-vcha
How do I troubleshoot this?
Kind Regards,
Kaspar Danielsen
A broken or incomplete update is often very difficult to debug, as some files have probably already been replaced and some configuration changes may have been made. It's hard to find out what exactly no longer fits together.
It'll probably go faster if you recreate the vCenter from scratch. If you are not using a dvSwitch, you can simple disconnect the ESXi hosts with the running VMs from the existing vCenter and add them to the new vCenter.
Best regards,
Sebastian
Hi sk84,
Thank you for your feedback in this thread.
I'm not able to deploy a new vSphere Appliance and make the necessary configurations.
I've been looking more into, what's going wrong at right now it looks like this known issue -> vcsa 6.5 update - VAMI not working (SOLVED)
I can't figure out how to read the log.. hmm
/opt/vmware/etc/lighttpd/lighttpd.conf
bash: /opt/vmware/etc/lighttpd/lighttpd.conf: Permission denied
Kind Regards,
Kaspar Danielsen
Hi Kaspar,
Files with the extension ".conf" are configuration files and not log files. Log files have the ending ".log" and are usually located in the path "/var/logs/vmware/".
You can use linux commands to read the files (or edit them):
"less" is for reading, for example.
"vi" is for editing.
If you need help with these 2 tools, I recommend to read the linux man pages:
less Man Page - Bash - SS64.com
The vi Editor - A quick reference guide. | SS64.com
Best regards,
Sebastian
Hi sk84,
It's much appreciated!
I'll give it a shot tomorrow - it's getting past midnight already.
Thank you.
Kind Regards,
Kaspar Danielsen
At this point, why aren't you opening a SR with VMware? If you can't deploy a new vCSA, as you said, and you didn't snapshot it, this would seem to be your next best option.
Hi daphnissov,
I've created a case and have to pay at least 300$ for each incident (at the best).
I have a Dell consultant for this task and he's on vacation at the moment.
Kind Regards,
Kaspar Danielsen
Hi sk84,
I did manage to open up the .conf file by using "vi".
When I finish the edit op the file and press ESC - how do I save the changes?
Kind Regards,
Kaspar Danielsen
Hi,
I managed to change and save the log. I did disable IPv6 but no change so far.
I have access to the Client for administrating the VM's etc. but no access to the Appliance Management through the port number FQDM:5480.
Even if I run the few commands to start the services, it's not accessible.. only looks like I can have one running at the same time hmm.
Kind Regards,
Kaspar Danielsen
I am having the same issue. I can tell you if you start the vami-lighttp service vami will work again, service vami-lighttp start .. however I still can't update my PSC. everytime it crashes out with the same error, lost network connection, even if I try update via iso.
Gb
Greetings. Any luck? I'm in your shoes...
Having the same issue in my lab environment while upgrading to vcsa 6.5 U2C via VAMI
No issue with PSC upgrade but vCSA failed with symptoms described in other replies.
I also find that vCenter indicates it is the latest version but VAMI does not.
Did not take a snapshot as it was my lab.
VMware telling me to rebuild. I do have vDS so this would be nightmare and I do not find this a good enough answer.
Imagine this happening in my huge production environment. Would not want to rebuild! Will definitely take a snapshot as I always do except this time 0-olll
I am going to wait until a new version of vCenter comes out (i.e. 6.5 U3 ) and try again. Will be interesting to see if I get same issue in production.
PS: Also tried updating from ISO and installer. Same issue.