I'll preface this with the fact that my VMWare knowledge is very limited. I know how to use vSphere and connect to a host to make simple changes. The issue is, for one of our clients we have a vCenter set up and we have Veeam backing it up. Backups started to fail with authentication errors, so I tried connecting to vCenter via vSphere and received login errors as well. I rebooted the vCenter, and now I receive 503 errors if I try to login via vSphere or the web.
After searching online, I saw documentation saying this could be caused by the storage being full. I tried allocating more storage from the host, but the host is being managed by the vCenter I guess so it won't let me. I was able to access it by it's IP:5480 but it logged me out immediately every time. Connected to it via shell by editing the boot sequence and accessing shell, then resetting the root pw and setting it to never expire. I could then login and it alerted me storage was low there too. I enabled SSH and connected to it and checked it using df -h. It was using 100% storage, and it appeared most was coming from the storage/logs. I cleared it out using info from KBs and now I'm at 53% usage. I also ran an update from the 5480 page and now it says there are no more, the current version is 6.0.0.30500.
I saw a forum post mentioning that someone fixed it by upgrading it to 6.5. Well I downloaded the installer and attempted this, but it won't connect to it to upgrade it.
Also just tried this but it didn't work:
Unable to Connect to VCSA - Unknown Connection Error (503 Server Unavailable)
I still have the same errors, but I can access it via SSH. I confirmed the IP and the loopback are listed in the hosts file. I've restarted all the services I could find.. I've attached some screenshots and I also pulled the vm-support package but it's over 12 GB and I'm not sure how to get anyone access to it as this would take two weeks to upload on their network. I also have no idea how to read these so any insight on what I need to check here would help too! I feel like it might be a simple fix, but I've read through so many articles and forum posts and I have no idea what I'm missing here.
Thank you ahead of time!
- Vince
I believe the DB was located on the same VM, but I wouldn't be able to tell you for sure. I appreciate the help from everyone, but due to the need of having to get backups running again, I just deployed a new VCSA and configured it. This issue can me marked as resolved, thank you.
Hello!
Can you show the following command output from CLI
service-control --status all
Also you may hit this issue vCenter 6.5 - vcenter appliance stops working out of the blue, AGAIN!!
Hi Finikiez,
Regarding that post, if I try to browse to that folder they posted about, it says it doesn't exist:
bbfc-vcenter:/etc # cd /etc/systemd/system
bash: cd: /etc/systemd/system: No such file or directory
bbfc-vcenter:/etc #
It appears systemd doesn't exist, and if I try to run systemctl, it says "Command not found".
When I run that command you posted from SSH, it displays: "Unknown services: all". If I run service-control --status it displays:
"service-control --status
INFO:root:Service: vmware-rhttpproxy, Action: status
Service: vmware-rhttpproxy, Action: status
2018-06-14T18:17:47.244Z Running command: ['/sbin/service', u'vmware-rhttpprox y', 'status']
2018-06-14T18:17:47.279Z Done running command
INFO:root:Service: vmware-cis-license, Action: status
Service: vmware-cis-license, Action: status
2018-06-14T18:17:47.280Z Running command: ['/sbin/service', u'vmware-cis-licen se', 'status']
2018-06-14T18:17:47.418Z Done running command
INFO:root:Service: vmware-psc-client, Action: status
Service: vmware-psc-client, Action: status
2018-06-14T18:17:47.418Z Running command: ['/sbin/service', u'vmware-psc-clien t', 'status']
2018-06-14T18:17:47.449Z Done running command
INFO:root:Service: vmware-sca, Action: status
Service: vmware-sca, Action: status
2018-06-14T18:17:47.450Z Running command: ['/sbin/service', u'vmware-sca', 'st atus']
2018-06-14T18:17:47.540Z Done running command
INFO:root:Service: vmware-netdumper, Action: status
Service: vmware-netdumper, Action: status
2018-06-14T18:17:47.540Z Running command: ['/sbin/service', u'vmware-netdumper ', 'status']
2018-06-14T18:17:47.585Z Done running command
INFO:root:Service: vmware-syslog, Action: status
Service: vmware-syslog, Action: status
2018-06-14T18:17:47.585Z Running command: ['/sbin/service', u'vmware-syslog', 'status']
2018-06-14T18:17:47.637Z Done running command
INFO:root:Service: vmware-syslog-health, Action: status
Service: vmware-syslog-health, Action: status
2018-06-14T18:17:47.638Z Running command: ['/sbin/service', u'vmware-syslog-he alth', 'status']
2018-06-14T18:17:47.750Z Done running command
INFO:root:Service: vmware-vapi-endpoint, Action: status
Service: vmware-vapi-endpoint, Action: status
2018-06-14T18:17:47.750Z Running command: ['/sbin/service', u'vmware-vapi-endp oint', 'status']
2018-06-14T18:17:47.798Z Done running command
INFO:root:Service: vmware-vpostgres, Action: status
Service: vmware-vpostgres, Action: status
2018-06-14T18:17:47.799Z Running command: ['/sbin/service', u'vmware-vpostgres ', 'status']
2018-06-14T18:17:47.924Z Done running command
INFO:root:Service: vmware-invsvc, Action: status
Service: vmware-invsvc, Action: status
2018-06-14T18:17:47.925Z Running command: ['/sbin/service', u'vmware-invsvc', 'status']
2018-06-14T18:17:48.025Z Done running command
INFO:root:Service: vmware-mbcs, Action: status
Service: vmware-mbcs, Action: status
2018-06-14T18:17:48.025Z Running command: ['/sbin/service', u'vmware-mbcs', 's tatus']
2018-06-14T18:17:48.123Z Done running command
INFO:root:Service: vmware-vpxd, Action: status
Service: vmware-vpxd, Action: status
2018-06-14T18:17:48.124Z Running command: ['/sbin/service', u'vmware-vpxd', 's tatus']
2018-06-14T18:17:48.509Z Done running command
INFO:root:Service: vmware-eam, Action: status
Service: vmware-eam, Action: status
2018-06-14T18:17:48.509Z Running command: ['/sbin/service', u'vmware-eam', 'st atus']
2018-06-14T18:17:48.575Z Done running command
INFO:root:Service: vmware-rbd-watchdog, Action: status
Service: vmware-rbd-watchdog, Action: status
2018-06-14T18:17:48.575Z Running command: ['/sbin/service', u'vmware-rbd-watch dog', 'status']
2018-06-14T18:17:48.617Z Done running command
INFO:root:Service: vmware-sps, Action: status
Service: vmware-sps, Action: status
2018-06-14T18:17:48.617Z Running command: ['/sbin/service', u'vmware-sps', 'st atus']
2018-06-14T18:17:48.747Z Done running command
INFO:root:Service: vmware-vdcs, Action: status
Service: vmware-vdcs, Action: status
2018-06-14T18:17:48.748Z Running command: ['/sbin/service', u'vmware-vdcs', 's tatus']
2018-06-14T18:17:48.838Z Done running command
INFO:root:Service: vmware-vpx-workflow, Action: status
Service: vmware-vpx-workflow, Action: status
2018-06-14T18:17:48.838Z Running command: ['/sbin/service', u'vmware-vpx-workf low', 'status']
2018-06-14T18:17:48.894Z Done running command
INFO:root:Service: vmware-vsan-health, Action: status
Service: vmware-vsan-health, Action: status
2018-06-14T18:17:48.894Z Running command: ['/sbin/service', u'vmware-vsan-heal th', 'status']
2018-06-14T18:17:48.902Z Done running command
INFO:root:Service: vmware-vsm, Action: status
Service: vmware-vsm, Action: status
2018-06-14T18:17:48.903Z Running command: ['/sbin/service', u'vmware-vsm', 'st atus']
2018-06-14T18:17:48.953Z Done running command
INFO:root:Service: vsphere-client, Action: status
Service: vsphere-client, Action: status
2018-06-14T18:17:48.953Z Running command: ['/sbin/service', u'vsphere-client', 'status']
2018-06-14T18:17:49.045Z Done running command
INFO:root:Service: vmware-perfcharts, Action: status
Service: vmware-perfcharts, Action: status
2018-06-14T18:17:49.045Z Running command: ['/sbin/service', u'vmware-perfchart s', 'status']
2018-06-14T18:17:49.106Z Done running command
INFO:root:Service: vmware-vws, Action: status
Service: vmware-vws, Action: status
2018-06-14T18:17:49.107Z Running command: ['/sbin/service', u'vmware-vws', 'st atus']
2018-06-14T18:17:49.168Z Done running command
INFO:root:Running:
vmware-cis-license (VMware License Service) vmware-perfcharts (VMware Performan ce Charts) vmware-psc-client (VMware Platform Services Controller Client) vmware -rhttpproxy (VMware HTTP Reverse Proxy) vmware-sps (VMware vSphere Profile-Drive n Storage Service) vmware-syslog (VMware Common Logging Service) vmware-syslog-h ealth (VMware Syslog Health Service) vmware-vpostgres (VMware Postgres) vmware-v san-health (VMware VSAN Health Service) vmware-vws (VMware System and Hardware H ealth Manager)
Running:
vmware-cis-license (VMware License Service) vmware-perfcharts (VMware Performan ce Charts) vmware-psc-client (VMware Platform Services Controller Client) vmware -rhttpproxy (VMware HTTP Reverse Proxy) vmware-sps (VMware vSphere Profile-Drive n Storage Service) vmware-syslog (VMware Common Logging Service) vmware-syslog-h ealth (VMware Syslog Health Service) vmware-vpostgres (VMware Postgres) vmware-v san-health (VMware VSAN Health Service) vmware-vws (VMware System and Hardware H ealth Manager)
INFO:root:Stopped:
vmware-eam (VMware ESX Agent Manager) vmware-invsvc (VMware Inventory Service) vmware-mbcs (VMware Message Bus Configuration Service) vmware-netdumper (VMware vSphere ESXi Dump Collector) vmware-rbd-watchdog (VMware vSphere Auto Deploy Wai ter) vmware-sca (VMware Service Control Agent) vmware-vapi-endpoint (VMware vAPI Endpoint) vmware-vdcs (VMware Content Library Service) vmware-vpx-workflow (VMw are vCenter Workflow Manager) vmware-vpxd (VMware vCenter Server) vmware-vsm (VM ware vService Manager) vsphere-client ()
Stopped:
vmware-eam (VMware ESX Agent Manager) vmware-invsvc (VMware Inventory Service) vmware-mbcs (VMware Message Bus Configuration Service) vmware-netdumper (VMware vSphere ESXi Dump Collector) vmware-rbd-watchdog (VMware vSphere Auto Deploy Wai ter) vmware-sca (VMware Service Control Agent) vmware-vapi-endpoint (VMware vAPI Endpoint) vmware-vdcs (VMware Content Library Service) vmware-vpx-workflow (VMw are vCenter Workflow Manager) vmware-vpxd (VMware vCenter Server) vmware-vsm (VM ware vService Manager) vsphere-client ()"
I missed that you use vcsa 6.0
You can't connect to VC because main services are stopped vmware-invsvc, vmware-vapi-endpoint, vmware-vpxd and vsphere-client
Try to start them manually and see what happened.
If they can't be started manually then you need to check the logs in /var/log/vmware directory
I can start the vmware-vapi-endpoint and vsphere-client. I tried the vmware-vpxd then it got stuck for several minutes and then I cancelled the start, but when I check the status it says running. When I try to start vmware-invsvs I get the errors:
bbfc-vcenter:~ # service vmware-invsvc start
Starting VMware Inventory Service...
Waiting for VMware Inventory Service..............
WARNING: VMware Inventory Service may have failed to start.
bbfc-vcenter:~ # service vmware-invsvc status
VMware Inventory Service is not running.
I'm not sure which logs to check in that directory.
Sorry, notifications are broken and I can't reply in time.
Can you go to /var/log/vmware/invsvc and check latest ds.log
There could inventory sevice database corruption. However it's better to check the logs.
I don't have VC 6.0 installed at the moment. Can you show a schreenshot? or can you check if there is a Logs directory or something on that path?
ok, try to start inventory service again and then attach inv-svc.log
Curious to know what database your using for the vCSA. Is it the one on the vCSA or are you connecting to an external db?? I have an issue with my vCenter Server (some time ago) when the MS SQL server drive was filled up by the transaction log files. Once I resolved that issue, vCenter ran just fine. But before then, it would exhibit pretty much the same behavior you're facing.
I believe the DB was located on the same VM, but I wouldn't be able to tell you for sure. I appreciate the help from everyone, but due to the need of having to get backups running again, I just deployed a new VCSA and configured it. This issue can me marked as resolved, thank you.