vCenter 6.0 Issues

[Viceman256]

I'll preface this with the fact that my VMWare knowledge is very limited. I know how to use vSphere and connect to a host to make simple changes. The issue is, for one of our clients we have a vCenter set up and we have Veeam backing it up. Backups started to fail with authentication errors, so I tried connecting to vCenter via vSphere and received login errors as well. I rebooted the vCenter, and now I receive 503 errors if I try to login via vSphere or the web.

After searching online, I saw documentation saying this could be caused by the storage being full. I tried allocating more storage from the host, but the host is being managed by the vCenter I guess so it won't let me. I was able to access it by it's IP:5480 but it logged me out immediately every time. Connected to it via shell by editing the boot sequence and accessing shell, then resetting the root pw and setting it to never expire. I could then login and it alerted me storage was low there too. I enabled SSH and connected to it and checked it using df -h. It was using 100% storage, and it appeared most was coming from the storage/logs. I cleared it out using info from KBs and now I'm at 53% usage. I also ran an update from the 5480 page and now it says there are no more, the current version is 6.0.0.30500.

I saw a forum post mentioning that someone fixed it by upgrading it to 6.5. Well I downloaded the installer and attempted this, but it won't connect to it to upgrade it.

Also just tried this but it didn't work:

Unable to Connect to VCSA - Unknown Connection Error (503 Server Unavailable)

I still have the same errors, but I can access it via SSH. I confirmed the IP and the loopback are listed in the hosts file. I've restarted all the services I could find.. I've attached some screenshots and I also pulled the vm-support package but it's over 12 GB and I'm not sure how to get anyone access to it as this would take two weeks to upload on their network. I also have no idea how to read these so any insight on what I need to check here would help too! I feel like it might be a simple fix, but I've read through so many articles and forum posts and I have no idea what I'm missing here.

Thank you ahead of time!

- Vince

[Finikiez]

Hello!

Can you show the following command output from CLI

service-control --status all

Also you may hit this issue vCenter 6.5 - vcenter appliance stops working out of the blue, AGAIN!!

[Viceman256]

Hi Finikiez,

Regarding that post, if I try to browse to that folder they posted about, it says it doesn't exist:

bbfc-vcenter:/etc # cd /etc/systemd/system

bash: cd: /etc/systemd/system: No such file or directory

bbfc-vcenter:/etc #

It appears systemd doesn't exist, and if I try to run systemctl, it says "Command not found".

When I run that command you posted from SSH, it displays: "Unknown services: all". If I run service-control --status it displays:

"service-control --status

INFO:root:Service: vmware-rhttpproxy, Action: status

Service: vmware-rhttpproxy, Action: status

2018-06-14T18:17:47.244Z   Running command: ['/sbin/service', u'vmware-rhttpprox                                                                             y', 'status']

2018-06-14T18:17:47.279Z   Done running command

INFO:root:Service: vmware-cis-license, Action: status

Service: vmware-cis-license, Action: status

2018-06-14T18:17:47.280Z   Running command: ['/sbin/service', u'vmware-cis-licen                                                                             se', 'status']

2018-06-14T18:17:47.418Z   Done running command

INFO:root:Service: vmware-psc-client, Action: status

Service: vmware-psc-client, Action: status

2018-06-14T18:17:47.418Z   Running command: ['/sbin/service', u'vmware-psc-clien                                                                             t', 'status']

2018-06-14T18:17:47.449Z   Done running command

INFO:root:Service: vmware-sca, Action: status

Service: vmware-sca, Action: status

2018-06-14T18:17:47.450Z   Running command: ['/sbin/service', u'vmware-sca', 'st                                                                             atus']

2018-06-14T18:17:47.540Z   Done running command

INFO:root:Service: vmware-netdumper, Action: status

Service: vmware-netdumper, Action: status

2018-06-14T18:17:47.540Z   Running command: ['/sbin/service', u'vmware-netdumper                                                                             ', 'status']

2018-06-14T18:17:47.585Z   Done running command

INFO:root:Service: vmware-syslog, Action: status

Service: vmware-syslog, Action: status

2018-06-14T18:17:47.585Z   Running command: ['/sbin/service', u'vmware-syslog',                                                                              'status']

2018-06-14T18:17:47.637Z   Done running command

INFO:root:Service: vmware-syslog-health, Action: status

Service: vmware-syslog-health, Action: status

2018-06-14T18:17:47.638Z   Running command: ['/sbin/service', u'vmware-syslog-he                                                                             alth', 'status']

2018-06-14T18:17:47.750Z   Done running command

INFO:root:Service: vmware-vapi-endpoint, Action: status

Service: vmware-vapi-endpoint, Action: status

2018-06-14T18:17:47.750Z   Running command: ['/sbin/service', u'vmware-vapi-endp                                                                             oint', 'status']

2018-06-14T18:17:47.798Z   Done running command

INFO:root:Service: vmware-vpostgres, Action: status

Service: vmware-vpostgres, Action: status

2018-06-14T18:17:47.799Z   Running command: ['/sbin/service', u'vmware-vpostgres                                                                             ', 'status']

2018-06-14T18:17:47.924Z   Done running command

INFO:root:Service: vmware-invsvc, Action: status

Service: vmware-invsvc, Action: status

2018-06-14T18:17:47.925Z   Running command: ['/sbin/service', u'vmware-invsvc',                                                                              'status']

2018-06-14T18:17:48.025Z   Done running command

INFO:root:Service: vmware-mbcs, Action: status

Service: vmware-mbcs, Action: status

2018-06-14T18:17:48.025Z   Running command: ['/sbin/service', u'vmware-mbcs', 's                                                                             tatus']

2018-06-14T18:17:48.123Z   Done running command

INFO:root:Service: vmware-vpxd, Action: status

Service: vmware-vpxd, Action: status

2018-06-14T18:17:48.124Z   Running command: ['/sbin/service', u'vmware-vpxd', 's                                                                             tatus']

2018-06-14T18:17:48.509Z   Done running command

INFO:root:Service: vmware-eam, Action: status

Service: vmware-eam, Action: status

2018-06-14T18:17:48.509Z   Running command: ['/sbin/service', u'vmware-eam', 'st                                                                             atus']

2018-06-14T18:17:48.575Z   Done running command

INFO:root:Service: vmware-rbd-watchdog, Action: status

Service: vmware-rbd-watchdog, Action: status

2018-06-14T18:17:48.575Z   Running command: ['/sbin/service', u'vmware-rbd-watch                                                                             dog', 'status']

2018-06-14T18:17:48.617Z   Done running command

INFO:root:Service: vmware-sps, Action: status

Service: vmware-sps, Action: status

2018-06-14T18:17:48.617Z   Running command: ['/sbin/service', u'vmware-sps', 'st                                                                             atus']

2018-06-14T18:17:48.747Z   Done running command

INFO:root:Service: vmware-vdcs, Action: status

Service: vmware-vdcs, Action: status

2018-06-14T18:17:48.748Z   Running command: ['/sbin/service', u'vmware-vdcs', 's                                                                             tatus']

2018-06-14T18:17:48.838Z   Done running command

INFO:root:Service: vmware-vpx-workflow, Action: status

Service: vmware-vpx-workflow, Action: status

2018-06-14T18:17:48.838Z   Running command: ['/sbin/service', u'vmware-vpx-workf                                                                             low', 'status']

2018-06-14T18:17:48.894Z   Done running command

INFO:root:Service: vmware-vsan-health, Action: status

Service: vmware-vsan-health, Action: status

2018-06-14T18:17:48.894Z   Running command: ['/sbin/service', u'vmware-vsan-heal                                                                             th', 'status']

2018-06-14T18:17:48.902Z   Done running command

INFO:root:Service: vmware-vsm, Action: status

Service: vmware-vsm, Action: status

2018-06-14T18:17:48.903Z   Running command: ['/sbin/service', u'vmware-vsm', 'st                                                                             atus']

2018-06-14T18:17:48.953Z   Done running command

INFO:root:Service: vsphere-client, Action: status

Service: vsphere-client, Action: status

2018-06-14T18:17:48.953Z   Running command: ['/sbin/service', u'vsphere-client',                                                                              'status']

2018-06-14T18:17:49.045Z   Done running command

INFO:root:Service: vmware-perfcharts, Action: status

Service: vmware-perfcharts, Action: status

2018-06-14T18:17:49.045Z   Running command: ['/sbin/service', u'vmware-perfchart                                                                             s', 'status']

2018-06-14T18:17:49.106Z   Done running command

INFO:root:Service: vmware-vws, Action: status

Service: vmware-vws, Action: status

2018-06-14T18:17:49.107Z   Running command: ['/sbin/service', u'vmware-vws', 'st                                                                             atus']

2018-06-14T18:17:49.168Z   Done running command

INFO:root:Running:

vmware-cis-license (VMware License Service) vmware-perfcharts (VMware Performan                                                                             ce Charts) vmware-psc-client (VMware Platform Services Controller Client) vmware                                                                             -rhttpproxy (VMware HTTP Reverse Proxy) vmware-sps (VMware vSphere Profile-Drive                                                                             n Storage Service) vmware-syslog (VMware Common Logging Service) vmware-syslog-h                                                                             ealth (VMware Syslog Health Service) vmware-vpostgres (VMware Postgres) vmware-v                                                                             san-health (VMware VSAN Health Service) vmware-vws (VMware System and Hardware H                                                                             ealth Manager)

Running:

vmware-cis-license (VMware License Service) vmware-perfcharts (VMware Performan                                                                             ce Charts) vmware-psc-client (VMware Platform Services Controller Client) vmware                                                                             -rhttpproxy (VMware HTTP Reverse Proxy) vmware-sps (VMware vSphere Profile-Drive                                                                             n Storage Service) vmware-syslog (VMware Common Logging Service) vmware-syslog-h                                                                             ealth (VMware Syslog Health Service) vmware-vpostgres (VMware Postgres) vmware-v                                                                             san-health (VMware VSAN Health Service) vmware-vws (VMware System and Hardware H                                                                             ealth Manager)

INFO:root:Stopped:

vmware-eam (VMware ESX Agent Manager) vmware-invsvc (VMware Inventory Service)                                                                              vmware-mbcs (VMware Message Bus Configuration Service) vmware-netdumper (VMware                                                                              vSphere ESXi Dump Collector) vmware-rbd-watchdog (VMware vSphere Auto Deploy Wai                                                                             ter) vmware-sca (VMware Service Control Agent) vmware-vapi-endpoint (VMware vAPI                                                                              Endpoint) vmware-vdcs (VMware Content Library Service) vmware-vpx-workflow (VMw                                                                             are vCenter Workflow Manager) vmware-vpxd (VMware vCenter Server) vmware-vsm (VM                                                                             ware vService Manager) vsphere-client ()

Stopped:

vmware-eam (VMware ESX Agent Manager) vmware-invsvc (VMware Inventory Service)                                                                              vmware-mbcs (VMware Message Bus Configuration Service) vmware-netdumper (VMware                                                                              vSphere ESXi Dump Collector) vmware-rbd-watchdog (VMware vSphere Auto Deploy Wai                                                                             ter) vmware-sca (VMware Service Control Agent) vmware-vapi-endpoint (VMware vAPI                                                                              Endpoint) vmware-vdcs (VMware Content Library Service) vmware-vpx-workflow (VMw                                                                             are vCenter Workflow Manager) vmware-vpxd (VMware vCenter Server) vmware-vsm (VM                                                                             ware vService Manager) vsphere-client ()"

[Finikiez]

I missed that you use vcsa 6.0

You can't connect to VC because main services are stopped vmware-invsvc, vmware-vapi-endpoint, vmware-vpxd and vsphere-client

Try to start them manually and see what happened.

If they can't be started manually then you need to check the logs in /var/log/vmware directory

[Viceman256]

I can start the vmware-vapi-endpoint and vsphere-client. I tried the vmware-vpxd then it got stuck for several minutes and then I cancelled the start, but when I check the status it says running.  When I try to start vmware-invsvs I get the errors:

bbfc-vcenter:~ # service vmware-invsvc start

Starting VMware Inventory Service...

Waiting for VMware Inventory Service..............

WARNING: VMware Inventory Service may have failed to start.

bbfc-vcenter:~ # service vmware-invsvc status

VMware Inventory Service is not running.

I'm not sure which logs to check in that directory.

[Finikiez]

Sorry, notifications are broken and I can't reply in time.

Can you go to /var/log/vmware/invsvc and check latest ds.log

There could inventory sevice database corruption. However it's better to check the logs.

[Viceman256]

Hi Finikiez,

When I browse to that directory, it actually has no ds.log file there.

Thanks,

Vince

[Finikiez]

I don't have VC 6.0 installed at the moment. Can you show a schreenshot? or can you check if there is a Logs directory or something on that path?

[Viceman256]

I've attached the screenshots. There are logs, I'm just not sure which one I'm looking for.

Thank you!

[Finikiez]

ok, try to start inventory service again and then attach inv-svc.log

[golddiggie]

Curious to know what database your using for the vCSA. Is it the one on the vCSA or are you connecting to an external db?? I have an issue with my vCenter Server (some time ago) when the MS SQL server drive was filled up by the transaction log files. Once I resolved that issue, vCenter ran just fine. But before then, it would exhibit pretty much the same behavior you're facing.

[Viceman256 · Accepted solution]

I believe the DB was located on the same VM, but I wouldn't be able to tell you for sure. I appreciate the help from everyone, but due to the need of having to get backups running again, I just deployed a new VCSA and configured it. This issue can me marked as resolved, thank you.