vCenter v6.0 u3
So I ran in to some problems when I came back to work after my vacation. vCenter Servers disk space was filled to max, because of not having changed the root password for awhile, so that the rotation of the logs didn't work and therefore filled my disks.
I free some space by following KB2069041 - https://kb.vmware.com/s/article/2069041
Change the root password and then went to
/var/log/audit
> audit.log
Restarted the vCenter and it work fine until the service Inventory Service comes along. This service woun't start.
I could here log in to the shell and check some more.
I can see that the rotation of the logs works
ls -l /var/spool/cron/lastrun/
To clear up even more log files I found this, don't know if its any good, but here it is.
http://easi123.4realit.nl/2016/07/safely-cleaning-up-log-files-vcentervrealize/
find /storage/log/ -mount -type f -mtime +1 -exec echo {} \; -exec truncate -cs 0 {} \; 2>&1 | tee /tmp/files_truncated.txt
Tried to start the service again manually
service-control --start --all
But when it comes down to Inventory Service this error appears and the startup seq. ends.
****************************
Stdout = Starting VMware Inventory Service...
Waiting for VMware Inventory Service...............
WARNING: VMware Inventory Service may have failed to start.
Stderr =
2018-07-25T06:05:31.155Z {
"resolution": null,
"detail": [
{
"args": [
"Command: ['/sbin/service', u'vmware-invsvc', 'start']\nStderr: "
],
"id": "install.ciscommon.command.errinvoke",
"localized": "An error occurred while invoking external command : 'Command: ['/sbin/service', u'vmware-invsvc', 'start']\nStderr: '",
"translatable": "An error occurred while invoking external command : '%(0)s'"
}
],
"componentKey": null,
"problemId": null
}
ERROR:root:Unable to start service vmware-invsvc, Exception: {
"resolution": null,
"detail": [
{
"args": [
"vmware-invsvc"
],
"id": "install.ciscommon.service.failstart",
"localized": "An error occurred while starting service 'vmware-invsvc'",
"translatable": "An error occurred while starting service '%(0)s'"
}
],
"componentKey": null,
"problemId": null
}
Unable to start service vmware-invsvc, Exception: {
"resolution": null,
"detail": [
{
"args": [
"vmware-invsvc"
],
"id": "install.ciscommon.service.failstart",
"localized": "An error occurred while starting service 'vmware-invsvc'",
"translatable": "An error occurred while starting service '%(0)s'"
}
],
"componentKey": null,
"problemId": null
}
****************************
I have tried to clear the Inventory by following KB2146248. This made no diffrens.
So I backed to the snapshot I did before trying KB2146248.
So now I ran out of ide's, anyone else have any suggestion what to do now
How to get the Inventory Service to start again.
/ Per W
Encountered similar issue recently.
Post clearing audit.log inventory service failed to start. Verified inventory service logs and resolved the issue by following the KB:
Make sure you take a snapshot before you proceed with the troubleshooting.
Encountered similar issue recently.
Post clearing audit.log inventory service failed to start. Verified inventory service logs and resolved the issue by following the KB:
Make sure you take a snapshot before you proceed with the troubleshooting.
Thanks! This worked for me.
The Inventory Service is now starting. Yeah!
But it only took me to the next service "vmware-eam" or VMware ESX Agent Manager, is not starting.
So the troubleshooting continues...
Look at this community thread:
ESX Agent Manager - vmware-eam will not start
You will need to change below entries:
cm.url=http://localhost:18090/cm/sdk/?hostid=0a12dfe2-b3b7-4eb6-9737-1ec5a8e770ec
vc.tunnelSdkUri=<vCenter_FQDN>:8089/sdk/vimService
You will have to paste the hostid from cat /etc/vmware/install-defaults/sca.hostid:
Were you able to fix eam service?
So the eam service didn't start, read some articles about the problem, then checked my file /etc/vmware-eam/eam.properties
It was completely blanked out except for one row. This might occur when the disk is full I read on some page.
Started two sessions to the vCenter and ran in one to see the live logs.
tail -f /var/log/vmware/eam/eam.log
Then started the service in the next.
service-control --start vmware-eam
Got the following in the log:
Exception encountered during context initialization - cancelling refresh attempt: org.springframework.beans.factory.BeanDefinitionStoreException: Invalid bean definition with name 'esxAgentMgr' defined in class path resource [eam-server-beans.xml]: Could not resolve placeholder 'eam.hostd_restart_timeout' in string value "${eam.hostd_restart_timeout}"; nested exception is java.lang.IllegalArgumentException: Could not resolve placeholder 'eam.hostd_restart_timeout' in string value "${eam.hostd_restart_timeout}"
Searched the internet for awhile and found a KB.
I Followed KB2148582, https://kb.vmware.com/s/article/2148582, to solve the problem.
I downloaded the attached file of the KB and ran the script to create a new eam.properties file.
After completing the file I tried to start the service again.
service-control --start vmware-eam
It worked.
All services on my vCenter is now starting!
Thanks for the help.
Removed all my error-searching-snapshots and did a fresh backup of the appliance and the PostgreSQL database.
Hopefully this error woun't occur again.
The root of the problem was that the root password hadn't been changed for a long while. This resulted in the log-files to grow, and never rotate.
So my tip is to change the root password once a year or just log in to your vCenter
https://<vCenter-address>:5480
and then go to administation and change the "Password expiry settings" to -
/ Per