Hello, we ran into the issue of the root volume running out of disk space due to very large log files the other day. I manged to clear up the space but I had to hard crash and reboot the vCenter virtual appliance in the process.
The system is back up and working, except vCenter is not starting up.
The issue I am running into is the vmware-invsvc and vmware-vpxd service will not start.
Starting VMware Inventory Service...
Waiting for VMware Inventory Service................................................
WARNING: VMware Inventory Service may have failed to start.
error logs from /var/log/vmware/invsvc/wrapper.log
STATUS | wrapper | 2017/09/25 19:30:51 |
INFO | wrapper | 2017/09/25 19:30:51 | Event Command 'wrapper_start': Command line[0] : /usr/sbin/cloudvm-ram-size
INFO | wrapper | 2017/09/25 19:30:51 | Event Command 'wrapper_start': Command line[1] : -J
INFO | wrapper | 2017/09/25 19:30:51 | Event Command 'wrapper_start': Command line[2] : vmware-invsvc
INFO | wrapper | 2017/09/25 19:30:51 | Event Command 'wrapper_start': Command line[3] : -O
INFO | wrapper | 2017/09/25 19:30:51 | Event Command 'wrapper_start': Command line[4] : /etc/vmware/java/vmware-invsvc_jvm.conf
INFO | wrapper | 2017/09/25 19:30:51 | Event Command 'wrapper_start': Command launched (pid: 4360), blocking for up to 3600 seconds...
INFO | wrapper | 2017/09/25 19:30:51 | Event Command 'wrapper_start': Command completed with exit code: 0
INFO | wrapper | 2017/09/25 19:30:51 | Event Command 'wrapper_start': Command completed with exit code: 0 Continuing.
INFO | wrapper | 2017/09/25 19:30:51 | Event Command 'jvm_prelaunch': Command line[0] : ln
INFO | wrapper | 2017/09/25 19:30:51 | Event Command 'jvm_prelaunch': Command line[1] : -f
INFO | wrapper | 2017/09/25 19:30:51 | Event Command 'jvm_prelaunch': Command line[2] : /usr/java/jre-vmware/bin/java
INFO | wrapper | 2017/09/25 19:30:51 | Event Command 'jvm_prelaunch': Command line[3] : /usr/java/jre-vmware/bin/vmware-invsvc
INFO | wrapper | 2017/09/25 19:30:51 | Event Command 'jvm_prelaunch': Command launched (pid: 4363), blocking for up to 15 seconds...
INFO | wrapper | 2017/09/25 19:30:51 | Event Command 'jvm_prelaunch': Command completed with exit code: 0
INFO | wrapper | 2017/09/25 19:30:51 | Event Command 'jvm_prelaunch': Command completed with exit code: 0 Continuing.
STATUS | wrapper | 2017/09/25 19:30:51 | Launching a JVM...
INFO | jvm 1 | 2017/09/25 19:30:52 | Picked up JAVA_TOOL_OPTIONS: -Xms32M -Xmx128M
INFO | jvm 1 | 2017/09/25 19:30:52 | WrapperManager: Initializing...
STATUS | wrapper | 2017/09/25 19:31:44 | <-- Wrapper Stopped
STATUS | wrapper | 2017/09/25 20:39:14 | --> Wrapper Started as Daemon
STATUS | wrapper | 2017/09/25 20:39:14 | Java Service Wrapper Professional Edition 64-bit 3.5.24
STATUS | wrapper | 2017/09/25 20:39:14 | Copyright (C) 1999-2014 Tanuki Software, Ltd. All Rights Reserved.
STATUS | wrapper | 2017/09/25 20:39:14 | http://wrapper.tanukisoftware.com
STATUS | wrapper | 2017/09/25 20:39:14 | Licensed to VMware Global, Inc. for VMware vCenter Inventory Service
STATUS | wrapper | 2017/09/25 20:39:14 |
INFO | wrapper | 2017/09/25 20:39:14 | Event Command 'wrapper_start': Command line[0] : /usr/sbin/cloudvm-ram-size
INFO | wrapper | 2017/09/25 20:39:14 | Event Command 'wrapper_start': Command line[1] : -J
INFO | wrapper | 2017/09/25 20:39:14 | Event Command 'wrapper_start': Command line[2] : vmware-invsvc
INFO | wrapper | 2017/09/25 20:39:14 | Event Command 'wrapper_start': Command line[3] : -O
INFO | wrapper | 2017/09/25 20:39:14 | Event Command 'wrapper_start': Command line[4] : /etc/vmware/java/vmware-invsvc_jvm.conf
INFO | wrapper | 2017/09/25 20:39:14 | Event Command 'wrapper_start': Command launched (pid: 26295), blocking for up to 3600 seconds...
INFO | wrapper | 2017/09/25 20:39:14 | Event Command 'wrapper_start': Command completed with exit code: 0
INFO | wrapper | 2017/09/25 20:39:14 | Event Command 'wrapper_start': Command completed with exit code: 0 Continuing.
INFO | wrapper | 2017/09/25 20:39:14 | Event Command 'jvm_prelaunch': Command line[0] : ln
INFO | wrapper | 2017/09/25 20:39:14 | Event Command 'jvm_prelaunch': Command line[1] : -f
INFO | wrapper | 2017/09/25 20:39:14 | Event Command 'jvm_prelaunch': Command line[2] : /usr/java/jre-vmware/bin/java
INFO | wrapper | 2017/09/25 20:39:14 | Event Command 'jvm_prelaunch': Command line[3] : /usr/java/jre-vmware/bin/vmware-invsvc
INFO | wrapper | 2017/09/25 20:39:14 | Event Command 'jvm_prelaunch': Command launched (pid: 26298), blocking for up to 15 seconds...
INFO | wrapper | 2017/09/25 20:39:14 | Event Command 'jvm_prelaunch': Command completed with exit code: 0
INFO | wrapper | 2017/09/25 20:39:14 | Event Command 'jvm_prelaunch': Command completed with exit code: 0 Continuing.
STATUS | wrapper | 2017/09/25 20:39:14 | Launching a JVM...
INFO | jvm 1 | 2017/09/25 20:39:14 | Picked up JAVA_TOOL_OPTIONS: -Xms32M -Xmx128M
INFO | jvm 1 | 2017/09/25 20:39:15 | WrapperManager: Initializing...
STATUS | wrapper | 2017/09/25 20:39:59 | <-- Wrapper Stopped
vcenter:/var/log/vmware/invsvc # cat vmware-invsvc-gc.log.0
2017-09-25T20:39:17.314+0000: 2.794: [GC2017-09-25T20:39:17.330+0000: 2.810: [SoftReference, 0 refs, 0.0003020 secs]2017-09-25T20:39:17.330+0000: 2.810: [WeakReference, 1097 refs, 0.0002410 secs]2017-09-25T20:39:17.330+0000: 2.810: [FinalReference, 3318 refs, 0.0055650 secs]2017-09-25T20:39:17.336+0000: 2.816: [PhantomReference, 1 refs, 0.0001440 secs]2017-09-25T20:39:17.336+0000: 2.816: [JNI Weak Reference, 0.0000230 secs] [PSYoungGen: 132096K->11180K(153600K)] 132096K->11188K(503296K), 0.0236970 secs] [Times: user=0.04 sys=0.02, real=0.02 secs]
2017-09-25T20:39:18.798+0000: 4.278: [GC2017-09-25T20:39:18.815+0000: 4.295: [SoftReference, 0 refs, 0.0004490 secs]2017-09-25T20:39:18.815+0000: 4.295: [WeakReference, 1055 refs, 0.0003090 secs]2017-09-25T20:39:18.816+0000: 4.296: [FinalReference, 4585 refs, 0.0044610 secs]2017-09-25T20:39:18.820+0000: 4.300: [PhantomReference, 0 refs, 0.0001230 secs]2017-09-25T20:39:18.820+0000: 4.300: [JNI Weak Reference, 0.0000170 secs] [PSYoungGen: 143276K->12052K(153600K)] 143284K->12132K(503296K), 0.0246030 secs] [Times: user=0.05 sys=0.02, real=0.03 secs]
2017-09-25T20:39:20.235+0000: 5.715: [GC2017-09-25T20:39:20.259+0000: 5.739: [SoftReference, 0 refs, 0.0002660 secs]2017-09-25T20:39:20.259+0000: 5.739: [WeakReference, 1058 refs, 0.0002230 secs]2017-09-25T20:39:20.259+0000: 5.739: [FinalReference, 3120 refs, 0.0037950 secs]2017-09-25T20:39:20.263+0000: 5.743: [PhantomReference, 0 refs, 0.0002100 secs]2017-09-25T20:39:20.263+0000: 5.743: [JNI Weak Reference, 0.0000430 secs] [PSYoungGen: 144148K->17825K(153600K)] 144228K->17913K(503296K), 0.0303560 secs] [Times: user=0.07 sys=0.01, real=0.03 secs]
2017-09-25T20:39:21.371+0000: 6.851: [GC2017-09-25T20:39:21.426+0000: 6.906: [SoftReference, 0 refs, 0.0006670 secs]2017-09-25T20:39:21.427+0000: 6.907: [WeakReference, 1061 refs, 0.0004830 secs]2017-09-25T20:39:21.427+0000: 6.907: [FinalReference, 3052 refs, 0.0066890 secs]2017-09-25T20:39:21.434+0000: 6.914: [PhantomReference, 0 refs, 0.0001320 secs]2017-09-25T20:39:21.434+0000: 6.914: [JNI Weak Reference, 0.0000100 secs] [PSYoungGen: 149921K->21495K(285696K)] 150009K->30004K(635392K), 0.0658520 secs] [Times: user=0.16 sys=0.02, real=0.07 secs]
2017-09-25T20:39:23.395+0000: 8.875: [GC2017-09-25T20:39:23.498+0000: 8.978: [SoftReference, 0 refs, 0.0004150 secs]2017-09-25T20:39:23.499+0000: 8.979: [WeakReference, 0 refs, 0.0001070 secs]2017-09-25T20:39:23.499+0000: 8.979: [FinalReference, 0 refs, 0.0001230 secs]2017-09-25T20:39:23.499+0000: 8.979: [PhantomReference, 0 refs, 0.0001010 secs]2017-09-25T20:39:23.499+0000: 8.979: [JNI Weak Reference, 0.0000170 secs] [PSYoungGen: 285687K->21504K(285696K)] 294196K->53238K(635392K), 0.1084870 secs] [Times: user=0.21 sys=0.10, real=0.11 secs]
Heap
PSYoungGen total 285696K, used 171249K [0x00000007b8800000, 0x00000007de700000, 0x0000000800000000)
eden space 264192K, 56% used [0x00000007b8800000,0x00000007c1a3c790,0x00000007c8a00000)
from space 21504K, 100% used [0x00000007c8a00000,0x00000007c9f00000,0x00000007c9f00000)
to space 46592K, 0% used [0x00000007db980000,0x00000007db980000,0x00000007de700000)
ParOldGen total 349696K, used 31734K [0x0000000729800000, 0x000000073ed80000, 0x00000007b8800000)
object space 349696K, 9% used [0x0000000729800000,0x000000072b6fdad8,0x000000073ed80000)
PSPermGen total 58368K, used 58223K [0x0000000719800000, 0x000000071d100000, 0x0000000729800000)
object space 58368K, 99% used [0x0000000719800000,0x000000071d0dbc08,0x000000071d100000)
error log from inv-svc.log indicated LDAP error.
; nested exception is org.springframework.beans.BeanInstantiationException: Could not instantiate bean class [com.vmware.vim.query.server.accesscontrol.impl.Lo tusInitializer]: Constructor threw exception; nested exception is java.lang.RuntimeException: com.vmware.identity.interop.ldap.InvalidCredentialsLdapException: Invalid credentials
LDAP error [code: 49]
at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveReference(BeanDefinitionValueResolver.java:328)
at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveValueIfNecessary(BeanDefinitionValueResolver.java:106)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.applyPropertyValues(AbstractAutowireCapableBeanFactory.java:1317)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.populateBean(AbstractAutowireCapableBeanFactory.java:1076)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:517)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:456)
at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:291)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:222)
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:288)
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:190)
at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveReference(BeanDefinitionValueResolver.java:322)
... 65 more
Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'memCache' defined in class path resource [server/config/secu rity-config.xml]: Cannot resolve reference to bean 'globalAclLotusCache' while setting bean property 'globalAclLotusCache'; nested exception is org.springframe work.beans.factory.BeanCreationException: Error creating bean with name 'aclLotusInitializer' defined in class path resource [server/config/authorization-confi g.xml]: Instantiation of bean failed; nested exception is org.springframework.beans.BeanInstantiationException: Could not instantiate bean class [com.vmware.vi m.query.server.accesscontrol.impl.LotusInitializer]: Constructor threw exception; nested exception is java.lang.RuntimeException: com.vmware.identity.interop.l dap.InvalidCredentialsLdapException: Invalid credentials
LDAP error [code: 49]
at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveReference(BeanDefinitionValueResolver.java:328)
at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveValueIfNecessary(BeanDefinitionValueResolver.java:106)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.applyPropertyValues(AbstractAutowireCapableBeanFactory.java:1317)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.populateBean(AbstractAutowireCapableBeanFactory.java:1076)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:517)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:456)
at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:291)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:222)
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:288)
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:190)
at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveReference(BeanDefinitionValueResolver.java:322)
... 75 more
Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'aclLotusInitializer' defined in class path resource [server/ config/authorization-config.xml]: Instantiation of bean failed; nested exception is org.springframework.beans.BeanInstantiationException: Could not instantiate bean class [com.vmware.vim.query.server.accesscontrol.impl.LotusInitializer]: Constructor threw exception; nested exception is java.lang.RuntimeException: com .vmware.identity.interop.ldap.InvalidCredentialsLdapException: Invalid credentials
LDAP error [code: 49]
at org.springframework.beans.factory.support.ConstructorResolver.autowireConstructor(ConstructorResolver.java:286)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.autowireConstructor(AbstractAutowireCapableBeanFactory.java:993)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:897)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:485)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:456)
at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:291)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:222)
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:288)
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:190)
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:281)
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:190)
at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveReference(BeanDefinitionValueResolver.java:322)
... 85 more
Caused by: org.springframework.beans.BeanInstantiationException: Could not instantiate bean class [com.vmware.vim.query.server.accesscontrol.impl.LotusInitiali zer]: Constructor threw exception; nested exception is java.lang.RuntimeException: com.vmware.identity.interop.ldap.InvalidCredentialsLdapException: Invalid cr edentials
LDAP error [code: 49]
at org.springframework.beans.BeanUtils.instantiateClass(BeanUtils.java:141)
at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:105)
at org.springframework.beans.factory.support.ConstructorResolver.autowireConstructor(ConstructorResolver.java:278)
... 96 more
Caused by: java.lang.RuntimeException: com.vmware.identity.interop.ldap.InvalidCredentialsLdapException: Invalid credentials
LDAP error [code: 49]
at com.vmware.cis.lotus.LdapConnectionFactory.getConnection(LdapConnectionFactory.java:142)
at com.vmware.vim.query.server.accesscontrol.impl.LotusInitializer.<init>(LotusInitializer.java:26)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(Unknown Source)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown Source)
at java.lang.reflect.Constructor.newInstance(Unknown Source)
at org.springframework.beans.BeanUtils.instantiateClass(BeanUtils.java:126)
... 98 more
I tried resetting the the password using /usr/lib/vmware-vmdir/bin/vdcadmintool
and that did not fix it.
any help or advise appreciated.
Logon that esxi where this virtual machine running, do the clean power off and power on.
Or try
You may need to follow the steps outlined in KB 2119422.
Thanks. Yes, I saw this article.
My concern is, this is a pretty complex setup. Will we lose all our settings in the environment? the vSwitches and connections to the backend storage if we do this?
The inventory service does not store anything having to do with host- or vCenter-level networking nor backend storage settings. It's used for things like tags and storage profiles. Open a case with GSS if you have specific concerns and need guidance.
Thanks.
I just tried the script. I am getting
###
# Step 5: Getting the current vCenter Server Solution User Certificates
##
Error: Failed to open the store.
vecs-cli failed. Error 1021: Cannot connect to vmafd service.
FORCEFULLY ABORTING! Could not fetch vCenter Server Solution User private key.
I assumed you have external PSC, try these step if your drive have enough space.
Shut down Vcenter
Shut down PSC
Power on in revers order
Power on psc
Power on VC.
It does not look like we have one.
vcenter:~/bin # service-control --start vmware-eam
INFO:root:Service: vmware-eam, Action: start
Service: vmware-eam, Action: start
2017-09-26T14:29:20.115Z Running command: ['/sbin/chkconfig', u'vmware-eam']
2017-09-26T14:29:21.073Z Done running command
2017-09-26T14:29:21.074Z Running command: ['/sbin/service', u'vmware-eam', 'status']
2017-09-26T14:29:21.396Z Done running command
2017-09-26T14:29:21.397Z Running command: ['/sbin/chkconfig', '--force', u'vmware-eam', 'on']
2017-09-26T14:29:21.452Z Done running command
2017-09-26T14:29:21.453Z Running command: ['/sbin/service', u'vmware-eam', 'start']
2017-09-26T14:29:28.060Z Done running command
2017-09-26T14:29:28.060Z Invoked command: ['/sbin/service', u'vmware-eam', 'start']
2017-09-26T14:29:28.060Z RC = 1
Stdout = Starting VMware ESX Agent Manager...
Waiting for VMware ESX Agent Manager........
WARNING: VMware ESX Agent Manager may have failed to start.
Last login: Tue Sep 26 14:23:24 UTC 2017
Stderr =
2017-09-26T14:29:28.061Z {
"resolution": null,
"detail": [
{
"args": [
"Command: ['/sbin/service', u'vmware-eam', 'start']\nStderr: "
],
"id": "install.ciscommon.command.errinvoke",
"localized": "An error occurred while invoking external command : 'Command: ['/sbin/service', u'vmware-eam', 'start']\nStderr: '",
"translatable": "An error occurred while invoking external command : '%(0)s'"
}
],
"componentKey": null,
"problemId": null
}
ERROR:root:Unable to start service vmware-eam, Exception: {
"resolution": null,
"detail": [
{
"args": [
"vmware-eam"
],
"id": "install.ciscommon.service.failstart",
"localized": "An error occurred while starting service 'vmware-eam'",
"translatable": "An error occurred while starting service '%(0)s'"
}
],
"componentKey": null,
"problemId": null
}
Unable to start service vmware-eam, Exception: {
"resolution": null,
"detail": [
{
"args": [
"vmware-eam"
],
"id": "install.ciscommon.service.failstart",
"localized": "An error occurred while starting service 'vmware-eam'",
"translatable": "An error occurred while starting service '%(0)s'"
}
],
"componentKey": null,
"problemId": null
}
Logon that esxi where this virtual machine running, do the clean power off and power on.
Or try
Thank you. After working almost 16+ hours on this, I finally got the built in Apache Tomcat application to boot on vCenter 6.0
The link you have was helpful.
I fixed it by:
- fixing NTP to work properly. The previous admin seems to have never setup NTP
- reset the password
- removed a rouge entry from an xml file /usr/lib/vmware-invsvc/lib/server/config/ . it had the password in plain text just sitting on a line by itself. i managed to debug that by looking through logs.
- i also re-generated certificates and re-registered them.
After a clean shutdown and reboot, the server came back up normally.
Lesson learnt -- never hard crash vCenter appliance.
Thank you.
By altering the password of Oracle, it works.