First noticed we have a problem when we experienced 503 errors accessing the web UI on our vCenter v6 box. After further investigations I found that the / volume was at 100% percent thanks to the root password expiring and writing to the audit.log. Resolved that problem by following a thread on increasing the / volume size and fixing the root password. We then had a problem with starting the inventory service, getting a LDAP 49 error which we resolved by following another thread on changing the account password using /usr/lib/vmware-vmdir/bin/vdcadmintool
With the Inventory service running, we now have the Agent Manager problem.
/var/log/vmware/eam/wrapper.log
INFO | wrapper | 2018/04/11 09:36:19 | Event Command 'jvm_prelaunch': Command launched (pid: 17813), blocking for up to 15 seconds...
INFO | wrapper | 2018/04/11 09:36:20 | Event Command 'jvm_prelaunch': Command completed with exit code: 1
INFO | wrapper | 2018/04/11 09:36:20 | Event Command 'jvm_prelaunch': Command completed with exit code: 1 Continuing.
STATUS | wrapper | 2018/04/11 09:36:20 | Launching a JVM...
INFO | jvm 1 | 2018/04/11 09:36:20 | Picked up JAVA_TOOL_OPTIONS: -Xms32M -Xmx128M
INFO | jvm 1 | 2018/04/11 09:36:20 | WrapperManager: Initializing...
INFO | jvm 1 | 2018/04/11 09:36:21 | WrapperSimpleApp:
INFO | jvm 1 | 2018/04/11 09:36:21 | WrapperSimpleApp Error: Encountered an error running main:
INFO | jvm 1 | 2018/04/11 09:36:21 | WrapperSimpleApp Error: org.springframework.beans.factory.BeanDefinitionStoreException: Invalid bean definition with name 'esxAgentMgr' defined in class path resource [eam-server-beans.xml]: Could not resolve placeholder 'eam.hostd_restart_timeout'
Searching the error message I found that our eam.properties file in /etc/vmware-eam was empty. We edited the existing file and copied and pasted from the thread we found.
My eam.properties file: -
#########################################################################
# Copyright 2013-2017 VMware, Inc. All rights reserved. VMware Confidential
#########################################################################
eam.web.root=/usr/lib/vmware-eam/web
vc.proxy.host=localhost
vc.proxy.port=80
# Hostname or IP of the EAM server
# Fill only if EAM is not running on the same host as VC
eam.host=
# EAM service port used to configure the HTTP connector of the application server.
eam.int.http.port=15005
# Port and scheme configuration which is used by the ESX 6.x hosts to reach EAM Vib
# file server.
eam.ext.port=443
eam.ext.scheme=https
# Port and scheme configuration which is used by the ESX 5.x hosts to reach EAM Vib
# file server.
eam.ext.port.deprecated=80
eam.ext.scheme.deprecated=http
eam.support_linked_clone=true
eam.clear_db_on_startup=false
eam.debug_ref_count=false
eam.recent_event_size=20
# Value is specified in minutes (set to 24h = 1440m)
eam.scan_for_unknown_agent_vms=1440
# The timeout to wait for hostd to restart on a host (set to 5m=300s)
eam.hostd_restart_timeout=300
# The following entries will be added verbatim to the advanced options
# of hosts on which EAM is enabled. All are optional.
Net.DVFilterBindIpAddress=169.254.0.1
Net.TrafficFilterIpAddress=
#The IP for the VSWIF NIC on the dvFilter switch (for ESX classic).
Net.DVFilterVswifIpAddress=169.254.0.2
# Resource bundle configuration
eameam.resourcebundle.filename=eam-resourcebundle.jar
# VLSI embedded tcServer configuration
#
tcserver.tmp.dir=/var/tmp/vmware/eam/tomcat
eam.web.root=/usr/lib/vmware-eam/web
# EAM SSL configuration
#
eam.keystore.type=VKS
eam.key.alias=vpxd-extension
eam.keystore.storename=vpxd-extension
# CM configuration
#
cm.url=http://localhost:18090/cm/sdk/?hostid=4f1feb00-5481-47e8-a7c3-6808106d7fdb4
cm.wait.attempts=360
cm.wait.intervalSeconds=5
# SSO configuration
#
sso.wait.attempts=360
sso.wait.intervalSeconds=5
# VC SSL configuration
#
vc.truststore.type=VKS
vc.truststore.storename=TRUSTED_ROOTS
vc.tunnelSdkUri.template=https://ls-vvcenter.corp.otmltd.com:8089/sdk/vimService
vc.tunnelSdkUri=https://ls-vvcenter.corp.otmltd.com:8089/sdk/vimService
drs.demandCapacityRatio=100
(hostid was change to reflect our servers hostid)
Now we are getting another error in the /var/log/vmware/eam/wrapper.log
STATUS | wrapper | 2018/04/11 11:53:54 | Java Service Wrapper Professional Edition 64-bit 3.5.24
STATUS | wrapper | 2018/04/11 11:53:54 | Copyright (C) 1999-2014 Tanuki Software, Ltd. All Rights Reserved.
STATUS | wrapper | 2018/04/11 11:53:54 | http://wrapper.tanukisoftware.com
STATUS | wrapper | 2018/04/11 11:53:54 | Licensed to VMware Global, Inc. for VMware vCenter ESX Agent Manager
STATUS | wrapper | 2018/04/11 11:53:54 |
INFO | wrapper | 2018/04/11 11:53:54 | Event Command 'wrapper_start': Command line[0] : /usr/sbin/cloudvm-ram-size
INFO | wrapper | 2018/04/11 11:53:54 | Event Command 'wrapper_start': Command line[1] : -J
INFO | wrapper | 2018/04/11 11:53:54 | Event Command 'wrapper_start': Command line[2] : vmware-eam
INFO | wrapper | 2018/04/11 11:53:54 | Event Command 'wrapper_start': Command line[3] : -O
INFO | wrapper | 2018/04/11 11:53:54 | Event Command 'wrapper_start': Command line[4] : /etc/vmware/java/vmware-eam_jvm.conf
INFO | wrapper | 2018/04/11 11:53:54 | Event Command 'wrapper_start': Command launched (pid: 12891), blocking for up to 3600 seconds...
INFO | wrapper | 2018/04/11 11:53:54 | Event Command 'wrapper_start': Command completed with exit code: 0
INFO | wrapper | 2018/04/11 11:53:54 | Event Command 'wrapper_start': Command completed with exit code: 0 Continuing.
INFO | wrapper | 2018/04/11 11:53:54 | Event Command 'jvm_prelaunch': Command line[0] : /opt/vmware/bin/python
INFO | wrapper | 2018/04/11 11:53:54 | Event Command 'jvm_prelaunch': Command line[1] : ../wrapper/bin/update_pnid.py
INFO | wrapper | 2018/04/11 11:53:54 | Event Command 'jvm_prelaunch': Command line[2] :
INFO | wrapper | 2018/04/11 11:53:54 | Event Command 'jvm_prelaunch': Command line[3] :
INFO | wrapper | 2018/04/11 11:53:54 | Event Command 'jvm_prelaunch': Command line[4] :
INFO | wrapper | 2018/04/11 11:53:54 | Event Command 'jvm_prelaunch': Command launched (pid: 12894), blocking for up to 15 seconds...
INFO | wrapper | 2018/04/11 11:53:54 | Event Command 'jvm_prelaunch': Command completed with exit code: 0
INFO | wrapper | 2018/04/11 11:53:54 | Event Command 'jvm_prelaunch': Command completed with exit code: 0 Continuing.
STATUS | wrapper | 2018/04/11 11:53:54 | Launching a JVM...
INFO | jvm 1 | 2018/04/11 11:53:55 | Picked up JAVA_TOOL_OPTIONS: -Xms32M -Xmx128M
INFO | jvm 1 | 2018/04/11 11:53:55 | WrapperManager: Initializing...
INFO | jvm 1 | 2018/04/11 11:53:56 | WrapperSimpleApp:
INFO | jvm 1 | 2018/04/11 11:53:56 | WrapperSimpleApp Error: Encountered an error running main:
INFO | jvm 1 | 2018/04/11 11:53:56 | WrapperSimpleApp Error: org.springframework.beans.factory.BeanDefinitionStoreException: Invalid bean definition with name 'esxAgentMgr' defined in class path resource [eam-server-beans.xml]: Could not resolve placeholder 'vum.integration'
INFO | jvm 1 | 2018/04/11 11:53:56 | WrapperSimpleApp Error: at org.springframework.beans.factory.config.PropertyPlaceholderConfigurer.processProperties(PropertyPlaceholderConfigurer.java:268)
INFO | jvm 1 | 2018/04/11 11:53:56 | WrapperSimpleApp Error: at org.springframework.beans.factory.config.PropertyResourceConfigurer.postProcessBeanFactory(PropertyResourceConfigurer.java:75)
INFO | jvm 1 | 2018/04/11 11:53:56 | WrapperSimpleApp Error: at org.springframework.context.support.AbstractApplicationContext.invokeBeanFactoryPostProcessors(AbstractApplicationContext.java:554)
INFO | jvm 1 | 2018/04/11 11:53:56 | WrapperSimpleApp Error: at org.springframework.context.support.AbstractApplicationContext.invokeBeanFactoryPostProcessors(AbstractApplicationContext.java:528)
INFO | jvm 1 | 2018/04/11 11:53:56 | WrapperSimpleApp Error: at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:363)
INFO | jvm 1 | 2018/04/11 11:53:56 | WrapperSimpleApp Error: at org.springframework.context.support.ClassPathXmlApplicationContext.<init>(ClassPathXmlApplicationContext.java:139)
INFO | jvm 1 | 2018/04/11 11:53:56 | WrapperSimpleApp Error: at org.springframework.context.support.ClassPathXmlApplicationContext.<init>(ClassPathXmlApplicationContext.java:83)
INFO | jvm 1 | 2018/04/11 11:53:56 | WrapperSimpleApp Error: at com.vmware.eam.util.Main.main(Main.java:50)
INFO | jvm 1 | 2018/04/11 11:53:56 | WrapperSimpleApp Error: at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
INFO | jvm 1 | 2018/04/11 11:53:56 | WrapperSimpleApp Error: at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
INFO | jvm 1 | 2018/04/11 11:53:56 | WrapperSimpleApp Error: at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
INFO | jvm 1 | 2018/04/11 11:53:56 | WrapperSimpleApp Error: at java.lang.reflect.Method.invoke(Unknown Source)
INFO | jvm 1 | 2018/04/11 11:53:56 | WrapperSimpleApp Error: at org.tanukisoftware.wrapper.WrapperSimpleApp.run(WrapperSimpleApp.java:290)
INFO | jvm 1 | 2018/04/11 11:53:56 | WrapperSimpleApp Error: at java.lang.Thread.run(Unknown Source)
STATUS | wrapper | 2018/04/11 11:53:58 | <-- Wrapper Stopped
This has been ongoing for a couple of days now. Any Ideas?
It looks like the eam.properties file is incorrectly updated.
# SSO configuration
#
sso.wait.attempts=360
sso.wait.intervalSeconds=5
# VC SSL configuration
#
vc.truststore.type=VKS
vc.truststore.storename=TRUSTED_ROOTS
vc.tunnelSdkUri.template=https://##{VC_HOST_NAME}##:8089/sdk/vimService <====== Keep this unchanged
vc.tunnelSdkUri=<vCenter_FQDN>:8089/sdk/vimService <====== Change this to your vCenter FQDN/IP
drs.demandCapacityRatio=100
Thanks for the reply.
I have changed the template to read what your have suggested. (I was thinking whether or not to change that) but still get the same error. The vCenter FQDN is correct but changed it to the IP instead but I still get the same error: -
INFO | wrapper | 2018/04/11 14:29:52 | Event Command 'jvm_prelaunch': Command completed with exit code: 0
INFO | wrapper | 2018/04/11 14:29:52 | Event Command 'jvm_prelaunch': Command completed with exit code: 0 Continuing.
STATUS | wrapper | 2018/04/11 14:29:52 | Launching a JVM...
INFO | jvm 1 | 2018/04/11 14:29:52 | Picked up JAVA_TOOL_OPTIONS: -Xms32M -Xmx128M
INFO | jvm 1 | 2018/04/11 14:29:52 | WrapperManager: Initializing...
INFO | jvm 1 | 2018/04/11 14:29:53 | WrapperSimpleApp:
INFO | jvm 1 | 2018/04/11 14:29:53 | WrapperSimpleApp Error: Encountered an error running main:
INFO | jvm 1 | 2018/04/11 14:29:53 | WrapperSimpleApp Error: org.springframework.beans.factory.BeanDefinitionStoreException: Invalid bean definition with name 'esxAgentMgr' defined in class path resource [eam-server-beans.xml]: Could not resolve placeholder 'vum.integration'
The eam.properties file which you have created has a lot of missing objects.
Run this command ti get the host ID:
cat /etc/vmware/install-defaults/sca.hostid
Actual content for eam.properties file:
#########################################################################
# Copyright 2013-2014 VMware, Inc. All rights reserved. VMware Confidential
#########################################################################
vc.proxy.host=localhost
vc.proxy.port=80
# Hostname or IP of the EAM server
# Fill only if EAM is not running on the same host as VC
eam.host=
# EAM service port used to configure the HTTP connector of the application server.
eam.int.http.port=15005
# Port and scheme configuration which is used by the ESX 6.x hosts to reach EAM Vib
# file server.
eam.ext.port=443
eam.ext.scheme=https
# Port and scheme configuration which is used by the ESX 5.x hosts to reach EAM Vib
# file server.
eam.ext.port.deprecated=80
eam.ext.scheme.deprecated=http
eam.support_linked_clone=true
eam.clear_db_on_startup=false
eam.debug_ref_count=false
eam.recent_event_size=20
# Enable/disable VUM integration
vum.integration=true
# Value is specified in minutes (set to 24h = 1440m)
eam.scan_for_unknown_agent_vms=1440
# The timeout to wait for hostd to restart on a host (set to 5m=300s)
eam.hostd_restart_timeout=300
# The following entries will be added verbatim to the advanced options
# of hosts on which EAM is enabled. All are optional.
Net.DVFilterBindIpAddress=169.254.0.1
Net.TrafficFilterIpAddress=
#The IP for the VSWIF NIC on the dvFilter switch (for ESX classic).
Net.DVFilterVswifIpAddress=169.254.0.2
# Resource bundle configuration
eam.resourcebundle.filename=eam-resourcebundle.jar
# VLSI embedded tcServer configuration
#
tcserver.tmp.dir=/var/tmp/vmware/eam/tomcat
# EAM SSL configuration
#
eam.keystore.type=VKS
eam.key.alias=vpxd-extension
eam.keystore.storename=vpxd-extension
# CM configuration
#
cm.url=http://localhost:18090/cm/sdk/?hostid=0a12dfe2-b3b7-4eb6-9737-1ec5a8e770ec <====== same string from command sca.hostid
cm.wait.attempts=360
cm.wait.intervalSeconds=5
# SSO configuration
#
sso.wait.attempts=360
sso.wait.intervalSeconds=5
# VC SSL configuration
#
vc.truststore.type=VKS
vc.truststore.storename=TRUSTED_ROOTS
vc.tunnelSdkUri.template=https://##{VC_HOST_NAME}##:8089/sdk/vimService <====== Keep this unchanged
vc.tunnelSdkUri=<vCenter_FQDN>:8089/sdk/vimService <====== Change this to your vCenter FQDN/IP
drs.demandCapacityRatio=100
====================================
If you have created a new file for eam.properties.
run the command ls -ltrh check the permissions.
Services should not be registered to root for this file.
If the file was copied from the existing eam.properties file.
chmod 0644 eam.properties
Thanks,
The hostid is correct in my eam.properties file. What I have noticed is comparing your copy of the eam.properties to the one I posted shows that vum.integration=true was missing from mine and also a copy/paste error eameam.resourcebundle.filename=eam-resourcebundle.jar, to many eam's.
I corrected these 2 problems and ran the service. I now receive a different error: -
STATUS | wrapper | 2018/04/11 15:45:40 | Java Service Wrapper Professional Edition 64-bit 3.5.24
STATUS | wrapper | 2018/04/11 15:45:40 | Copyright (C) 1999-2014 Tanuki Software, Ltd. All Rights Reserved.
STATUS | wrapper | 2018/04/11 15:45:40 | http://wrapper.tanukisoftware.com
STATUS | wrapper | 2018/04/11 15:45:40 | Licensed to VMware Global, Inc. for VMware vCenter ESX Agent Manager
STATUS | wrapper | 2018/04/11 15:45:40 |
INFO | wrapper | 2018/04/11 15:45:40 | Event Command 'wrapper_start': Command line[0] : /usr/sbin/cloudvm-ram-size
INFO | wrapper | 2018/04/11 15:45:40 | Event Command 'wrapper_start': Command line[1] : -J
INFO | wrapper | 2018/04/11 15:45:40 | Event Command 'wrapper_start': Command line[2] : vmware-eam
INFO | wrapper | 2018/04/11 15:45:40 | Event Command 'wrapper_start': Command line[3] : -O
INFO | wrapper | 2018/04/11 15:45:40 | Event Command 'wrapper_start': Command line[4] : /etc/vmware/java/vmware-eam_jvm.conf
INFO | wrapper | 2018/04/11 15:45:40 | Event Command 'wrapper_start': Command launched (pid: 18533), blocking for up to 3600 seconds...
INFO | wrapper | 2018/04/11 15:45:40 | Event Command 'wrapper_start': Command completed with exit code: 0
INFO | wrapper | 2018/04/11 15:45:40 | Event Command 'wrapper_start': Command completed with exit code: 0 Continuing.
INFO | wrapper | 2018/04/11 15:45:40 | Event Command 'jvm_prelaunch': Command line[0] : /opt/vmware/bin/python
INFO | wrapper | 2018/04/11 15:45:40 | Event Command 'jvm_prelaunch': Command line[1] : ../wrapper/bin/update_pnid.py
INFO | wrapper | 2018/04/11 15:45:40 | Event Command 'jvm_prelaunch': Command line[2] :
INFO | wrapper | 2018/04/11 15:45:40 | Event Command 'jvm_prelaunch': Command line[3] :
INFO | wrapper | 2018/04/11 15:45:40 | Event Command 'jvm_prelaunch': Command line[4] :
INFO | wrapper | 2018/04/11 15:45:40 | Event Command 'jvm_prelaunch': Command launched (pid: 18536), blocking for up to 15 seconds...
INFO | wrapper | 2018/04/11 15:45:40 | Event Command 'jvm_prelaunch': Command completed with exit code: 0
INFO | wrapper | 2018/04/11 15:45:40 | Event Command 'jvm_prelaunch': Command completed with exit code: 0 Continuing.
STATUS | wrapper | 2018/04/11 15:45:40 | Launching a JVM...
INFO | jvm 1 | 2018/04/11 15:45:40 | Picked up JAVA_TOOL_OPTIONS: -Xms32M -Xmx128M
INFO | jvm 1 | 2018/04/11 15:45:41 | WrapperManager: Initializing...
INFO | jvm 1 | 2018/04/11 15:45:42 | WrapperSimpleApp:
INFO | jvm 1 | 2018/04/11 15:45:42 | WrapperSimpleApp Error: Encountered an error running main:
INFO | jvm 1 | 2018/04/11 15:45:42 | WrapperSimpleApp Error: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'stsCertManager' defined in class path resource [eam-server-beans.xml]: Cannot resolve reference to bean 'cmClient' while setting constructor argument; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'cmClient' defined in class path resource [eam-server-beans.xml]: Cannot resolve reference to bean 'vcTrustStore' while setting constructor argument; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'vcTrustStore' defined in class path resource [eam-server-beans.xml]: Instantiation of bean failed; nested exception is org.springframework.beans.factory.BeanDefinitionStoreException: Factory method [public static java.security.KeyStore com.vmware.eam.identity.KeyStoreFactory.createKeyStore(java.lang.String,java.io.File,java.lang.String,java.lang.String)] threw exception; nested exception is com.vmware.eam.EamException: Unsupported keystore type: VKS
INFO | jvm 1 | 2018/04/11 15:45:42 | WrapperSimpleApp Error: at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveReference(BeanDefinitionValueResolver.java:275)
INFO | jvm 1 | 2018/04/11 15:45:42 | WrapperSimpleApp Error: at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveValueIfNecessary(BeanDefinitionValueResolver.java:104)
INFO | jvm 1 | 2018/04/11 15:45:42 | WrapperSimpleApp Error: at org.springframework.beans.factory.support.ConstructorResolver.resolveConstructorArguments(ConstructorResolver.java:474)
INFO | jvm 1 | 2018/04/11 15:45:42 | WrapperSimpleApp Error: at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:342)
INFO | jvm 1 | 2018/04/11 15:45:42 | WrapperSimpleApp Error: at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateUsingFactoryMethod(AbstractAutowireCapableBeanFactory.java:901)
INFO | jvm 1 | 2018/04/11 15:45:42 | WrapperSimpleApp Error: at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:815)
INFO | jvm 1 | 2018/04/11 15:45:42 | WrapperSimpleApp Error: at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:440)
INFO | jvm 1 | 2018/04/11 15:45:42 | WrapperSimpleApp Error: at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory$1.run(AbstractAutowireCapableBeanFactory.java:409)
INFO | jvm 1 | 2018/04/11 15:45:42 | WrapperSimpleApp Error: at java.security.AccessController.doPrivileged(Native Method)
INFO | jvm 1 | 2018/04/11 15:45:42 | WrapperSimpleApp Error: at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:380)
INFO | jvm 1 | 2018/04/11 15:45:42 | WrapperSimpleApp Error: at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:264)
INFO | jvm 1 | 2018/04/11 15:45:42 | WrapperSimpleApp Error: at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:220)
INFO | jvm 1 | 2018/04/11 15:45:42 | WrapperSimpleApp Error: at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:261)
INFO | jvm 1 | 2018/04/11 15:45:42 | WrapperSimpleApp Error: at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:185)
INFO | jvm 1 | 2018/04/11 15:45:42 | WrapperSimpleApp Error: at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:164)
INFO | jvm 1 | 2018/04/11 15:45:42 | WrapperSimpleApp Error: at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:254)
INFO | jvm 1 | 2018/04/11 15:45:42 | WrapperSimpleApp Error: at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:185)
INFO | jvm 1 | 2018/04/11 15:45:42 | WrapperSimpleApp Error: at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:164)
INFO | jvm 1 | 2018/04/11 15:45:42 | WrapperSimpleApp Error: at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:254)
INFO | jvm 1 | 2018/04/11 15:45:42 | WrapperSimpleApp Error: at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:185)
INFO | jvm 1 | 2018/04/11 15:45:42 | WrapperSimpleApp Error: at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:164)
INFO | jvm 1 | 2018/04/11 15:45:42 | WrapperSimpleApp Error: at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:254)
INFO | jvm 1 | 2018/04/11 15:45:42 | WrapperSimpleApp Error: at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:185)
INFO | jvm 1 | 2018/04/11 15:45:42 | WrapperSimpleApp Error: at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:164)
INFO | jvm 1 | 2018/04/11 15:45:42 | WrapperSimpleApp Error: at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:429)
INFO | jvm 1 | 2018/04/11 15:45:42 | WrapperSimpleApp Error: at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:729)
INFO | jvm 1 | 2018/04/11 15:45:42 | WrapperSimpleApp Error: at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:381)
INFO | jvm 1 | 2018/04/11 15:45:42 | WrapperSimpleApp Error: at org.springframework.context.support.ClassPathXmlApplicationContext.<init>(ClassPathXmlApplicationContext.java:139)
INFO | jvm 1 | 2018/04/11 15:45:42 | WrapperSimpleApp Error: at org.springframework.context.support.ClassPathXmlApplicationContext.<init>(ClassPathXmlApplicationContext.java:83)
INFO | jvm 1 | 2018/04/11 15:45:42 | WrapperSimpleApp Error: at com.vmware.eam.util.Main.main(Main.java:50)
INFO | jvm 1 | 2018/04/11 15:45:42 | WrapperSimpleApp Error: at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
INFO | jvm 1 | 2018/04/11 15:45:42 | WrapperSimpleApp Error: at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
INFO | jvm 1 | 2018/04/11 15:45:42 | WrapperSimpleApp Error: at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
INFO | jvm 1 | 2018/04/11 15:45:42 | WrapperSimpleApp Error: at java.lang.reflect.Method.invoke(Unknown Source)
INFO | jvm 1 | 2018/04/11 15:45:42 | WrapperSimpleApp Error: at org.tanukisoftware.wrapper.WrapperSimpleApp.run(WrapperSimpleApp.java:290)
INFO | jvm 1 | 2018/04/11 15:45:42 | WrapperSimpleApp Error: at java.lang.Thread.run(Unknown Source)
INFO | jvm 1 | 2018/04/11 15:45:42 | WrapperSimpleApp Error: Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'cmClient' defined in class path resource [eam-server-beans.xml]: Cannot resolve reference to bean 'vcTrustStore' while setting constructor argument; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'vcTrustStore' defined in class path resource [eam-server-beans.xml]: Instantiation of bean failed; nested exception is org.springframework.beans.factory.BeanDefinitionStoreException: Factory method [public static java.security.KeyStore com.vmware.eam.identity.KeyStoreFactory.createKeyStore(java.lang.String,java.io.File,java.lang.String,java.lang.String)] threw exception; nested exception is com.vmware.eam.EamException: Unsupported keystore type: VKS
INFO | jvm 1 | 2018/04/11 15:45:42 | WrapperSimpleApp Error: at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveReference(BeanDefinitionValueResolver.java:275)
INFO | jvm 1 | 2018/04/11 15:45:42 | WrapperSimpleApp Error: at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveValueIfNecessary(BeanDefinitionValueResolver.java:104)
INFO | jvm 1 | 2018/04/11 15:45:42 | WrapperSimpleApp Error: at org.springframework.beans.factory.support.ConstructorResolver.resolveConstructorArguments(ConstructorResolver.java:474)
INFO | jvm 1 | 2018/04/11 15:45:42 | WrapperSimpleApp Error: at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:342)
INFO | jvm 1 | 2018/04/11 15:45:42 | WrapperSimpleApp Error: at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateUsingFactoryMethod(AbstractAutowireCapableBeanFactory.java:901)
INFO | jvm 1 | 2018/04/11 15:45:42 | WrapperSimpleApp Error: at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:815)
INFO | jvm 1 | 2018/04/11 15:45:42 | WrapperSimpleApp Error: at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:440)
INFO | jvm 1 | 2018/04/11 15:45:42 | WrapperSimpleApp Error: at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory$1.run(AbstractAutowireCapableBeanFactory.java:409)
INFO | jvm 1 | 2018/04/11 15:45:42 | WrapperSimpleApp Error: at java.security.AccessController.doPrivileged(Native Method)
INFO | jvm 1 | 2018/04/11 15:45:42 | WrapperSimpleApp Error: at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:380)
INFO | jvm 1 | 2018/04/11 15:45:42 | WrapperSimpleApp Error: at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:264)
INFO | jvm 1 | 2018/04/11 15:45:42 | WrapperSimpleApp Error: at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:220)
INFO | jvm 1 | 2018/04/11 15:45:42 | WrapperSimpleApp Error: at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:261)
INFO | jvm 1 | 2018/04/11 15:45:42 | WrapperSimpleApp Error: at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:185)
INFO | jvm 1 | 2018/04/11 15:45:42 | WrapperSimpleApp Error: at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:164)
INFO | jvm 1 | 2018/04/11 15:45:42 | WrapperSimpleApp Error: at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveReference(BeanDefinitionValueResolver.java:269)
INFO | jvm 1 | 2018/04/11 15:45:42 | WrapperSimpleApp Error: ... 35 more
INFO | jvm 1 | 2018/04/11 15:45:42 | WrapperSimpleApp Error: Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'vcTrustStore' defined in class path resource [eam-server-beans.xml]: Instantiation of bean failed; nested exception is org.springframework.beans.factory.BeanDefinitionStoreException: Factory method [public static java.security.KeyStore com.vmware.eam.identity.KeyStoreFactory.createKeyStore(java.lang.String,java.io.File,java.lang.String,java.lang.String)] threw exception; nested exception is com.vmware.eam.EamException: Unsupported keystore type: VKS
INFO | jvm 1 | 2018/04/11 15:45:42 | WrapperSimpleApp Error: at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:423)
INFO | jvm 1 | 2018/04/11 15:45:42 | WrapperSimpleApp Error: at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateUsingFactoryMethod(AbstractAutowireCapableBeanFactory.java:901)
INFO | jvm 1 | 2018/04/11 15:45:42 | WrapperSimpleApp Error: at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:815)
INFO | jvm 1 | 2018/04/11 15:45:42 | WrapperSimpleApp Error: at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:440)
INFO | jvm 1 | 2018/04/11 15:45:42 | WrapperSimpleApp Error: at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory$1.run(AbstractAutowireCapableBeanFactory.java:409)
INFO | jvm 1 | 2018/04/11 15:45:42 | WrapperSimpleApp Error: at java.security.AccessController.doPrivileged(Native Method)
INFO | jvm 1 | 2018/04/11 15:45:42 | WrapperSimpleApp Error: at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:380)
INFO | jvm 1 | 2018/04/11 15:45:42 | WrapperSimpleApp Error: at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:264)
INFO | jvm 1 | 2018/04/11 15:45:42 | WrapperSimpleApp Error: at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:220)
INFO | jvm 1 | 2018/04/11 15:45:42 | WrapperSimpleApp Error: at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:261)
INFO | jvm 1 | 2018/04/11 15:45:42 | WrapperSimpleApp Error: at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:185)
INFO | jvm 1 | 2018/04/11 15:45:42 | WrapperSimpleApp Error: at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:164)
INFO | jvm 1 | 2018/04/11 15:45:42 | WrapperSimpleApp Error: at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveReference(BeanDefinitionValueResolver.java:269)
INFO | jvm 1 | 2018/04/11 15:45:42 | WrapperSimpleApp Error: ... 50 more
INFO | jvm 1 | 2018/04/11 15:45:42 | WrapperSimpleApp Error: Caused by: org.springframework.beans.factory.BeanDefinitionStoreException: Factory method [public static java.security.KeyStore com.vmware.eam.identity.KeyStoreFactory.createKeyStore(java.lang.String,java.io.File,java.lang.String,java.lang.String)] threw exception; nested exception is com.vmware.eam.EamException: Unsupported keystore type: VKS
INFO | jvm 1 | 2018/04/11 15:45:42 | WrapperSimpleApp Error: at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:127)
INFO | jvm 1 | 2018/04/11 15:45:42 | WrapperSimpleApp Error: at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:414)
INFO | jvm 1 | 2018/04/11 15:45:42 | WrapperSimpleApp Error: ... 62 more
INFO | jvm 1 | 2018/04/11 15:45:42 | WrapperSimpleApp Error: Caused by: com.vmware.eam.EamException: Unsupported keystore type: VKS
INFO | jvm 1 | 2018/04/11 15:45:42 | WrapperSimpleApp Error: at com.vmware.eam.identity.KeyStoreFactory.createKeyStore(KeyStoreFactory.java:48)
INFO | jvm 1 | 2018/04/11 15:45:42 | WrapperSimpleApp Error: at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
INFO | jvm 1 | 2018/04/11 15:45:42 | WrapperSimpleApp Error: at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
INFO | jvm 1 | 2018/04/11 15:45:42 | WrapperSimpleApp Error: at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
INFO | jvm 1 | 2018/04/11 15:45:42 | WrapperSimpleApp Error: at java.lang.reflect.Method.invoke(Unknown Source)
INFO | jvm 1 | 2018/04/11 15:45:42 | WrapperSimpleApp Error: at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:115)
INFO | jvm 1 | 2018/04/11 15:45:42 | WrapperSimpleApp Error: ... 63 more
STATUS | wrapper | 2018/04/11 15:45:44 | <-- Wrapper Stopped
nearly forgot....
ls -ltrh eam.properties
-rw-r--r-- 1 eam cis 2.3K Apr 11 15:45 eam.properties
I never created the file, I edited the blank one.
Could you please check the logs under:
/var/log/vmware/invsvc/inv-svc.log
/var/log/vmware/vmdird/vmdird-syslog.log
Also check if you have sufficient space.
Run the command df -h
Also check the logs for component manager.
Volume info: -
Filesystem Size Used Avail Use% Mounted on
/dev/sda3 21G 5.2G 15G 27% /
udev 7.9G 164K 7.9G 1% /dev
tmpfs 7.9G 40K 7.9G 1% /dev/shm
/dev/sda1 128M 41M 81M 34% /boot
/dev/mapper/core_vg-core 50G 13G 35G 27% /storage/core
/dev/mapper/log_vg-log 25G 12G 12G 50% /storage/log
/dev/mapper/db_vg-db 18G 234M 17G 2% /storage/db
/dev/mapper/dblog_vg-dblog 5.0G 219M 4.5G 5% /storage/dblog
/dev/mapper/seat_vg-seat 25G 463M 23G 2% /storage/seat
/dev/mapper/netdump_vg-netdump 1001M 18M 932M 2% /storage/netdump
/dev/mapper/autodeploy_vg-autodeploy 11G 154M 11G 2% /storage/autodeploy
/dev/mapper/invsvc_vg-invsvc 11G 181M 11G 2% /storage/invsvc
tmpfs 1.0M 0 1.0M 0% /var/spool/snmp
/var/log/vmware/invsvc/inv-svc.log contains INFO lines, no errors or debug lines.
/var/log/vmware/vmdird/vmdird-syslog.log
2018-04-12T02:37:20.227668+00:00 warning vmdird t@139624981378816: ScanIndex: non-indexed attribute. attrType = vmwLicSvcObjectClass
2018-04-12T02:43:20.223385+00:00 warning vmdird t@139624981378816: ScanIndex: non-indexed attribute. attrType = vmwLicSvcObjectClass
2018-04-12T02:56:16.043312+00:00 info vmdird t@139624981378816: Modify Entry (cn=ls-vvcenter.corp.otmltd.com,ou=Domain Controllers,dc=vsphere,dc=local)
2018-04-12T02:56:16.046861+00:00 info vmdird t@139624981378816: Modify Entry (cn=ls-vvcenter.corp.otmltd.com,ou=Domain Controllers,dc=vsphere,dc=local)
2018-04-12T02:56:16.046952+00:00 info vmdird t@139624981378816: User account control - (cn=ls-vvcenter.corp.otmltd.com,ou=domain controllers,dc=vsphere,dc=local): (800010) flag unset, new value=(0)
2018-04-12T02:56:16.046988+00:00 info vmdird t@139624981378816: Password Modification Successful ([23] 192.168.100.5:389<-192.168.100.5:53844). Bind DN: "cn=ls-vvcenter.corp.otmltd.com,ou=domain controllers,dc=vsphere,dc=local". Modified DN: "cn=ls-vvcenter.corp.otmltd.com,ou=Domain Controllers,dc=vsphere,dc=local"
2018-04-12T02:57:20.225419+00:00 warning vmdird t@139624981378816: ScanIndex: non-indexed attribute. attrType = vmwLicSvcObjectClass
2018-04-12T02:57:20.293456+00:00 info vmdird t@139624989771520: Bind failed () (9234)
2018-04-12T02:57:20.293500+00:00 err vmdird t@139624989771520: VmDirSendLdapResult: Request (96), Error (49), Message (), (0) socket ([33] 127.0.0.1:389<-127.0.0.1:50786)
2018-04-12T02:57:20.293510+00:00 err vmdird t@139624989771520: Bind Request Failed ([33] 127.0.0.1:389<-127.0.0.1:50786) error 49: Protocol version: 3, Bind DN: "cn=ls-vvcenter.corp.otmltd.com,ou=Domain Controllers,dc=vsphere,dc=local", Method: 128
2018-04-12T02:58:34.129236+00:00 warning vmdird t@139624981378816: ScanIndex: non-indexed attribute. attrType = vmwLicSvcObjectClass
2018-04-12T02:58:34.376989+00:00 warning vmdird t@139624981378816: ScanIndex: non-indexed attribute. attrType = vmwLicSvcObjectClass
2018-04-12T03:17:20.227219+00:00 warning vmdird t@139624989771520: ScanIndex: non-indexed attribute. attrType = vmwLicSvcObjectClass
2018-04-12T03:23:20.221835+00:00 warning vmdird t@139624989771520: ScanIndex: non-indexed attribute. attrType = vmwLicSvcObjectClass
/var/log/vmware/cm/cm.log
2018-04-12T09:08:38.899Z [pool-14-thread-1 DEBUG com.vmware.vim.vmomi.client.http.impl.ThumbprintTrustManager (299c822b-9337-49f3-a8a3-855ae3ac21dd)] Server certificate chain is trusted
2018-04-12T09:08:38.899Z [pool-14-thread-1 DEBUG com.vmware.vim.vmomi.client.http.impl.ThumbprintTrustManager (299c822b-9337-49f3-a8a3-855ae3ac21dd)] Server certificate chain is trusted
2018-04-12T09:08:38.899Z [pool-14-thread-1 DEBUG com.vmware.vim.vmomi.client.http.impl.ThumbprintTrustManager (299c822b-9337-49f3-a8a3-855ae3ac21dd)] Server certificate chain is trusted
2018-04-12T09:08:38.939Z [pool-14-thread-1 DEBUG com.vmware.vim.vmomi.client.http.impl.ThumbprintTrustManager (299c822b-9337-49f3-a8a3-855ae3ac21dd)] Server certificate assertion verified
2018-04-12T09:08:38.939Z [pool-14-thread-1 DEBUG com.vmware.vim.vmomi.client.http.impl.ThumbprintTrustManager (299c822b-9337-49f3-a8a3-855ae3ac21dd)] Server certificate assertion verified
2018-04-12T09:08:38.939Z [pool-14-thread-1 DEBUG com.vmware.vim.vmomi.client.http.impl.ThumbprintTrustManager (299c822b-9337-49f3-a8a3-855ae3ac21dd)] Server certificate assertion verified
2018-04-12T09:11:19.759Z [pool-1-thread-1 INFO com.vmware.cis.services.common.sso.LoginHelper (41e192f1-b4a7-4062-ab61-90b65c5715ef)] loginByToken: User 'vsphere-webclient-4f1feb00-5481-47e8-a7c3-6808106d7fdb@vsphere.local' is already authenticated
2018-04-12T09:11:19.770Z [pool-14-thread-1 DEBUG com.vmware.vim.vmomi.client.http.impl.ThumbprintTrustManager (299c822b-9337-49f3-a8a3-855ae3ac21dd)] Server certificate chain is trusted
2018-04-12T09:11:19.770Z [pool-14-thread-1 DEBUG com.vmware.vim.vmomi.client.http.impl.ThumbprintTrustManager (299c822b-9337-49f3-a8a3-855ae3ac21dd)] Server certificate chain is trusted
2018-04-12T09:11:19.770Z [pool-14-thread-1 DEBUG com.vmware.vim.vmomi.client.http.impl.ThumbprintTrustManager (299c822b-9337-49f3-a8a3-855ae3ac21dd)] Server certificate chain is trusted
2018-04-12T09:11:19.811Z [pool-14-thread-1 DEBUG com.vmware.vim.vmomi.client.http.impl.ThumbprintTrustManager (299c822b-9337-49f3-a8a3-855ae3ac21dd)] Server certificate assertion verified
2018-04-12T09:11:19.811Z [pool-14-thread-1 DEBUG com.vmware.vim.vmomi.client.http.impl.ThumbprintTrustManager (299c822b-9337-49f3-a8a3-855ae3ac21dd)] Server certificate assertion verified
2018-04-12T09:11:19.811Z [pool-14-thread-1 DEBUG com.vmware.vim.vmomi.client.http.impl.ThumbprintTrustManager (299c822b-9337-49f3-a8a3-855ae3ac21dd)] Server certificate assertion verified
2018-04-12T09:11:20.190Z [pool-1-thread-1 INFO com.vmware.cis.services.common.sso.LoginHelper (6e8e7761-33ce-4d7a-a940-9cfc308a36ca)] loginByToken: User 'vsphere-webclient-4f1feb00-5481-47e8-a7c3-6808106d7fdb@vsphere.local' is already authenticated
Thank you for sharing the details. As per the errror observed, You would have to reset the password for the account:
Bind DN: "cn=ls-vvcenter.corp.otmltd.com,ou=Domain Controllers,dc=vsphere,dc=local"
You could follow the steps mentioned below in the article to reset the password.
Keep generating vdcadmintool password until the password does not contain * @ , \ " or space.
