VMware Cloud Community
ITDeptVM
Contributor
Contributor

VCenter Appliance - Esx Agent Manager fails to start

First noticed we have a problem when we experienced 503 errors accessing the web UI on our vCenter v6 box.  After further investigations I found that the / volume was at 100% percent thanks to the root password expiring and writing to the audit.log.  Resolved that problem by following a thread on increasing the / volume size and fixing the root password.  We then had a problem with starting the inventory service, getting a LDAP 49 error which we resolved by following another thread on changing the account password using /usr/lib/vmware-vmdir/bin/vdcadmintool

With the Inventory service running, we now have the Agent Manager problem.

/var/log/vmware/eam/wrapper.log

INFO   | wrapper  | 2018/04/11 09:36:19 | Event Command 'jvm_prelaunch': Command launched (pid: 17813), blocking for up to 15 seconds...

INFO   | wrapper  | 2018/04/11 09:36:20 | Event Command 'jvm_prelaunch': Command completed with exit code: 1

INFO   | wrapper  | 2018/04/11 09:36:20 | Event Command 'jvm_prelaunch': Command completed with exit code: 1  Continuing.

STATUS | wrapper  | 2018/04/11 09:36:20 | Launching a JVM...

INFO   | jvm 1    | 2018/04/11 09:36:20 | Picked up JAVA_TOOL_OPTIONS: -Xms32M -Xmx128M

INFO   | jvm 1    | 2018/04/11 09:36:20 | WrapperManager: Initializing...

INFO   | jvm 1    | 2018/04/11 09:36:21 | WrapperSimpleApp:

INFO   | jvm 1    | 2018/04/11 09:36:21 | WrapperSimpleApp Error: Encountered an error running main:

INFO   | jvm 1    | 2018/04/11 09:36:21 | WrapperSimpleApp Error: org.springframework.beans.factory.BeanDefinitionStoreException: Invalid bean definition with name 'esxAgentMgr' defined in class path resource [eam-server-beans.xml]: Could not resolve placeholder 'eam.hostd_restart_timeout'

Searching the error message I found that our eam.properties file in /etc/vmware-eam was empty.  We edited the existing file and copied and pasted from the thread we found.

My eam.properties file: -

#########################################################################

# Copyright 2013-2017 VMware, Inc.  All rights reserved. VMware Confidential

#########################################################################

eam.web.root=/usr/lib/vmware-eam/web

vc.proxy.host=localhost

vc.proxy.port=80

# Hostname or IP of the EAM server

# Fill only if EAM is not running on the same host as VC

eam.host=

# EAM service port used to configure the HTTP connector of the application server.

eam.int.http.port=15005

# Port and scheme configuration which is used by the ESX 6.x hosts to reach EAM Vib

# file server.

eam.ext.port=443

eam.ext.scheme=https

# Port and scheme configuration which is used by the ESX 5.x hosts to reach EAM Vib

# file server.

eam.ext.port.deprecated=80

eam.ext.scheme.deprecated=http

eam.support_linked_clone=true

eam.clear_db_on_startup=false

eam.debug_ref_count=false

eam.recent_event_size=20

# Value is specified in minutes (set to 24h = 1440m)

eam.scan_for_unknown_agent_vms=1440

# The timeout to wait for hostd to restart on a host (set to 5m=300s)

eam.hostd_restart_timeout=300

# The following entries will be added verbatim to the advanced options

# of hosts on which EAM is enabled. All are optional.

Net.DVFilterBindIpAddress=169.254.0.1

Net.TrafficFilterIpAddress=

#The IP for the VSWIF NIC on the dvFilter switch (for ESX classic).

Net.DVFilterVswifIpAddress=169.254.0.2

# Resource bundle configuration

eameam.resourcebundle.filename=eam-resourcebundle.jar

# VLSI embedded tcServer configuration

#

tcserver.tmp.dir=/var/tmp/vmware/eam/tomcat

eam.web.root=/usr/lib/vmware-eam/web

# EAM SSL configuration

#

eam.keystore.type=VKS

eam.key.alias=vpxd-extension

eam.keystore.storename=vpxd-extension

# CM configuration

#

cm.url=http://localhost:18090/cm/sdk/?hostid=4f1feb00-5481-47e8-a7c3-6808106d7fdb4

cm.wait.attempts=360

cm.wait.intervalSeconds=5

# SSO configuration

#

sso.wait.attempts=360

sso.wait.intervalSeconds=5

# VC SSL configuration

#

vc.truststore.type=VKS

vc.truststore.storename=TRUSTED_ROOTS

vc.tunnelSdkUri.template=https://ls-vvcenter.corp.otmltd.com:8089/sdk/vimService

vc.tunnelSdkUri=https://ls-vvcenter.corp.otmltd.com:8089/sdk/vimService

drs.demandCapacityRatio=100

(hostid was change to reflect our servers hostid)

Now we are getting another error in the /var/log/vmware/eam/wrapper.log

STATUS | wrapper  | 2018/04/11 11:53:54 | Java Service Wrapper Professional Edition 64-bit 3.5.24

STATUS | wrapper  | 2018/04/11 11:53:54 |   Copyright (C) 1999-2014 Tanuki Software, Ltd. All Rights Reserved.

STATUS | wrapper  | 2018/04/11 11:53:54 |     http://wrapper.tanukisoftware.com

STATUS | wrapper  | 2018/04/11 11:53:54 |   Licensed to VMware Global, Inc. for VMware vCenter ESX Agent Manager

STATUS | wrapper  | 2018/04/11 11:53:54 |

INFO   | wrapper  | 2018/04/11 11:53:54 | Event Command 'wrapper_start': Command line[0] : /usr/sbin/cloudvm-ram-size

INFO   | wrapper  | 2018/04/11 11:53:54 | Event Command 'wrapper_start': Command line[1] : -J

INFO   | wrapper  | 2018/04/11 11:53:54 | Event Command 'wrapper_start': Command line[2] : vmware-eam

INFO   | wrapper  | 2018/04/11 11:53:54 | Event Command 'wrapper_start': Command line[3] : -O

INFO   | wrapper  | 2018/04/11 11:53:54 | Event Command 'wrapper_start': Command line[4] : /etc/vmware/java/vmware-eam_jvm.conf

INFO   | wrapper  | 2018/04/11 11:53:54 | Event Command 'wrapper_start': Command launched (pid: 12891), blocking for up to 3600 seconds...

INFO   | wrapper  | 2018/04/11 11:53:54 | Event Command 'wrapper_start': Command completed with exit code: 0

INFO   | wrapper  | 2018/04/11 11:53:54 | Event Command 'wrapper_start': Command completed with exit code: 0  Continuing.

INFO   | wrapper  | 2018/04/11 11:53:54 | Event Command 'jvm_prelaunch': Command line[0] : /opt/vmware/bin/python

INFO   | wrapper  | 2018/04/11 11:53:54 | Event Command 'jvm_prelaunch': Command line[1] : ../wrapper/bin/update_pnid.py

INFO   | wrapper  | 2018/04/11 11:53:54 | Event Command 'jvm_prelaunch': Command line[2] :

INFO   | wrapper  | 2018/04/11 11:53:54 | Event Command 'jvm_prelaunch': Command line[3] :

INFO   | wrapper  | 2018/04/11 11:53:54 | Event Command 'jvm_prelaunch': Command line[4] :

INFO   | wrapper  | 2018/04/11 11:53:54 | Event Command 'jvm_prelaunch': Command launched (pid: 12894), blocking for up to 15 seconds...

INFO   | wrapper  | 2018/04/11 11:53:54 | Event Command 'jvm_prelaunch': Command completed with exit code: 0

INFO   | wrapper  | 2018/04/11 11:53:54 | Event Command 'jvm_prelaunch': Command completed with exit code: 0  Continuing.

STATUS | wrapper  | 2018/04/11 11:53:54 | Launching a JVM...

INFO   | jvm 1    | 2018/04/11 11:53:55 | Picked up JAVA_TOOL_OPTIONS: -Xms32M -Xmx128M

INFO   | jvm 1    | 2018/04/11 11:53:55 | WrapperManager: Initializing...

INFO   | jvm 1    | 2018/04/11 11:53:56 | WrapperSimpleApp:

INFO   | jvm 1    | 2018/04/11 11:53:56 | WrapperSimpleApp Error: Encountered an error running main:

INFO   | jvm 1    | 2018/04/11 11:53:56 | WrapperSimpleApp Error: org.springframework.beans.factory.BeanDefinitionStoreException: Invalid bean definition with name 'esxAgentMgr' defined in class path resource [eam-server-beans.xml]: Could not resolve placeholder 'vum.integration'

INFO   | jvm 1    | 2018/04/11 11:53:56 | WrapperSimpleApp Error:       at org.springframework.beans.factory.config.PropertyPlaceholderConfigurer.processProperties(PropertyPlaceholderConfigurer.java:268)

INFO   | jvm 1    | 2018/04/11 11:53:56 | WrapperSimpleApp Error:       at org.springframework.beans.factory.config.PropertyResourceConfigurer.postProcessBeanFactory(PropertyResourceConfigurer.java:75)

INFO   | jvm 1    | 2018/04/11 11:53:56 | WrapperSimpleApp Error:       at org.springframework.context.support.AbstractApplicationContext.invokeBeanFactoryPostProcessors(AbstractApplicationContext.java:554)

INFO   | jvm 1    | 2018/04/11 11:53:56 | WrapperSimpleApp Error:       at org.springframework.context.support.AbstractApplicationContext.invokeBeanFactoryPostProcessors(AbstractApplicationContext.java:528)

INFO   | jvm 1    | 2018/04/11 11:53:56 | WrapperSimpleApp Error:       at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:363)

INFO   | jvm 1    | 2018/04/11 11:53:56 | WrapperSimpleApp Error:       at org.springframework.context.support.ClassPathXmlApplicationContext.<init>(ClassPathXmlApplicationContext.java:139)

INFO   | jvm 1    | 2018/04/11 11:53:56 | WrapperSimpleApp Error:       at org.springframework.context.support.ClassPathXmlApplicationContext.<init>(ClassPathXmlApplicationContext.java:83)

INFO   | jvm 1    | 2018/04/11 11:53:56 | WrapperSimpleApp Error:       at com.vmware.eam.util.Main.main(Main.java:50)

INFO   | jvm 1    | 2018/04/11 11:53:56 | WrapperSimpleApp Error:       at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

INFO   | jvm 1    | 2018/04/11 11:53:56 | WrapperSimpleApp Error:       at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)

INFO   | jvm 1    | 2018/04/11 11:53:56 | WrapperSimpleApp Error:       at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)

INFO   | jvm 1    | 2018/04/11 11:53:56 | WrapperSimpleApp Error:       at java.lang.reflect.Method.invoke(Unknown Source)

INFO   | jvm 1    | 2018/04/11 11:53:56 | WrapperSimpleApp Error:       at org.tanukisoftware.wrapper.WrapperSimpleApp.run(WrapperSimpleApp.java:290)

INFO   | jvm 1    | 2018/04/11 11:53:56 | WrapperSimpleApp Error:       at java.lang.Thread.run(Unknown Source)

STATUS | wrapper  | 2018/04/11 11:53:58 | <-- Wrapper Stopped

This has been ongoing for a couple of days now.  Any Ideas?

Reply
0 Kudos
8 Replies
ashwin_prakash
VMware Employee
VMware Employee

It looks like the eam.properties file is incorrectly updated.

# SSO configuration
#
sso.wait.attempts=360
sso.wait.intervalSeconds=5
# VC SSL configuration
#
vc.truststore.type=VKS
vc.truststore.storename=TRUSTED_ROOTS
vc.tunnelSdkUri.template=https://##{VC_HOST_NAME}##:8089/sdk/vimService
<====== Keep this unchanged

vc.tunnelSdkUri=<vCenter_FQDN>:8089/sdk/vimService <====== Change this to your vCenter FQDN/IP
drs.demandCapacityRatio=100

Sincerely,
Ashwin Prakash
Skyline Support Moderator
Reply
0 Kudos
ITDeptVM
Contributor
Contributor

Thanks for the reply.

I have changed the template to read what your have suggested.  (I was thinking whether or not to change that) but still get the same error. The vCenter FQDN is correct but changed it to the IP instead but I still get the same error: -

INFO   | wrapper  | 2018/04/11 14:29:52 | Event Command 'jvm_prelaunch': Command completed with exit code: 0

INFO   | wrapper  | 2018/04/11 14:29:52 | Event Command 'jvm_prelaunch': Command completed with exit code: 0  Continuing.

STATUS | wrapper  | 2018/04/11 14:29:52 | Launching a JVM...

INFO   | jvm 1    | 2018/04/11 14:29:52 | Picked up JAVA_TOOL_OPTIONS: -Xms32M -Xmx128M

INFO   | jvm 1    | 2018/04/11 14:29:52 | WrapperManager: Initializing...

INFO   | jvm 1    | 2018/04/11 14:29:53 | WrapperSimpleApp:

INFO   | jvm 1    | 2018/04/11 14:29:53 | WrapperSimpleApp Error: Encountered an error running main:

INFO   | jvm 1    | 2018/04/11 14:29:53 | WrapperSimpleApp Error: org.springframework.beans.factory.BeanDefinitionStoreException: Invalid bean definition with name 'esxAgentMgr' defined in class path resource [eam-server-beans.xml]: Could not resolve placeholder 'vum.integration'

Reply
0 Kudos
ashwin_prakash
VMware Employee
VMware Employee

The eam.properties file which you have created has a lot of missing objects.

Run this command ti get the host ID:

cat /etc/vmware/install-defaults/sca.hostid

Actual content for eam.properties file:

#########################################################################
# Copyright 2013-2014 VMware, Inc. All rights reserved. VMware Confidential
#########################################################################
vc.proxy.host=localhost
vc.proxy.port=80
# Hostname or IP of the EAM server
# Fill only if EAM is not running on the same host as VC
eam.host=
# EAM service port used to configure the HTTP connector of the application server.
eam.int.http.port=15005
# Port and scheme configuration which is used by the ESX 6.x hosts to reach EAM Vib
# file server.
eam.ext.port=443
eam.ext.scheme=https
# Port and scheme configuration which is used by the ESX 5.x hosts to reach EAM Vib
# file server.
eam.ext.port.deprecated=80
eam.ext.scheme.deprecated=http
eam.support_linked_clone=true
eam.clear_db_on_startup=false
eam.debug_ref_count=false
eam.recent_event_size=20
# Enable/disable VUM integration
vum.integration=true
# Value is specified in minutes (set to 24h = 1440m)
eam.scan_for_unknown_agent_vms=1440
# The timeout to wait for hostd to restart on a host (set to 5m=300s)
eam.hostd_restart_timeout=300
# The following entries will be added verbatim to the advanced options
# of hosts on which EAM is enabled. All are optional.
Net.DVFilterBindIpAddress=169.254.0.1
Net.TrafficFilterIpAddress=
#The IP for the VSWIF NIC on the dvFilter switch (for ESX classic).
Net.DVFilterVswifIpAddress=169.254.0.2
# Resource bundle configuration
eam.resourcebundle.filename=eam-resourcebundle.jar
# VLSI embedded tcServer configuration
#
tcserver.tmp.dir=/var/tmp/vmware/eam/tomcat
# EAM SSL configuration
#
eam.keystore.type=VKS
eam.key.alias=vpxd-extension
eam.keystore.storename=vpxd-extension
# CM configuration
#
cm.url=http://localhost:18090/cm/sdk/?hostid=0a12dfe2-b3b7-4eb6-9737-1ec5a8e770ec
<====== same string from command sca.hostid
cm.wait.attempts=360
cm.wait.intervalSeconds=5
# SSO configuration
#
sso.wait.attempts=360
sso.wait.intervalSeconds=5
# VC SSL configuration
#
vc.truststore.type=VKS
vc.truststore.storename=TRUSTED_ROOTS
vc.tunnelSdkUri.template=https://##{VC_HOST_NAME}##:8089/sdk/vimService
<====== Keep this unchanged

vc.tunnelSdkUri=<vCenter_FQDN>:8089/sdk/vimService <====== Change this to your vCenter FQDN/IP
drs.demandCapacityRatio=100

====================================

If you have created a new file for eam.properties.

run the command ls -ltrh check the permissions.

Services should not be registered to root for this file.

If the file was copied from the existing eam.properties file.

chmod 0644 eam.properties

Sincerely,
Ashwin Prakash
Skyline Support Moderator
Reply
0 Kudos
ITDeptVM
Contributor
Contributor

Thanks,

The hostid is correct in my eam.properties file.  What I have noticed is comparing your copy of the eam.properties to the one I posted shows that vum.integration=true was missing from mine and also a copy/paste error eameam.resourcebundle.filename=eam-resourcebundle.jar, to many eam's.

I corrected these 2 problems and ran the service.  I now receive a different error: -

STATUS | wrapper  | 2018/04/11 15:45:40 | Java Service Wrapper Professional Edition 64-bit 3.5.24

STATUS | wrapper  | 2018/04/11 15:45:40 |   Copyright (C) 1999-2014 Tanuki Software, Ltd. All Rights Reserved.

STATUS | wrapper  | 2018/04/11 15:45:40 |     http://wrapper.tanukisoftware.com

STATUS | wrapper  | 2018/04/11 15:45:40 |   Licensed to VMware Global, Inc. for VMware vCenter ESX Agent Manager

STATUS | wrapper  | 2018/04/11 15:45:40 |

INFO   | wrapper  | 2018/04/11 15:45:40 | Event Command 'wrapper_start': Command line[0] : /usr/sbin/cloudvm-ram-size

INFO   | wrapper  | 2018/04/11 15:45:40 | Event Command 'wrapper_start': Command line[1] : -J

INFO   | wrapper  | 2018/04/11 15:45:40 | Event Command 'wrapper_start': Command line[2] : vmware-eam

INFO   | wrapper  | 2018/04/11 15:45:40 | Event Command 'wrapper_start': Command line[3] : -O

INFO   | wrapper  | 2018/04/11 15:45:40 | Event Command 'wrapper_start': Command line[4] : /etc/vmware/java/vmware-eam_jvm.conf

INFO   | wrapper  | 2018/04/11 15:45:40 | Event Command 'wrapper_start': Command launched (pid: 18533), blocking for up to 3600 seconds...

INFO   | wrapper  | 2018/04/11 15:45:40 | Event Command 'wrapper_start': Command completed with exit code: 0

INFO   | wrapper  | 2018/04/11 15:45:40 | Event Command 'wrapper_start': Command completed with exit code: 0  Continuing.

INFO   | wrapper  | 2018/04/11 15:45:40 | Event Command 'jvm_prelaunch': Command line[0] : /opt/vmware/bin/python

INFO   | wrapper  | 2018/04/11 15:45:40 | Event Command 'jvm_prelaunch': Command line[1] : ../wrapper/bin/update_pnid.py

INFO   | wrapper  | 2018/04/11 15:45:40 | Event Command 'jvm_prelaunch': Command line[2] :

INFO   | wrapper  | 2018/04/11 15:45:40 | Event Command 'jvm_prelaunch': Command line[3] :

INFO   | wrapper  | 2018/04/11 15:45:40 | Event Command 'jvm_prelaunch': Command line[4] :

INFO   | wrapper  | 2018/04/11 15:45:40 | Event Command 'jvm_prelaunch': Command launched (pid: 18536), blocking for up to 15 seconds...

INFO   | wrapper  | 2018/04/11 15:45:40 | Event Command 'jvm_prelaunch': Command completed with exit code: 0

INFO   | wrapper  | 2018/04/11 15:45:40 | Event Command 'jvm_prelaunch': Command completed with exit code: 0  Continuing.

STATUS | wrapper  | 2018/04/11 15:45:40 | Launching a JVM...

INFO   | jvm 1    | 2018/04/11 15:45:40 | Picked up JAVA_TOOL_OPTIONS: -Xms32M -Xmx128M

INFO   | jvm 1    | 2018/04/11 15:45:41 | WrapperManager: Initializing...

INFO   | jvm 1    | 2018/04/11 15:45:42 | WrapperSimpleApp:

INFO   | jvm 1    | 2018/04/11 15:45:42 | WrapperSimpleApp Error: Encountered an error running main:

INFO   | jvm 1    | 2018/04/11 15:45:42 | WrapperSimpleApp Error: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'stsCertManager' defined in class path resource [eam-server-beans.xml]: Cannot resolve reference to bean 'cmClient' while setting constructor argument; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'cmClient' defined in class path resource [eam-server-beans.xml]: Cannot resolve reference to bean 'vcTrustStore' while setting constructor argument; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'vcTrustStore' defined in class path resource [eam-server-beans.xml]: Instantiation of bean failed; nested exception is org.springframework.beans.factory.BeanDefinitionStoreException: Factory method [public static java.security.KeyStore com.vmware.eam.identity.KeyStoreFactory.createKeyStore(java.lang.String,java.io.File,java.lang.String,java.lang.String)] threw exception; nested exception is com.vmware.eam.EamException: Unsupported keystore type: VKS

INFO   | jvm 1    | 2018/04/11 15:45:42 | WrapperSimpleApp Error:       at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveReference(BeanDefinitionValueResolver.java:275)

INFO   | jvm 1    | 2018/04/11 15:45:42 | WrapperSimpleApp Error:       at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveValueIfNecessary(BeanDefinitionValueResolver.java:104)

INFO   | jvm 1    | 2018/04/11 15:45:42 | WrapperSimpleApp Error:       at org.springframework.beans.factory.support.ConstructorResolver.resolveConstructorArguments(ConstructorResolver.java:474)

INFO   | jvm 1    | 2018/04/11 15:45:42 | WrapperSimpleApp Error:       at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:342)

INFO   | jvm 1    | 2018/04/11 15:45:42 | WrapperSimpleApp Error:       at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateUsingFactoryMethod(AbstractAutowireCapableBeanFactory.java:901)

INFO   | jvm 1    | 2018/04/11 15:45:42 | WrapperSimpleApp Error:       at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:815)

INFO   | jvm 1    | 2018/04/11 15:45:42 | WrapperSimpleApp Error:       at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:440)

INFO   | jvm 1    | 2018/04/11 15:45:42 | WrapperSimpleApp Error:       at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory$1.run(AbstractAutowireCapableBeanFactory.java:409)

INFO   | jvm 1    | 2018/04/11 15:45:42 | WrapperSimpleApp Error:       at java.security.AccessController.doPrivileged(Native Method)

INFO   | jvm 1    | 2018/04/11 15:45:42 | WrapperSimpleApp Error:       at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:380)

INFO   | jvm 1    | 2018/04/11 15:45:42 | WrapperSimpleApp Error:       at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:264)

INFO   | jvm 1    | 2018/04/11 15:45:42 | WrapperSimpleApp Error:       at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:220)

INFO   | jvm 1    | 2018/04/11 15:45:42 | WrapperSimpleApp Error:       at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:261)

INFO   | jvm 1    | 2018/04/11 15:45:42 | WrapperSimpleApp Error:       at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:185)

INFO   | jvm 1    | 2018/04/11 15:45:42 | WrapperSimpleApp Error:       at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:164)

INFO   | jvm 1    | 2018/04/11 15:45:42 | WrapperSimpleApp Error:       at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:254)

INFO   | jvm 1    | 2018/04/11 15:45:42 | WrapperSimpleApp Error:       at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:185)

INFO   | jvm 1    | 2018/04/11 15:45:42 | WrapperSimpleApp Error:       at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:164)

INFO   | jvm 1    | 2018/04/11 15:45:42 | WrapperSimpleApp Error:       at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:254)

INFO   | jvm 1    | 2018/04/11 15:45:42 | WrapperSimpleApp Error:       at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:185)

INFO   | jvm 1    | 2018/04/11 15:45:42 | WrapperSimpleApp Error:       at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:164)

INFO   | jvm 1    | 2018/04/11 15:45:42 | WrapperSimpleApp Error:       at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:254)

INFO   | jvm 1    | 2018/04/11 15:45:42 | WrapperSimpleApp Error:       at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:185)

INFO   | jvm 1    | 2018/04/11 15:45:42 | WrapperSimpleApp Error:       at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:164)

INFO   | jvm 1    | 2018/04/11 15:45:42 | WrapperSimpleApp Error:       at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:429)

INFO   | jvm 1    | 2018/04/11 15:45:42 | WrapperSimpleApp Error:       at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:729)

INFO   | jvm 1    | 2018/04/11 15:45:42 | WrapperSimpleApp Error:       at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:381)

INFO   | jvm 1    | 2018/04/11 15:45:42 | WrapperSimpleApp Error:       at org.springframework.context.support.ClassPathXmlApplicationContext.<init>(ClassPathXmlApplicationContext.java:139)

INFO   | jvm 1    | 2018/04/11 15:45:42 | WrapperSimpleApp Error:       at org.springframework.context.support.ClassPathXmlApplicationContext.<init>(ClassPathXmlApplicationContext.java:83)

INFO   | jvm 1    | 2018/04/11 15:45:42 | WrapperSimpleApp Error:       at com.vmware.eam.util.Main.main(Main.java:50)

INFO   | jvm 1    | 2018/04/11 15:45:42 | WrapperSimpleApp Error:       at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

INFO   | jvm 1    | 2018/04/11 15:45:42 | WrapperSimpleApp Error:       at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)

INFO   | jvm 1    | 2018/04/11 15:45:42 | WrapperSimpleApp Error:       at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)

INFO   | jvm 1    | 2018/04/11 15:45:42 | WrapperSimpleApp Error:       at java.lang.reflect.Method.invoke(Unknown Source)

INFO   | jvm 1    | 2018/04/11 15:45:42 | WrapperSimpleApp Error:       at org.tanukisoftware.wrapper.WrapperSimpleApp.run(WrapperSimpleApp.java:290)

INFO   | jvm 1    | 2018/04/11 15:45:42 | WrapperSimpleApp Error:       at java.lang.Thread.run(Unknown Source)

INFO   | jvm 1    | 2018/04/11 15:45:42 | WrapperSimpleApp Error: Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'cmClient' defined in class path resource [eam-server-beans.xml]: Cannot resolve reference to bean 'vcTrustStore' while setting constructor argument; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'vcTrustStore' defined in class path resource [eam-server-beans.xml]: Instantiation of bean failed; nested exception is org.springframework.beans.factory.BeanDefinitionStoreException: Factory method [public static java.security.KeyStore com.vmware.eam.identity.KeyStoreFactory.createKeyStore(java.lang.String,java.io.File,java.lang.String,java.lang.String)] threw exception; nested exception is com.vmware.eam.EamException: Unsupported keystore type: VKS

INFO   | jvm 1    | 2018/04/11 15:45:42 | WrapperSimpleApp Error:       at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveReference(BeanDefinitionValueResolver.java:275)

INFO   | jvm 1    | 2018/04/11 15:45:42 | WrapperSimpleApp Error:       at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveValueIfNecessary(BeanDefinitionValueResolver.java:104)

INFO   | jvm 1    | 2018/04/11 15:45:42 | WrapperSimpleApp Error:       at org.springframework.beans.factory.support.ConstructorResolver.resolveConstructorArguments(ConstructorResolver.java:474)

INFO   | jvm 1    | 2018/04/11 15:45:42 | WrapperSimpleApp Error:       at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:342)

INFO   | jvm 1    | 2018/04/11 15:45:42 | WrapperSimpleApp Error:       at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateUsingFactoryMethod(AbstractAutowireCapableBeanFactory.java:901)

INFO   | jvm 1    | 2018/04/11 15:45:42 | WrapperSimpleApp Error:       at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:815)

INFO   | jvm 1    | 2018/04/11 15:45:42 | WrapperSimpleApp Error:       at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:440)

INFO   | jvm 1    | 2018/04/11 15:45:42 | WrapperSimpleApp Error:       at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory$1.run(AbstractAutowireCapableBeanFactory.java:409)

INFO   | jvm 1    | 2018/04/11 15:45:42 | WrapperSimpleApp Error:       at java.security.AccessController.doPrivileged(Native Method)

INFO   | jvm 1    | 2018/04/11 15:45:42 | WrapperSimpleApp Error:       at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:380)

INFO   | jvm 1    | 2018/04/11 15:45:42 | WrapperSimpleApp Error:       at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:264)

INFO   | jvm 1    | 2018/04/11 15:45:42 | WrapperSimpleApp Error:       at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:220)

INFO   | jvm 1    | 2018/04/11 15:45:42 | WrapperSimpleApp Error:       at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:261)

INFO   | jvm 1    | 2018/04/11 15:45:42 | WrapperSimpleApp Error:       at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:185)

INFO   | jvm 1    | 2018/04/11 15:45:42 | WrapperSimpleApp Error:       at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:164)

INFO   | jvm 1    | 2018/04/11 15:45:42 | WrapperSimpleApp Error:       at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveReference(BeanDefinitionValueResolver.java:269)

INFO   | jvm 1    | 2018/04/11 15:45:42 | WrapperSimpleApp Error:       ... 35 more

INFO   | jvm 1    | 2018/04/11 15:45:42 | WrapperSimpleApp Error: Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'vcTrustStore' defined in class path resource [eam-server-beans.xml]: Instantiation of bean failed; nested exception is org.springframework.beans.factory.BeanDefinitionStoreException: Factory method [public static java.security.KeyStore com.vmware.eam.identity.KeyStoreFactory.createKeyStore(java.lang.String,java.io.File,java.lang.String,java.lang.String)] threw exception; nested exception is com.vmware.eam.EamException: Unsupported keystore type: VKS

INFO   | jvm 1    | 2018/04/11 15:45:42 | WrapperSimpleApp Error:       at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:423)

INFO   | jvm 1    | 2018/04/11 15:45:42 | WrapperSimpleApp Error:       at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateUsingFactoryMethod(AbstractAutowireCapableBeanFactory.java:901)

INFO   | jvm 1    | 2018/04/11 15:45:42 | WrapperSimpleApp Error:       at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:815)

INFO   | jvm 1    | 2018/04/11 15:45:42 | WrapperSimpleApp Error:       at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:440)

INFO   | jvm 1    | 2018/04/11 15:45:42 | WrapperSimpleApp Error:       at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory$1.run(AbstractAutowireCapableBeanFactory.java:409)

INFO   | jvm 1    | 2018/04/11 15:45:42 | WrapperSimpleApp Error:       at java.security.AccessController.doPrivileged(Native Method)

INFO   | jvm 1    | 2018/04/11 15:45:42 | WrapperSimpleApp Error:       at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:380)

INFO   | jvm 1    | 2018/04/11 15:45:42 | WrapperSimpleApp Error:       at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:264)

INFO   | jvm 1    | 2018/04/11 15:45:42 | WrapperSimpleApp Error:       at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:220)

INFO   | jvm 1    | 2018/04/11 15:45:42 | WrapperSimpleApp Error:       at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:261)

INFO   | jvm 1    | 2018/04/11 15:45:42 | WrapperSimpleApp Error:       at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:185)

INFO   | jvm 1    | 2018/04/11 15:45:42 | WrapperSimpleApp Error:       at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:164)

INFO   | jvm 1    | 2018/04/11 15:45:42 | WrapperSimpleApp Error:       at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveReference(BeanDefinitionValueResolver.java:269)

INFO   | jvm 1    | 2018/04/11 15:45:42 | WrapperSimpleApp Error:       ... 50 more

INFO   | jvm 1    | 2018/04/11 15:45:42 | WrapperSimpleApp Error: Caused by: org.springframework.beans.factory.BeanDefinitionStoreException: Factory method [public static java.security.KeyStore com.vmware.eam.identity.KeyStoreFactory.createKeyStore(java.lang.String,java.io.File,java.lang.String,java.lang.String)] threw exception; nested exception is com.vmware.eam.EamException: Unsupported keystore type: VKS

INFO   | jvm 1    | 2018/04/11 15:45:42 | WrapperSimpleApp Error:       at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:127)

INFO   | jvm 1    | 2018/04/11 15:45:42 | WrapperSimpleApp Error:       at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:414)

INFO   | jvm 1    | 2018/04/11 15:45:42 | WrapperSimpleApp Error:       ... 62 more

INFO   | jvm 1    | 2018/04/11 15:45:42 | WrapperSimpleApp Error: Caused by: com.vmware.eam.EamException: Unsupported keystore type: VKS

INFO   | jvm 1    | 2018/04/11 15:45:42 | WrapperSimpleApp Error:       at com.vmware.eam.identity.KeyStoreFactory.createKeyStore(KeyStoreFactory.java:48)

INFO   | jvm 1    | 2018/04/11 15:45:42 | WrapperSimpleApp Error:       at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

INFO   | jvm 1    | 2018/04/11 15:45:42 | WrapperSimpleApp Error:       at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)

INFO   | jvm 1    | 2018/04/11 15:45:42 | WrapperSimpleApp Error:       at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)

INFO   | jvm 1    | 2018/04/11 15:45:42 | WrapperSimpleApp Error:       at java.lang.reflect.Method.invoke(Unknown Source)

INFO   | jvm 1    | 2018/04/11 15:45:42 | WrapperSimpleApp Error:       at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:115)

INFO   | jvm 1    | 2018/04/11 15:45:42 | WrapperSimpleApp Error:       ... 63 more

STATUS | wrapper  | 2018/04/11 15:45:44 | <-- Wrapper Stopped

Reply
0 Kudos
ITDeptVM
Contributor
Contributor

nearly forgot....

ls -ltrh eam.properties

-rw-r--r-- 1 eam cis 2.3K Apr 11 15:45 eam.properties

I never created the file, I edited the blank one.

Reply
0 Kudos
ashwin_prakash
VMware Employee
VMware Employee

Could you please check the logs under:

/var/log/vmware/invsvc/inv-svc.log

/var/log/vmware/vmdird/vmdird-syslog.log

Also check if you have sufficient space.

Run the command df -h

Also check the logs for component manager.

Sincerely,
Ashwin Prakash
Skyline Support Moderator
Reply
0 Kudos
ITDeptVM
Contributor
Contributor

Volume info: -

Filesystem                            Size  Used Avail Use% Mounted on

/dev/sda3                              21G  5.2G   15G  27% /

udev                                  7.9G  164K  7.9G   1% /dev

tmpfs                                 7.9G   40K  7.9G   1% /dev/shm

/dev/sda1                             128M   41M   81M  34% /boot

/dev/mapper/core_vg-core               50G   13G   35G  27% /storage/core

/dev/mapper/log_vg-log                 25G   12G   12G  50% /storage/log

/dev/mapper/db_vg-db                   18G  234M   17G   2% /storage/db

/dev/mapper/dblog_vg-dblog            5.0G  219M  4.5G   5% /storage/dblog

/dev/mapper/seat_vg-seat               25G  463M   23G   2% /storage/seat

/dev/mapper/netdump_vg-netdump       1001M   18M  932M   2% /storage/netdump

/dev/mapper/autodeploy_vg-autodeploy   11G  154M   11G   2% /storage/autodeploy

/dev/mapper/invsvc_vg-invsvc           11G  181M   11G   2% /storage/invsvc

tmpfs                                 1.0M     0  1.0M   0% /var/spool/snmp

/var/log/vmware/invsvc/inv-svc.log contains INFO lines, no errors or debug lines.

/var/log/vmware/vmdird/vmdird-syslog.log

2018-04-12T02:37:20.227668+00:00 warning vmdird  t@139624981378816: ScanIndex: non-indexed attribute. attrType = vmwLicSvcObjectClass

2018-04-12T02:43:20.223385+00:00 warning vmdird  t@139624981378816: ScanIndex: non-indexed attribute. attrType = vmwLicSvcObjectClass

2018-04-12T02:56:16.043312+00:00 info vmdird  t@139624981378816: Modify Entry (cn=ls-vvcenter.corp.otmltd.com,ou=Domain Controllers,dc=vsphere,dc=local)

2018-04-12T02:56:16.046861+00:00 info vmdird  t@139624981378816: Modify Entry (cn=ls-vvcenter.corp.otmltd.com,ou=Domain Controllers,dc=vsphere,dc=local)

2018-04-12T02:56:16.046952+00:00 info vmdird  t@139624981378816: User account control - (cn=ls-vvcenter.corp.otmltd.com,ou=domain controllers,dc=vsphere,dc=local): (800010) flag unset, new value=(0)

2018-04-12T02:56:16.046988+00:00 info vmdird  t@139624981378816: Password Modification Successful ([23] 192.168.100.5:389<-192.168.100.5:53844). Bind DN: "cn=ls-vvcenter.corp.otmltd.com,ou=domain controllers,dc=vsphere,dc=local". Modified DN: "cn=ls-vvcenter.corp.otmltd.com,ou=Domain Controllers,dc=vsphere,dc=local"

2018-04-12T02:57:20.225419+00:00 warning vmdird  t@139624981378816: ScanIndex: non-indexed attribute. attrType = vmwLicSvcObjectClass

2018-04-12T02:57:20.293456+00:00 info vmdird  t@139624989771520: Bind failed () (9234)

2018-04-12T02:57:20.293500+00:00 err vmdird  t@139624989771520: VmDirSendLdapResult: Request (96), Error (49), Message (), (0) socket ([33] 127.0.0.1:389<-127.0.0.1:50786)

2018-04-12T02:57:20.293510+00:00 err vmdird  t@139624989771520: Bind Request Failed ([33] 127.0.0.1:389<-127.0.0.1:50786) error 49: Protocol version: 3, Bind DN: "cn=ls-vvcenter.corp.otmltd.com,ou=Domain Controllers,dc=vsphere,dc=local", Method: 128

2018-04-12T02:58:34.129236+00:00 warning vmdird  t@139624981378816: ScanIndex: non-indexed attribute. attrType = vmwLicSvcObjectClass

2018-04-12T02:58:34.376989+00:00 warning vmdird  t@139624981378816: ScanIndex: non-indexed attribute. attrType = vmwLicSvcObjectClass

2018-04-12T03:17:20.227219+00:00 warning vmdird  t@139624989771520: ScanIndex: non-indexed attribute. attrType = vmwLicSvcObjectClass

2018-04-12T03:23:20.221835+00:00 warning vmdird  t@139624989771520: ScanIndex: non-indexed attribute. attrType = vmwLicSvcObjectClass

/var/log/vmware/cm/cm.log

2018-04-12T09:08:38.899Z [pool-14-thread-1  DEBUG com.vmware.vim.vmomi.client.http.impl.ThumbprintTrustManager (299c822b-9337-49f3-a8a3-855ae3ac21dd)] Server certificate chain is trusted

2018-04-12T09:08:38.899Z [pool-14-thread-1  DEBUG com.vmware.vim.vmomi.client.http.impl.ThumbprintTrustManager (299c822b-9337-49f3-a8a3-855ae3ac21dd)] Server certificate chain is trusted

2018-04-12T09:08:38.899Z [pool-14-thread-1  DEBUG com.vmware.vim.vmomi.client.http.impl.ThumbprintTrustManager (299c822b-9337-49f3-a8a3-855ae3ac21dd)] Server certificate chain is trusted

2018-04-12T09:08:38.939Z [pool-14-thread-1  DEBUG com.vmware.vim.vmomi.client.http.impl.ThumbprintTrustManager (299c822b-9337-49f3-a8a3-855ae3ac21dd)] Server certificate assertion verified

2018-04-12T09:08:38.939Z [pool-14-thread-1  DEBUG com.vmware.vim.vmomi.client.http.impl.ThumbprintTrustManager (299c822b-9337-49f3-a8a3-855ae3ac21dd)] Server certificate assertion verified

2018-04-12T09:08:38.939Z [pool-14-thread-1  DEBUG com.vmware.vim.vmomi.client.http.impl.ThumbprintTrustManager (299c822b-9337-49f3-a8a3-855ae3ac21dd)] Server certificate assertion verified

2018-04-12T09:11:19.759Z [pool-1-thread-1  INFO  com.vmware.cis.services.common.sso.LoginHelper (41e192f1-b4a7-4062-ab61-90b65c5715ef)] loginByToken: User 'vsphere-webclient-4f1feb00-5481-47e8-a7c3-6808106d7fdb@vsphere.local' is already authenticated

2018-04-12T09:11:19.770Z [pool-14-thread-1  DEBUG com.vmware.vim.vmomi.client.http.impl.ThumbprintTrustManager (299c822b-9337-49f3-a8a3-855ae3ac21dd)] Server certificate chain is trusted

2018-04-12T09:11:19.770Z [pool-14-thread-1  DEBUG com.vmware.vim.vmomi.client.http.impl.ThumbprintTrustManager (299c822b-9337-49f3-a8a3-855ae3ac21dd)] Server certificate chain is trusted

2018-04-12T09:11:19.770Z [pool-14-thread-1  DEBUG com.vmware.vim.vmomi.client.http.impl.ThumbprintTrustManager (299c822b-9337-49f3-a8a3-855ae3ac21dd)] Server certificate chain is trusted

2018-04-12T09:11:19.811Z [pool-14-thread-1  DEBUG com.vmware.vim.vmomi.client.http.impl.ThumbprintTrustManager (299c822b-9337-49f3-a8a3-855ae3ac21dd)] Server certificate assertion verified

2018-04-12T09:11:19.811Z [pool-14-thread-1  DEBUG com.vmware.vim.vmomi.client.http.impl.ThumbprintTrustManager (299c822b-9337-49f3-a8a3-855ae3ac21dd)] Server certificate assertion verified

2018-04-12T09:11:19.811Z [pool-14-thread-1  DEBUG com.vmware.vim.vmomi.client.http.impl.ThumbprintTrustManager (299c822b-9337-49f3-a8a3-855ae3ac21dd)] Server certificate assertion verified

2018-04-12T09:11:20.190Z [pool-1-thread-1  INFO  com.vmware.cis.services.common.sso.LoginHelper (6e8e7761-33ce-4d7a-a940-9cfc308a36ca)] loginByToken: User 'vsphere-webclient-4f1feb00-5481-47e8-a7c3-6808106d7fdb@vsphere.local' is already authenticated

Reply
0 Kudos
ashwin_prakash
VMware Employee
VMware Employee

Thank you for sharing the details. As per the errror observed, You would have to reset the password for the account:

Bind DN: "cn=ls-vvcenter.corp.otmltd.com,ou=Domain Controllers,dc=vsphere,dc=local"

You could follow the steps mentioned below in the article to reset the password.

VMware Knowledge Base

Keep generating vdcadmintool password until the password does not contain * @ , \ " or space.

Sincerely,
Ashwin Prakash
Skyline Support Moderator
Reply
0 Kudos