Hi all
I have a problem with EAM. We have been using NSX guest introspection and it started playing up recently, which attracted our attention.
In vSphere Web Client, under Administration\vSphere ESX Agent Manager \ Solution I get the following error:
503 Service Unavailable (Failed to connect to endpoint: [N7Vmacore4Http16LocalServiceSpecE:0x7fdc140fea00] _serverNamespace = /eam _isRedirect = false _port = 15005)
On VCSA, the VMware-eam service is not running. Starting it results in an error, the details of which are found in /var/log/VMware-eam/wrapper.log
STATUS | wrapper | 2017/09/06 15:45:55 | --> Wrapper Started as Daemon
STATUS | wrapper | 2017/09/06 15:45:55 | Java Service Wrapper Professional Edition 64-bit 3.5.24
STATUS | wrapper | 2017/09/06 15:45:55 | Copyright (C) 1999-2014 Tanuki Software, Ltd. All Rights Reserved.
STATUS | wrapper | 2017/09/06 15:45:55 | http://wrapper.tanukisoftware.com
STATUS | wrapper | 2017/09/06 15:45:55 | Licensed to VMware Global, Inc. for VMware vCenter ESX Agent Manager
STATUS | wrapper | 2017/09/06 15:45:55 |
INFO | wrapper | 2017/09/06 15:45:55 | Event Command 'wrapper_start': Command line[0] : /usr/sbin/cloudvm-ram-size
INFO | wrapper | 2017/09/06 15:45:55 | Event Command 'wrapper_start': Command line[1] : -J
INFO | wrapper | 2017/09/06 15:45:55 | Event Command 'wrapper_start': Command line[2] : vmware-eam
INFO | wrapper | 2017/09/06 15:45:55 | Event Command 'wrapper_start': Command line[3] : -O
INFO | wrapper | 2017/09/06 15:45:55 | Event Command 'wrapper_start': Command line[4] : /etc/vmware/java/vmware-eam_jvm.conf
INFO | wrapper | 2017/09/06 15:45:55 | Event Command 'wrapper_start': Command launched (pid: 23310), blocking for up to 3600 seconds...
INFO | wrapper | 2017/09/06 15:45:56 | Event Command 'wrapper_start': Command completed with exit code: 0
INFO | wrapper | 2017/09/06 15:45:56 | Event Command 'wrapper_start': Command completed with exit code: 0 Continuing.
INFO | wrapper | 2017/09/06 15:45:56 | Event Command 'jvm_prelaunch': Command line[0] : /opt/vmware/bin/python
INFO | wrapper | 2017/09/06 15:45:56 | Event Command 'jvm_prelaunch': Command line[1] : ../wrapper/bin/update_pnid.py
INFO | wrapper | 2017/09/06 15:45:56 | Event Command 'jvm_prelaunch': Command line[2] :
INFO | wrapper | 2017/09/06 15:45:56 | Event Command 'jvm_prelaunch': Command line[3] :
INFO | wrapper | 2017/09/06 15:45:56 | Event Command 'jvm_prelaunch': Command line[4] :
INFO | wrapper | 2017/09/06 15:45:56 | Event Command 'jvm_prelaunch': Command launched (pid: 23313), blocking for up to 15 seconds...
INFO | wrapper | 2017/09/06 15:45:56 | Event Command 'jvm_prelaunch': Command completed with exit code: 1
INFO | wrapper | 2017/09/06 15:45:56 | Event Command 'jvm_prelaunch': Command completed with exit code: 1 Continuing.
STATUS | wrapper | 2017/09/06 15:45:56 | Launching a JVM...
INFO | jvm 1 | 2017/09/06 15:45:56 | Picked up JAVA_TOOL_OPTIONS: -Xms32M -Xmx128M
INFO | jvm 1 | 2017/09/06 15:45:56 | WrapperManager: Initializing...
INFO | jvm 1 | 2017/09/06 15:45:57 | WrapperSimpleApp:
INFO | jvm 1 | 2017/09/06 15:45:57 | WrapperSimpleApp Error: Encountered an error running main:
INFO | jvm 1 | 2017/09/06 15:45:57 | WrapperSimpleApp Error: org.springframework.beans.factory.BeanDefinitionStoreException: Invalid bean definition with name 'esxAgentMgr' defined in class path re source [eam-server-beans.xml]: Could not resolve placeholder 'eam.hostd_restart_timeout'
INFO | jvm 1 | 2017/09/06 15:45:57 | WrapperSimpleApp Error: at org.springframework.beans.factory.config.PropertyPlaceholderConfigurer.processProperties(PropertyPlaceholderConfigurer.java:268 )
INFO | jvm 1 | 2017/09/06 15:45:57 | WrapperSimpleApp Error: at org.springframework.beans.factory.config.PropertyResourceConfigurer.postProcessBeanFactory(PropertyResourceConfigurer.java:75)
INFO | jvm 1 | 2017/09/06 15:45:57 | WrapperSimpleApp Error: at org.springframework.context.support.AbstractApplicationContext.invokeBeanFactoryPostProcessors(AbstractApplicationContext.java: 554)
INFO | jvm 1 | 2017/09/06 15:45:57 | WrapperSimpleApp Error: at org.springframework.context.support.AbstractApplicationContext.invokeBeanFactoryPostProcessors(AbstractApplicationContext.java: 528)
INFO | jvm 1 | 2017/09/06 15:45:57 | WrapperSimpleApp Error: at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:363)
INFO | jvm 1 | 2017/09/06 15:45:57 | WrapperSimpleApp Error: at org.springframework.context.support.ClassPathXmlApplicationContext.<init>(ClassPathXmlApplicationContext.java:139)
INFO | jvm 1 | 2017/09/06 15:45:57 | WrapperSimpleApp Error: at org.springframework.context.support.ClassPathXmlApplicationContext.<init>(ClassPathXmlApplicationContext.java:83)
INFO | jvm 1 | 2017/09/06 15:45:57 | WrapperSimpleApp Error: at com.vmware.eam.util.Main.main(Main.java:50)
INFO | jvm 1 | 2017/09/06 15:45:57 | WrapperSimpleApp Error: at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
INFO | jvm 1 | 2017/09/06 15:45:57 | WrapperSimpleApp Error: at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
INFO | jvm 1 | 2017/09/06 15:45:57 | WrapperSimpleApp Error: at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
INFO | jvm 1 | 2017/09/06 15:45:57 | WrapperSimpleApp Error: at java.lang.reflect.Method.invoke(Unknown Source)
INFO | jvm 1 | 2017/09/06 15:45:57 | WrapperSimpleApp Error: at org.tanukisoftware.wrapper.WrapperSimpleApp.run(WrapperSimpleApp.java:290)
INFO | jvm 1 | 2017/09/06 15:45:57 | WrapperSimpleApp Error: at java.lang.Thread.run(Unknown Source)
STATUS | wrapper | 2017/09/06 15:45:59 | <-- Wrapper Stopped
So following my nose, a potentially relevant snip from the aforementioned eam-server-beans.xml found in /usr/lib/VMware-eam/conf is:
<!-- ESX Agent Manager beans -->
<bean id="esxAgentMgr"
class="com.vmware.eam.EsxAgentManagerImpl"
lazy-init="false"
depends-on="springContext,
vcInventory,
vcListener,
httpNfcLeaseTimeoutHandler,
nameManager,
vcDatastoreManager,
vcEventManager,
vcKeyValueStore,
l10n">
<constructor-arg ref="adapterServer" />
<constructor-arg value="${vc.proxy.host}" />
<constructor-arg value="${vc.proxy.port}" />
<constructor-arg value="${vc.tunnelSdkUri}" />
<constructor-arg ref="clientAuthenticator" />
<constructor-arg value="${eam.clear_db_on_startup}" />
<constructor-arg value="${eam.support_linked_clone}" />
<constructor-arg value="${eam.debug_ref_count}" />
<constructor-arg value="${vum.integration}" />
<constructor-arg value="${Net.DVFilterBindIpAddress}" />
<constructor-arg value="${Net.TrafficFilterIpAddress}" />
<constructor-arg value="${Net.DVFilterVswifIpAddress}" />
<constructor-arg ref="commonMapStore" />
<constructor-arg ref="vpxdConnection" />
<constructor-arg ref="vumConnection" />
<constructor-arg ref="nameManager" />
<constructor-arg ref="vibPackageDepots" />
<property name="hostdRestartTimeout" value="${eam.hostd_restart_timeout}" />
</bean>
I'm struggling as I need to get my NSX piece back up and running (ironically so I can gracefully remove it), but believe I need the EAM to be working properly first.
Any ideas before I open a ticket?
Thanks.
vDave
Thanks for taking the time to post Rajeev.
But i already went through that article before posting my question. None of the symptoms listed seem to apply to my problem shown in the logs above.
In a nutshell, my vmware-eam service on VCSA will not start.
I have seen similar before - Issue was Splunk was installed on the VM and was using the same port for forwarding etc.
We have just had the exact same issue trying to upgrade vSphere 6.0U3 to 6.5. In our case the eam.properties file located at C:\ProgramData\VMware\vCenterServer\cfg\vmware-eam was completely blank. Luckily we had another vSphere server running 6.0U3, so we copied the contents of the eam.properties file from that server and pasted it in to the blank file on the faulty server. There is one URI at the bottom of the file you will need to change to match the hostname of the server. After that the Agent Manager service started OK.
Hey,
By any chance, do you remember what you ended up doing to resolve the issue?
Thanks,
What was the contents on your 6.0U3 eam.properties file you pasted in?
Having same issue with EAM service not starting, Invalid bean definition with name 'esxAgentMgr' defined
vcenter:~ # cat /etc/vmware-eam/eam.properties
eam.web.root=/usr/lib/vmware-eam/web
eam.web.root=/usr/lib/vmware-eam/web
the files eam.properties just has one line dupped twice in it and we have no other vcenter server to crib from.
I just encountered this same issue and was able to resolve it by copying the contents of /etc/vmware-eam/eam.properties from a 6.0U3 appliance in my lab. I was then able to start the vmware-eam service and proceed with my 6.5 upgrade.
Below are the contents of the file I used. Maybe it will help someone else who doesn't have a spare appliance sitting around. Note that you need to change the the line vc.tunnelSdkUri=https://vcenter:8089/sdk/vimService to match your hostname.
#########################################################################
# Copyright 2013-2017 VMware, Inc. All rights reserved. VMware Confidential
#########################################################################
vc.proxy.host=localhost
vc.proxy.port=80
# Hostname or IP of the EAM server
# Fill only if EAM is not running on the same host as VC
eam.host=
# EAM service port used to configure the HTTP connector of the application server.
eam.int.http.port=15005
# Port and scheme configuration which is used by the ESX 6.x hosts to reach EAM Vib
# file server.
eam.ext.port=443
eam.ext.scheme=https
# Port and scheme configuration which is used by the ESX 5.x hosts to reach EAM Vib
# file server.
eam.ext.port.deprecated=80
eam.ext.scheme.deprecated=http
eam.support_linked_clone=true
eam.clear_db_on_startup=false
eam.debug_ref_count=false
eam.recent_event_size=20
# Value is specified in minutes (set to 24h = 1440m)
eam.scan_for_unknown_agent_vms=1440
# The timeout to wait for hostd to restart on a host (set to 5m=300s)
eam.hostd_restart_timeout=300
# The following entries will be added verbatim to the advanced options
# of hosts on which EAM is enabled. All are optional.
Net.DVFilterBindIpAddress=169.254.0.1
Net.TrafficFilterIpAddress=
#The IP for the VSWIF NIC on the dvFilter switch (for ESX classic).
Net.DVFilterVswifIpAddress=169.254.0.2
# Resource bundle configuration
eam.resourcebundle.filename=eam-resourcebundle.jar
# VLSI embedded tcServer configuration
#
tcserver.tmp.dir=/var/tmp/vmware/eam/tomcat
eam.web.root=/usr/lib/vmware-eam/web
# EAM SSL configuration
#
eam.keystore.type=VKS
eam.key.alias=vpxd-extension
eam.keystore.storename=vpxd-extension
# CM configuration
#
cm.url=http://localhost:18090/cm/sdk/?hostid=fa3bd34a-4b3c-4ff0-ad4f-1c97ad6f93b5
cm.wait.attempts=360
cm.wait.intervalSeconds=5
# SSO configuration
#
sso.wait.attempts=360
sso.wait.intervalSeconds=5
# VC SSL configuration
#
vc.truststore.type=VKS
vc.truststore.storename=TRUSTED_ROOTS
vc.tunnelSdkUri.template=https://##{VC_HOST_NAME}##:8089/sdk/vimService
vc.tunnelSdkUri=https://vcenter:8089/sdk/vimService
drs.demandCapacityRatio=100
This was really helpful for me as I just ran into the same problem which was also blocking my 6.5 upgrade. Through discussion with VMware - in addition, line 46 in your example should be updated to include your specific hostID obtained using:
cat /etc/vmware/install-defaults/sca.hostid
After that I found that an additional line was required before the service would start correctly:
eam.web.root=/usr/lib/vmware-eam/web
Finally, in case this was done using the root user you might find it necessary to set the permissions correctly for the properties file if it was deleted/recreated:
chown eam:cis eam.properties
chmod -R 644 eam.properties
Hope this helps if someone else runs into this issue during an upgrade..
Attention when copy/paste this template 2 spaces are on end of lines , must be removed
"eam.keystore.type=VKS "
"vc.truststore.type=VKS "
sed to the rescue
sed -ire 's/ $//g' /etc/vmware-eam/eam.properties
on line 46:
change with your hostid with content of
cat /etc/vmware/install-defaults/sca.hostid
on line 24: (eameam is one eam only)
I tired everything under the sun to get eam to run. The chmown worked perfect. I am no able to upgrade vcenter 6.0 to 6.7.
chown eam:cis eam.properties
Thanks,
Chris