topic Re: vSphere 6.0U3 EAM Error in VMware vCenter™ Discussions

vSphere 6.0U3 EAM Error

virtual_dave1 — Wed, 06 Sep 2017 15:27:48 GMT

Hi all

I have a problem with EAM. We have been using NSX guest introspection and it started playing up recently, which attracted our attention.

In vSphere Web Client, under Administration\vSphere ESX Agent Manager \ Solution I get the following error:

503 Service Unavailable (Failed to connect to endpoint: [N7Vmacore4Http16LocalServiceSpecE:0x7fdc140fea00] _serverNamespace = /eam _isRedirect = false _port = 15005)

On VCSA, the VMware-eam service is not running. Starting it results in an error, the details of which are found in /var/log/VMware-eam/wrapper.log

STATUS | wrapper | 2017/09/06 15:45:55 | --> Wrapper Started as Daemon

STATUS | wrapper | 2017/09/06 15:45:55 | Java Service Wrapper Professional Edition 64-bit 3.5.24

STATUS | wrapper | 2017/09/06 15:45:55 | http://wrapper.tanukisoftware.com

STATUS | wrapper | 2017/09/06 15:45:55 | Licensed to VMware Global, Inc. for VMware vCenter ESX Agent Manager

STATUS | wrapper | 2017/09/06 15:45:55 |

INFO | wrapper | 2017/09/06 15:45:55 | Event Command 'wrapper_start': Command line[0] : /usr/sbin/cloudvm-ram-size

INFO | wrapper | 2017/09/06 15:45:55 | Event Command 'wrapper_start': Command line[1] : -J

INFO | wrapper | 2017/09/06 15:45:55 | Event Command 'wrapper_start': Command line[2] : vmware-eam

INFO | wrapper | 2017/09/06 15:45:55 | Event Command 'wrapper_start': Command line[3] : -O

INFO | wrapper | 2017/09/06 15:45:55 | Event Command 'wrapper_start': Command line[4] : /etc/vmware/java/vmware-eam_jvm.conf

INFO | wrapper | 2017/09/06 15:45:55 | Event Command 'wrapper_start': Command launched (pid: 23310), blocking for up to 3600 seconds...

INFO | wrapper | 2017/09/06 15:45:56 | Event Command 'wrapper_start': Command completed with exit code: 0

INFO | wrapper | 2017/09/06 15:45:56 | Event Command 'wrapper_start': Command completed with exit code: 0 Continuing.

INFO | wrapper | 2017/09/06 15:45:56 | Event Command 'jvm_prelaunch': Command line[0] : /opt/vmware/bin/python

INFO | wrapper | 2017/09/06 15:45:56 | Event Command 'jvm_prelaunch': Command line[1] : ../wrapper/bin/update_pnid.py

INFO | wrapper | 2017/09/06 15:45:56 | Event Command 'jvm_prelaunch': Command line[2] :

INFO | wrapper | 2017/09/06 15:45:56 | Event Command 'jvm_prelaunch': Command line[3] :

INFO | wrapper | 2017/09/06 15:45:56 | Event Command 'jvm_prelaunch': Command line[4] :

INFO | wrapper | 2017/09/06 15:45:56 | Event Command 'jvm_prelaunch': Command launched (pid: 23313), blocking for up to 15 seconds...

INFO | wrapper | 2017/09/06 15:45:56 | Event Command 'jvm_prelaunch': Command completed with exit code: 1

INFO | wrapper | 2017/09/06 15:45:56 | Event Command 'jvm_prelaunch': Command completed with exit code: 1 Continuing.

STATUS | wrapper | 2017/09/06 15:45:56 | Launching a JVM...

INFO | jvm 1 | 2017/09/06 15:45:56 | Picked up JAVA_TOOL_OPTIONS: -Xms32M -Xmx128M

INFO | jvm 1 | 2017/09/06 15:45:56 | WrapperManager: Initializing...

INFO | jvm 1 | 2017/09/06 15:45:57 | WrapperSimpleApp:

INFO | jvm 1 | 2017/09/06 15:45:57 | WrapperSimpleApp Error: Encountered an error running main:

INFO | jvm 1 | 2017/09/06 15:45:57 | WrapperSimpleApp Error: org.springframework.beans.factory.BeanDefinitionStoreException: Invalid bean definition with name 'esxAgentMgr' defined in class path re source [eam-server-beans.xml]: Could not resolve placeholder 'eam.hostd_restart_timeout'

INFO | jvm 1 | 2017/09/06 15:45:57 | WrapperSimpleApp Error: at org.springframework.beans.factory.config.PropertyPlaceholderConfigurer.processProperties(PropertyPlaceholderConfigurer.java:268 )

INFO | jvm 1 | 2017/09/06 15:45:57 | WrapperSimpleApp Error: at org.springframework.beans.factory.config.PropertyResourceConfigurer.postProcessBeanFactory(PropertyResourceConfigurer.java:75)

INFO | jvm 1 | 2017/09/06 15:45:57 | WrapperSimpleApp Error: at org.springframework.context.support.AbstractApplicationContext.invokeBeanFactoryPostProcessors(AbstractApplicationContext.java: 554)

INFO | jvm 1 | 2017/09/06 15:45:57 | WrapperSimpleApp Error: at org.springframework.context.support.AbstractApplicationContext.invokeBeanFactoryPostProcessors(AbstractApplicationContext.java: 528)

INFO | jvm 1 | 2017/09/06 15:45:57 | WrapperSimpleApp Error: at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:363)

INFO | jvm 1 | 2017/09/06 15:45:57 | WrapperSimpleApp Error: at org.springframework.context.support.ClassPathXmlApplicationContext.<init>(ClassPathXmlApplicationContext.java:139)

INFO | jvm 1 | 2017/09/06 15:45:57 | WrapperSimpleApp Error: at org.springframework.context.support.ClassPathXmlApplicationContext.<init>(ClassPathXmlApplicationContext.java:83)

INFO | jvm 1 | 2017/09/06 15:45:57 | WrapperSimpleApp Error: at com.vmware.eam.util.Main.main(Main.java:50)

INFO | jvm 1 | 2017/09/06 15:45:57 | WrapperSimpleApp Error: at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

INFO | jvm 1 | 2017/09/06 15:45:57 | WrapperSimpleApp Error: at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)

INFO | jvm 1 | 2017/09/06 15:45:57 | WrapperSimpleApp Error: at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)

INFO | jvm 1 | 2017/09/06 15:45:57 | WrapperSimpleApp Error: at java.lang.reflect.Method.invoke(Unknown Source)

INFO | jvm 1 | 2017/09/06 15:45:57 | WrapperSimpleApp Error: at org.tanukisoftware.wrapper.WrapperSimpleApp.run(WrapperSimpleApp.java:290)

INFO | jvm 1 | 2017/09/06 15:45:57 | WrapperSimpleApp Error: at java.lang.Thread.run(Unknown Source)

STATUS | wrapper | 2017/09/06 15:45:59 | <-- Wrapper Stopped

So following my nose, a potentially relevant snip from the aforementioned eam-server-beans.xml found in /usr/lib/VMware-eam/conf is:

<bean id="esxAgentMgr"

class="com.vmware.eam.EsxAgentManagerImpl"

lazy-init="false"

depends-on="springContext,

vcInventory,

vcListener,

httpNfcLeaseTimeoutHandler,

nameManager,

vcDatastoreManager,

vcEventManager,

vcKeyValueStore,

l10n">

<constructor-arg ref="adapterServer" />

<constructor-arg value="${vc.proxy.host}" />

<constructor-arg value="${vc.proxy.port}" />

<constructor-arg value="${vc.tunnelSdkUri}" />

<constructor-arg ref="clientAuthenticator" />

<constructor-arg value="${eam.clear_db_on_startup}" />

<constructor-arg value="${eam.support_linked_clone}" />

<constructor-arg value="${eam.debug_ref_count}" />

<constructor-arg value="${vum.integration}" />

<constructor-arg value="${Net.DVFilterBindIpAddress}" />

<constructor-arg value="${Net.TrafficFilterIpAddress}" />

<constructor-arg value="${Net.DVFilterVswifIpAddress}" />

<constructor-arg ref="commonMapStore" />

<constructor-arg ref="vpxdConnection" />

<constructor-arg ref="vumConnection" />

<constructor-arg ref="nameManager" />

<constructor-arg ref="vibPackageDepots" />

<property name="hostdRestartTimeout" value="${eam.hostd_restart_timeout}" />

</bean>

I'm struggling as I need to get my NSX piece back up and running (ironically so I can gracefully remove it), but believe I need the EAM to be working properly first.

Any ideas before I open a ticket?

Thanks.

vDave

Re: vSphere 6.0U3 EAM Error

RajeevVCP4 — Wed, 06 Sep 2017 17:29:55 GMT

https://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2122392

Re: vSphere 6.0U3 EAM Error

virtual_dave1 — Thu, 07 Sep 2017 08:18:42 GMT

Thanks for taking the time to post Rajeev.

But i already went through that article before posting my question. None of the symptoms listed seem to apply to my problem shown in the logs above.

In a nutshell, my vmware-eam service on VCSA will not start.

Re: vSphere 6.0U3 EAM Error

cliffcahill — Thu, 07 Sep 2017 10:52:01 GMT

I have seen similar before - Issue was Splunk was installed on the VM and was using the same port for forwarding etc.

Re: vSphere 6.0U3 EAM Error

airfrog7 — Tue, 17 Oct 2017 15:21:39 GMT

We have just had the exact same issue trying to upgrade vSphere 6.0U3 to 6.5. In our case the eam.properties file located at C:\ProgramData\VMware\vCenterServer\cfg\vmware-eam was completely blank. Luckily we had another vSphere server running 6.0U3, so we copied the contents of the eam.properties file from that server and pasted it in to the blank file on the faulty server. There is one URI at the bottom of the file you will need to change to match the hostname of the server. After that the Agent Manager service started OK.

Re: vSphere 6.0U3 EAM Error

MarkSen — Tue, 31 Oct 2017 19:25:45 GMT

Hey,

By any chance, do you remember what you ended up doing to resolve the issue?

Thanks,

Re: vSphere 6.0U3 EAM Error

Pattonville — Sat, 20 Jan 2018 15:57:17 GMT

What was the contents on your 6.0U3 eam.properties file you pasted in?

Having same issue with EAM service not starting, Invalid bean definition with name 'esxAgentMgr' defined

vcenter:~ # cat /etc/vmware-eam/eam.properties

eam.web.root=/usr/lib/vmware-eam/web

the files eam.properties just has one line dupped twice in it and we have no other vcenter server to crib from.

Re: vSphere 6.0U3 EAM Error

ImmedionChris — Wed, 07 Feb 2018 14:51:24 GMT

I just encountered this same issue and was able to resolve it by copying the contents of /etc/vmware-eam/eam.properties from a 6.0U3 appliance in my lab. I was then able to start the vmware-eam service and proceed with my 6.5 upgrade.

Below are the contents of the file I used. Maybe it will help someone else who doesn't have a spare appliance sitting around. Note that you need to change the the line vc.tunnelSdkUri=https://vcenter:8089/sdk/vimService to match your hostname.

#########################################################################

vc.proxy.host=localhost

vc.proxy.port=80

# Hostname or IP of the EAM server

# Fill only if EAM is not running on the same host as VC

eam.host=

# EAM service port used to configure the HTTP connector of the application server.

eam.int.http.port=15005

# Port and scheme configuration which is used by the ESX 6.x hosts to reach EAM Vib

# file server.

eam.ext.port=443

eam.ext.scheme=https

# Port and scheme configuration which is used by the ESX 5.x hosts to reach EAM Vib

# file server.

eam.ext.port.deprecated=80

eam.ext.scheme.deprecated=http

eam.support_linked_clone=true

eam.clear_db_on_startup=false

eam.debug_ref_count=false

eam.recent_event_size=20

# Value is specified in minutes (set to 24h = 1440m)

eam.scan_for_unknown_agent_vms=1440

# The timeout to wait for hostd to restart on a host (set to 5m=300s)

eam.hostd_restart_timeout=300

# The following entries will be added verbatim to the advanced options

# of hosts on which EAM is enabled. All are optional.

Net.DVFilterBindIpAddress=169.254.0.1

Net.TrafficFilterIpAddress=

#The IP for the VSWIF NIC on the dvFilter switch (for ESX classic).

Net.DVFilterVswifIpAddress=169.254.0.2

# Resource bundle configuration

eam.resourcebundle.filename=eam-resourcebundle.jar

# VLSI embedded tcServer configuration

tcserver.tmp.dir=/var/tmp/vmware/eam/tomcat

eam.web.root=/usr/lib/vmware-eam/web

# EAM SSL configuration

eam.keystore.type=VKS

eam.key.alias=vpxd-extension

eam.keystore.storename=vpxd-extension

# CM configuration

cm.url=http://localhost:18090/cm/sdk/?hostid=fa3bd34a-4b3c-4ff0-ad4f-1c97ad6f93b5

cm.wait.attempts=360

cm.wait.intervalSeconds=5

# SSO configuration

sso.wait.attempts=360

sso.wait.intervalSeconds=5

# VC SSL configuration

vc.truststore.type=VKS

vc.truststore.storename=TRUSTED_ROOTS

vc.tunnelSdkUri.template=https://##{VC_HOST_NAME}##:8089/sdk/vimService

vc.tunnelSdkUri=https://vcenter:8089/sdk/vimService

drs.demandCapacityRatio=100

Re: vSphere 6.0U3 EAM Error

swalker_LST — Thu, 29 Mar 2018 13:27:26 GMT

This was really helpful for me as I just ran into the same problem which was also blocking my 6.5 upgrade. Through discussion with VMware - in addition, line 46 in your example should be updated to include your specific hostID obtained using:

cat /etc/vmware/install-defaults/sca.hostid

After that I found that an additional line was required before the service would start correctly:

eam.web.root=/usr/lib/vmware-eam/web

Finally, in case this was done using the root user you might find it necessary to set the permissions correctly for the properties file if it was deleted/recreated:

chown eam:cis eam.properties
chmod -R 644 eam.properties

Hope this helps if someone else runs into this issue during an upgrade..

Re: vSphere 6.0U3 EAM Error

SupOE — Tue, 25 Sep 2018 09:46:33 GMT

Attention when copy/paste this template 2 spaces are on end of lines , must be removed

"eam.keystore.type=VKS "

"vc.truststore.type=VKS "

sed to the rescue

sed -ire 's/ $//g' /etc/vmware-eam/eam.properties

on line 46:

cm.url=http://localhost:18090/cm/sdk/?hostid=fa3bd34a-4b3c-4ff0-ad4f-1c97ad6f93b5

change with your hostid with content of

cat /etc/vmware/install-defaults/sca.hostid

on line 24: (eameam is one eam only)

eameam.resourcebundle.filename=eam-resourcebundle.jar
=>>
eam.resourcebundle.filename=eam-resourcebundle.jar

Re: vSphere 6.0U3 EAM Error

offlinetn — Mon, 26 Aug 2019 18:32:49 GMT

I tired everything under the sun to get eam to run. The chmown worked perfect. I am no able to upgrade vcenter 6.0 to 6.7.

chown eam:cis eam.properties

Thanks,

Chris