VMware Cloud Community
virtual_dave1
Contributor
Contributor

vSphere 6.0U3 EAM Error

Hi all

I have a problem with EAM.  We have been using NSX guest introspection and it started playing up recently, which attracted our attention.

In vSphere Web Client, under Administration\vSphere ESX Agent Manager \ Solution I get the following error:

503 Service Unavailable (Failed to connect to endpoint: [N7Vmacore4Http16LocalServiceSpecE:0x7fdc140fea00] _serverNamespace = /eam _isRedirect = false _port = 15005)

On VCSA, the VMware-eam service is not running.  Starting it results in an error, the details of which are found in /var/log/VMware-eam/wrapper.log

STATUS | wrapper  | 2017/09/06 15:45:55 | --> Wrapper Started as Daemon

STATUS | wrapper  | 2017/09/06 15:45:55 | Java Service Wrapper Professional Edition 64-bit 3.5.24

STATUS | wrapper  | 2017/09/06 15:45:55 |   Copyright (C) 1999-2014 Tanuki Software, Ltd. All Rights Reserved.

STATUS | wrapper  | 2017/09/06 15:45:55 |     http://wrapper.tanukisoftware.com

STATUS | wrapper  | 2017/09/06 15:45:55 |   Licensed to VMware Global, Inc. for VMware vCenter ESX Agent Manager

STATUS | wrapper  | 2017/09/06 15:45:55 |

INFO   | wrapper  | 2017/09/06 15:45:55 | Event Command 'wrapper_start': Command line[0] : /usr/sbin/cloudvm-ram-size

INFO   | wrapper  | 2017/09/06 15:45:55 | Event Command 'wrapper_start': Command line[1] : -J

INFO   | wrapper  | 2017/09/06 15:45:55 | Event Command 'wrapper_start': Command line[2] : vmware-eam

INFO   | wrapper  | 2017/09/06 15:45:55 | Event Command 'wrapper_start': Command line[3] : -O

INFO   | wrapper  | 2017/09/06 15:45:55 | Event Command 'wrapper_start': Command line[4] : /etc/vmware/java/vmware-eam_jvm.conf

INFO   | wrapper  | 2017/09/06 15:45:55 | Event Command 'wrapper_start': Command launched (pid: 23310), blocking for up to 3600 seconds...

INFO   | wrapper  | 2017/09/06 15:45:56 | Event Command 'wrapper_start': Command completed with exit code: 0

INFO   | wrapper  | 2017/09/06 15:45:56 | Event Command 'wrapper_start': Command completed with exit code: 0  Continuing.

INFO   | wrapper  | 2017/09/06 15:45:56 | Event Command 'jvm_prelaunch': Command line[0] : /opt/vmware/bin/python

INFO   | wrapper  | 2017/09/06 15:45:56 | Event Command 'jvm_prelaunch': Command line[1] : ../wrapper/bin/update_pnid.py

INFO   | wrapper  | 2017/09/06 15:45:56 | Event Command 'jvm_prelaunch': Command line[2] :

INFO   | wrapper  | 2017/09/06 15:45:56 | Event Command 'jvm_prelaunch': Command line[3] :

INFO   | wrapper  | 2017/09/06 15:45:56 | Event Command 'jvm_prelaunch': Command line[4] :

INFO   | wrapper  | 2017/09/06 15:45:56 | Event Command 'jvm_prelaunch': Command launched (pid: 23313), blocking for up to 15 seconds...

INFO   | wrapper  | 2017/09/06 15:45:56 | Event Command 'jvm_prelaunch': Command completed with exit code: 1

INFO   | wrapper  | 2017/09/06 15:45:56 | Event Command 'jvm_prelaunch': Command completed with exit code: 1  Continuing.

STATUS | wrapper  | 2017/09/06 15:45:56 | Launching a JVM...

INFO   | jvm 1    | 2017/09/06 15:45:56 | Picked up JAVA_TOOL_OPTIONS: -Xms32M -Xmx128M

INFO   | jvm 1    | 2017/09/06 15:45:56 | WrapperManager: Initializing...

INFO   | jvm 1    | 2017/09/06 15:45:57 | WrapperSimpleApp:

INFO   | jvm 1    | 2017/09/06 15:45:57 | WrapperSimpleApp Error: Encountered an error running main:

INFO   | jvm 1    | 2017/09/06 15:45:57 | WrapperSimpleApp Error: org.springframework.beans.factory.BeanDefinitionStoreException: Invalid bean definition with name 'esxAgentMgr' defined in class path re                                   source [eam-server-beans.xml]: Could not resolve placeholder 'eam.hostd_restart_timeout'

INFO   | jvm 1    | 2017/09/06 15:45:57 | WrapperSimpleApp Error:       at org.springframework.beans.factory.config.PropertyPlaceholderConfigurer.processProperties(PropertyPlaceholderConfigurer.java:268                                   )

INFO   | jvm 1    | 2017/09/06 15:45:57 | WrapperSimpleApp Error:       at org.springframework.beans.factory.config.PropertyResourceConfigurer.postProcessBeanFactory(PropertyResourceConfigurer.java:75)

INFO   | jvm 1    | 2017/09/06 15:45:57 | WrapperSimpleApp Error:       at org.springframework.context.support.AbstractApplicationContext.invokeBeanFactoryPostProcessors(AbstractApplicationContext.java:                                   554)

INFO   | jvm 1    | 2017/09/06 15:45:57 | WrapperSimpleApp Error:       at org.springframework.context.support.AbstractApplicationContext.invokeBeanFactoryPostProcessors(AbstractApplicationContext.java:                                   528)

INFO   | jvm 1    | 2017/09/06 15:45:57 | WrapperSimpleApp Error:       at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:363)

INFO   | jvm 1    | 2017/09/06 15:45:57 | WrapperSimpleApp Error:       at org.springframework.context.support.ClassPathXmlApplicationContext.<init>(ClassPathXmlApplicationContext.java:139)

INFO   | jvm 1    | 2017/09/06 15:45:57 | WrapperSimpleApp Error:       at org.springframework.context.support.ClassPathXmlApplicationContext.<init>(ClassPathXmlApplicationContext.java:83)

INFO   | jvm 1    | 2017/09/06 15:45:57 | WrapperSimpleApp Error:       at com.vmware.eam.util.Main.main(Main.java:50)

INFO   | jvm 1    | 2017/09/06 15:45:57 | WrapperSimpleApp Error:       at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

INFO   | jvm 1    | 2017/09/06 15:45:57 | WrapperSimpleApp Error:       at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)

INFO   | jvm 1    | 2017/09/06 15:45:57 | WrapperSimpleApp Error:       at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)

INFO   | jvm 1    | 2017/09/06 15:45:57 | WrapperSimpleApp Error:       at java.lang.reflect.Method.invoke(Unknown Source)

INFO   | jvm 1    | 2017/09/06 15:45:57 | WrapperSimpleApp Error:       at org.tanukisoftware.wrapper.WrapperSimpleApp.run(WrapperSimpleApp.java:290)

INFO   | jvm 1    | 2017/09/06 15:45:57 | WrapperSimpleApp Error:       at java.lang.Thread.run(Unknown Source)

STATUS | wrapper  | 2017/09/06 15:45:59 | <-- Wrapper Stopped

So following my nose, a potentially relevant snip from the aforementioned eam-server-beans.xml found in /usr/lib/VMware-eam/conf is:

<!-- ESX Agent Manager beans -->

   <bean id="esxAgentMgr"

         class="com.vmware.eam.EsxAgentManagerImpl"

         lazy-init="false"

         depends-on="springContext,

                  vcInventory,

                  vcListener,

                  httpNfcLeaseTimeoutHandler,

                  nameManager,

                  vcDatastoreManager,

                  vcEventManager,

                  vcKeyValueStore,

                  l10n">

      <constructor-arg ref="adapterServer" />

      <constructor-arg value="${vc.proxy.host}" />

      <constructor-arg value="${vc.proxy.port}" />

      <constructor-arg value="${vc.tunnelSdkUri}" />

      <constructor-arg ref="clientAuthenticator" />

      <constructor-arg value="${eam.clear_db_on_startup}" />

      <constructor-arg value="${eam.support_linked_clone}" />

      <constructor-arg value="${eam.debug_ref_count}" />

      <constructor-arg value="${vum.integration}" />

      <constructor-arg value="${Net.DVFilterBindIpAddress}" />

      <constructor-arg value="${Net.TrafficFilterIpAddress}" />

      <constructor-arg value="${Net.DVFilterVswifIpAddress}" />

      <constructor-arg ref="commonMapStore" />

      <constructor-arg ref="vpxdConnection" />

      <constructor-arg ref="vumConnection" />

      <constructor-arg ref="nameManager" />

      <constructor-arg ref="vibPackageDepots" />

     <property name="hostdRestartTimeout" value="${eam.hostd_restart_timeout}" />

   </bean>

I'm struggling as I need to get my NSX piece back up and running (ironically so I can gracefully remove it), but believe I need the EAM to be working properly first.

Any ideas before I open a ticket?

Thanks.

vDave

10 Replies
RajeevVCP4
Expert
Expert

https://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=21223...

Rajeev Chauhan
VCIX-DCV6.5/VSAN/VXRAIL
Please mark help full or correct if my answer is use full for you
Reply
0 Kudos
virtual_dave1
Contributor
Contributor

Thanks for taking the time to post Rajeev.

But i already went through that article before posting my question.  None of the symptoms listed seem to apply to my problem shown in the logs above.

In a nutshell, my vmware-eam service on VCSA will not start.

Reply
0 Kudos
cliffcahill
Enthusiast
Enthusiast

I have seen similar before - Issue was Splunk was installed on the VM and was using the same port for forwarding etc.

Reply
0 Kudos
airfrog7
Enthusiast
Enthusiast

We have just had the exact same issue trying to upgrade vSphere 6.0U3 to 6.5. In our case the eam.properties file located at C:\ProgramData\VMware\vCenterServer\cfg\vmware-eam was completely blank. Luckily we had another vSphere server running 6.0U3, so we copied the contents of the eam.properties file from that server and pasted it in to the blank file on the faulty server. There is one URI at the bottom of the file you will need to change to match the hostname of the server. After that the Agent Manager service started OK.

Reply
0 Kudos
MarkSen
Contributor
Contributor

Hey,

By any chance, do you remember what you ended up doing to resolve the issue?

Thanks,

Reply
0 Kudos
Pattonville
Contributor
Contributor

What was the contents on your 6.0U3 eam.properties file you pasted in?

Having same issue with EAM service not starting, Invalid bean definition with name 'esxAgentMgr' defined

vcenter:~ # cat /etc/vmware-eam/eam.properties

eam.web.root=/usr/lib/vmware-eam/web

eam.web.root=/usr/lib/vmware-eam/web

the files eam.properties just has one line dupped twice in it and we have no other vcenter server to crib from.

Reply
0 Kudos
ImmedionChris
Contributor
Contributor

I just encountered this same issue and was able to resolve it by copying the contents of /etc/vmware-eam/eam.properties from a 6.0U3 appliance in my lab. I was then able to start the vmware-eam service and proceed with my 6.5 upgrade.

Below are the contents of the file I used. Maybe it will help someone else who doesn't have a spare appliance sitting around. Note that you need to change the the line vc.tunnelSdkUri=https://vcenter:8089/sdk/vimService to match your hostname.

#########################################################################

# Copyright 2013-2017 VMware, Inc.  All rights reserved. VMware Confidential

#########################################################################

vc.proxy.host=localhost

vc.proxy.port=80

# Hostname or IP of the EAM server

# Fill only if EAM is not running on the same host as VC

eam.host=

# EAM service port used to configure the HTTP connector of the application server.

eam.int.http.port=15005

# Port and scheme configuration which is used by the ESX 6.x hosts to reach EAM Vib

# file server.

eam.ext.port=443

eam.ext.scheme=https

# Port and scheme configuration which is used by the ESX 5.x hosts to reach EAM Vib

# file server.

eam.ext.port.deprecated=80

eam.ext.scheme.deprecated=http

eam.support_linked_clone=true

eam.clear_db_on_startup=false

eam.debug_ref_count=false

eam.recent_event_size=20

# Value is specified in minutes (set to 24h = 1440m)

eam.scan_for_unknown_agent_vms=1440

# The timeout to wait for hostd to restart on a host (set to 5m=300s)

eam.hostd_restart_timeout=300

# The following entries will be added verbatim to the advanced options

# of hosts on which EAM is enabled. All are optional.

Net.DVFilterBindIpAddress=169.254.0.1

Net.TrafficFilterIpAddress=

#The IP for the VSWIF NIC on the dvFilter switch (for ESX classic).

Net.DVFilterVswifIpAddress=169.254.0.2

# Resource bundle configuration

eam.resourcebundle.filename=eam-resourcebundle.jar

# VLSI embedded tcServer configuration

#

tcserver.tmp.dir=/var/tmp/vmware/eam/tomcat

eam.web.root=/usr/lib/vmware-eam/web

# EAM SSL configuration

#

eam.keystore.type=VKS

eam.key.alias=vpxd-extension

eam.keystore.storename=vpxd-extension

# CM configuration

#

cm.url=http://localhost:18090/cm/sdk/?hostid=fa3bd34a-4b3c-4ff0-ad4f-1c97ad6f93b5

cm.wait.attempts=360

cm.wait.intervalSeconds=5

# SSO configuration

#

sso.wait.attempts=360

sso.wait.intervalSeconds=5

# VC SSL configuration

#

vc.truststore.type=VKS

vc.truststore.storename=TRUSTED_ROOTS

vc.tunnelSdkUri.template=https://##{VC_HOST_NAME}##:8089/sdk/vimService

vc.tunnelSdkUri=https://vcenter:8089/sdk/vimService

drs.demandCapacityRatio=100

swalker_LST
Contributor
Contributor

This was really helpful for me as I just ran into the same problem which was also blocking my 6.5 upgrade. Through discussion with VMware - in addition, line 46 in your example should be updated to include your specific hostID obtained using:

cat /etc/vmware/install-defaults/sca.hostid

After that I found that an additional line was required before the service would start correctly:

eam.web.root=/usr/lib/vmware-eam/web

Finally, in case this was done using the root user you might find it necessary to set the permissions correctly for the properties file if it was deleted/recreated:

chown eam:cis eam.properties

chmod -R 644 eam.properties

Hope this helps if someone else runs into this issue during an upgrade..

SupOE
Contributor
Contributor

Attention when copy/paste this template 2 spaces are on end of lines , must be removed

"eam.keystore.type=VKS  "

"vc.truststore.type=VKS  "

sed to the rescue

sed -ire 's/  $//g' /etc/vmware-eam/eam.properties

on line 46:

  1. cm.url=http://localhost:18090/cm/sdk/?hostid=fa3bd34a-4b3c-4ff0-ad4f-1c97ad6f93b5 

change with your hostid with content of

cat /etc/vmware/install-defaults/sca.hostid

on line 24: (eameam is one eam only)

  1. eameam.resourcebundle.filename=eam-resourcebundle.jar 
  2. =>>
  3. eam.resourcebundle.filename=eam-resourcebundle.jar 
Reply
0 Kudos
offlinetn
Contributor
Contributor

I tired everything under the sun to get eam to run.  The chmown worked perfect.  I am no able to upgrade vcenter 6.0 to 6.7.

chown eam:cis eam.properties

Thanks,

Chris

Reply
0 Kudos