Hi all

I have a problem with EAM.  We have been using NSX guest introspection and it started playing up recently, which attracted our attention.

In vSphere Web Client, under Administration\vSphere ESX Agent Manager \ Solution I get the following error:

503 Service Unavailable (Failed to connect to endpoint: [N7Vmacore4Http16LocalServiceSpecE:0x7fdc140fea00] _serverNamespace = /eam _isRedirect = false _port = 15005)

On VCSA, the VMware-eam service is not running.  Starting it results in an error, the details of which are found in /var/log/VMware-eam/wrapper.log

STATUS | wrapper  | 2017/09/06 15:45:55 | --> Wrapper Started as Daemon

STATUS | wrapper  | 2017/09/06 15:45:55 | Java Service Wrapper Professional Edition 64-bit 3.5.24

STATUS | wrapper  | 2017/09/06 15:45:55 |   Copyright (C) 1999-2014 Tanuki Software, Ltd. All Rights Reserved.

STATUS | wrapper  | 2017/09/06 15:45:55 |     http://wrapper.tanukisoftware.com

STATUS | wrapper  | 2017/09/06 15:45:55 |   Licensed to VMware Global, Inc. for VMware vCenter ESX Agent Manager

STATUS | wrapper  | 2017/09/06 15:45:55 |

INFO   | wrapper  | 2017/09/06 15:45:55 | Event Command 'wrapper_start': Command line[0] : /usr/sbin/cloudvm-ram-size

INFO   | wrapper  | 2017/09/06 15:45:55 | Event Command 'wrapper_start': Command line[1] : -J

INFO   | wrapper  | 2017/09/06 15:45:55 | Event Command 'wrapper_start': Command line[2] : vmware-eam

INFO   | wrapper  | 2017/09/06 15:45:55 | Event Command 'wrapper_start': Command line[3] : -O

INFO   | wrapper  | 2017/09/06 15:45:55 | Event Command 'wrapper_start': Command line[4] : /etc/vmware/java/vmware-eam_jvm.conf

INFO   | wrapper  | 2017/09/06 15:45:55 | Event Command 'wrapper_start': Command launched (pid: 23310), blocking for up to 3600 seconds...

INFO   | wrapper  | 2017/09/06 15:45:56 | Event Command 'wrapper_start': Command completed with exit code: 0

INFO   | wrapper  | 2017/09/06 15:45:56 | Event Command 'wrapper_start': Command completed with exit code: 0  Continuing.

INFO   | wrapper  | 2017/09/06 15:45:56 | Event Command 'jvm_prelaunch': Command line[0] : /opt/vmware/bin/python

INFO   | wrapper  | 2017/09/06 15:45:56 | Event Command 'jvm_prelaunch': Command line[1] : ../wrapper/bin/update_pnid.py

INFO   | wrapper  | 2017/09/06 15:45:56 | Event Command 'jvm_prelaunch': Command line[2] :

INFO   | wrapper  | 2017/09/06 15:45:56 | Event Command 'jvm_prelaunch': Command line[3] :

INFO   | wrapper  | 2017/09/06 15:45:56 | Event Command 'jvm_prelaunch': Command line[4] :

INFO   | wrapper  | 2017/09/06 15:45:56 | Event Command 'jvm_prelaunch': Command launched (pid: 23313), blocking for up to 15 seconds...

INFO   | wrapper  | 2017/09/06 15:45:56 | Event Command 'jvm_prelaunch': Command completed with exit code: 1

INFO   | wrapper  | 2017/09/06 15:45:56 | Event Command 'jvm_prelaunch': Command completed with exit code: 1  Continuing.

STATUS | wrapper  | 2017/09/06 15:45:56 | Launching a JVM...

INFO   | jvm 1    | 2017/09/06 15:45:56 | Picked up JAVA_TOOL_OPTIONS: -Xms32M -Xmx128M

INFO   | jvm 1    | 2017/09/06 15:45:56 | WrapperManager: Initializing...

INFO   | jvm 1    | 2017/09/06 15:45:57 | WrapperSimpleApp:

INFO   | jvm 1    | 2017/09/06 15:45:57 | WrapperSimpleApp Error: Encountered an error running main:

INFO   | jvm 1    | 2017/09/06 15:45:57 | WrapperSimpleApp Error: org.springframework.beans.factory.BeanDefinitionStoreException: Invalid bean definition with name 'esxAgentMgr' defined in class path re                                   source [eam-server-beans.xml]: Could not resolve placeholder 'eam.hostd_restart_timeout'

INFO   | jvm 1    | 2017/09/06 15:45:57 | WrapperSimpleApp Error:       at org.springframework.beans.factory.config.PropertyPlaceholderConfigurer.processProperties(PropertyPlaceholderConfigurer.java:268                                   )

INFO   | jvm 1    | 2017/09/06 15:45:57 | WrapperSimpleApp Error:       at org.springframework.beans.factory.config.PropertyResourceConfigurer.postProcessBeanFactory(PropertyResourceConfigurer.java:75)

INFO   | jvm 1    | 2017/09/06 15:45:57 | WrapperSimpleApp Error:       at org.springframework.context.support.AbstractApplicationContext.invokeBeanFactoryPostProcessors(AbstractApplicationContext.java:                                   554)

INFO   | jvm 1    | 2017/09/06 15:45:57 | WrapperSimpleApp Error:       at org.springframework.context.support.AbstractApplicationContext.invokeBeanFactoryPostProcessors(AbstractApplicationContext.java:                                   528)

INFO   | jvm 1    | 2017/09/06 15:45:57 | WrapperSimpleApp Error:       at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:363)

INFO   | jvm 1    | 2017/09/06 15:45:57 | WrapperSimpleApp Error:       at org.springframework.context.support.ClassPathXmlApplicationContext.<init>(ClassPathXmlApplicationContext.java:139)

INFO   | jvm 1    | 2017/09/06 15:45:57 | WrapperSimpleApp Error:       at org.springframework.context.support.ClassPathXmlApplicationContext.<init>(ClassPathXmlApplicationContext.java:83)

INFO   | jvm 1    | 2017/09/06 15:45:57 | WrapperSimpleApp Error:       at com.vmware.eam.util.Main.main(Main.java:50)

INFO   | jvm 1    | 2017/09/06 15:45:57 | WrapperSimpleApp Error:       at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

INFO   | jvm 1    | 2017/09/06 15:45:57 | WrapperSimpleApp Error:       at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)

INFO   | jvm 1    | 2017/09/06 15:45:57 | WrapperSimpleApp Error:       at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)

INFO   | jvm 1    | 2017/09/06 15:45:57 | WrapperSimpleApp Error:       at java.lang.reflect.Method.invoke(Unknown Source)

INFO   | jvm 1    | 2017/09/06 15:45:57 | WrapperSimpleApp Error:       at org.tanukisoftware.wrapper.WrapperSimpleApp.run(WrapperSimpleApp.java:290)

INFO   | jvm 1    | 2017/09/06 15:45:57 | WrapperSimpleApp Error:       at java.lang.Thread.run(Unknown Source)

STATUS | wrapper  | 2017/09/06 15:45:59 | <-- Wrapper Stopped

So following my nose, a potentially relevant snip from the aforementioned eam-server-beans.xml found in /usr/lib/VMware-eam/conf is:

<!-- ESX Agent Manager beans -->

   <bean id="esxAgentMgr"

         class="com.vmware.eam.EsxAgentManagerImpl"

         lazy-init="false"

         depends-on="springContext,

                  vcInventory,

                  vcListener,

                  httpNfcLeaseTimeoutHandler,

                  nameManager,

                  vcDatastoreManager,

                  vcEventManager,

                  vcKeyValueStore,

                  l10n">

      <constructor-arg ref="adapterServer" />

      <constructor-arg value="${vc.proxy.host}" />

      <constructor-arg value="${vc.proxy.port}" />

      <constructor-arg value="${vc.tunnelSdkUri}" />

      <constructor-arg ref="clientAuthenticator" />

      <constructor-arg value="${eam.clear_db_on_startup}" />

      <constructor-arg value="${eam.support_linked_clone}" />

      <constructor-arg value="${eam.debug_ref_count}" />

      <constructor-arg value="${vum.integration}" />

      <constructor-arg value="${Net.DVFilterBindIpAddress}" />

      <constructor-arg value="${Net.TrafficFilterIpAddress}" />

      <constructor-arg value="${Net.DVFilterVswifIpAddress}" />

      <constructor-arg ref="commonMapStore" />

      <constructor-arg ref="vpxdConnection" />

      <constructor-arg ref="vumConnection" />

      <constructor-arg ref="nameManager" />

      <constructor-arg ref="vibPackageDepots" />

     <property name="hostdRestartTimeout" value="${eam.hostd_restart_timeout}" />

   </bean>

I'm struggling as I need to get my NSX piece back up and running (ironically so I can gracefully remove it), but believe I need the EAM to be working properly first.

Any ideas before I open a ticket?

Thanks.

vDave