8 Replies Latest reply on Mar 29, 2018 6:27 AM by swalker_LST

    vSphere 6.0U3 EAM Error

    virtual_dave Novice

      Hi all

       

      I have a problem with EAM.  We have been using NSX guest introspection and it started playing up recently, which attracted our attention.

       

      In vSphere Web Client, under Administration\vSphere ESX Agent Manager \ Solution I get the following error:

       

      503 Service Unavailable (Failed to connect to endpoint: [N7Vmacore4Http16LocalServiceSpecE:0x7fdc140fea00] _serverNamespace = /eam _isRedirect = false _port = 15005)

       

      On VCSA, the VMware-eam service is not running.  Starting it results in an error, the details of which are found in /var/log/VMware-eam/wrapper.log

       

      STATUS | wrapper  | 2017/09/06 15:45:55 | --> Wrapper Started as Daemon

      STATUS | wrapper  | 2017/09/06 15:45:55 | Java Service Wrapper Professional Edition 64-bit 3.5.24

      STATUS | wrapper  | 2017/09/06 15:45:55 |   Copyright (C) 1999-2014 Tanuki Software, Ltd. All Rights Reserved.

      STATUS | wrapper  | 2017/09/06 15:45:55 |     http://wrapper.tanukisoftware.com

      STATUS | wrapper  | 2017/09/06 15:45:55 |   Licensed to VMware Global, Inc. for VMware vCenter ESX Agent Manager

      STATUS | wrapper  | 2017/09/06 15:45:55 |

      INFO   | wrapper  | 2017/09/06 15:45:55 | Event Command 'wrapper_start': Command line[0] : /usr/sbin/cloudvm-ram-size

      INFO   | wrapper  | 2017/09/06 15:45:55 | Event Command 'wrapper_start': Command line[1] : -J

      INFO   | wrapper  | 2017/09/06 15:45:55 | Event Command 'wrapper_start': Command line[2] : vmware-eam

      INFO   | wrapper  | 2017/09/06 15:45:55 | Event Command 'wrapper_start': Command line[3] : -O

      INFO   | wrapper  | 2017/09/06 15:45:55 | Event Command 'wrapper_start': Command line[4] : /etc/vmware/java/vmware-eam_jvm.conf

      INFO   | wrapper  | 2017/09/06 15:45:55 | Event Command 'wrapper_start': Command launched (pid: 23310), blocking for up to 3600 seconds...

      INFO   | wrapper  | 2017/09/06 15:45:56 | Event Command 'wrapper_start': Command completed with exit code: 0

      INFO   | wrapper  | 2017/09/06 15:45:56 | Event Command 'wrapper_start': Command completed with exit code: 0  Continuing.

      INFO   | wrapper  | 2017/09/06 15:45:56 | Event Command 'jvm_prelaunch': Command line[0] : /opt/vmware/bin/python

      INFO   | wrapper  | 2017/09/06 15:45:56 | Event Command 'jvm_prelaunch': Command line[1] : ../wrapper/bin/update_pnid.py

      INFO   | wrapper  | 2017/09/06 15:45:56 | Event Command 'jvm_prelaunch': Command line[2] :

      INFO   | wrapper  | 2017/09/06 15:45:56 | Event Command 'jvm_prelaunch': Command line[3] :

      INFO   | wrapper  | 2017/09/06 15:45:56 | Event Command 'jvm_prelaunch': Command line[4] :

      INFO   | wrapper  | 2017/09/06 15:45:56 | Event Command 'jvm_prelaunch': Command launched (pid: 23313), blocking for up to 15 seconds...

      INFO   | wrapper  | 2017/09/06 15:45:56 | Event Command 'jvm_prelaunch': Command completed with exit code: 1

      INFO   | wrapper  | 2017/09/06 15:45:56 | Event Command 'jvm_prelaunch': Command completed with exit code: 1  Continuing.

      STATUS | wrapper  | 2017/09/06 15:45:56 | Launching a JVM...

      INFO   | jvm 1    | 2017/09/06 15:45:56 | Picked up JAVA_TOOL_OPTIONS: -Xms32M -Xmx128M

      INFO   | jvm 1    | 2017/09/06 15:45:56 | WrapperManager: Initializing...

      INFO   | jvm 1    | 2017/09/06 15:45:57 | WrapperSimpleApp:

      INFO   | jvm 1    | 2017/09/06 15:45:57 | WrapperSimpleApp Error: Encountered an error running main:

      INFO   | jvm 1    | 2017/09/06 15:45:57 | WrapperSimpleApp Error: org.springframework.beans.factory.BeanDefinitionStoreException: Invalid bean definition with name 'esxAgentMgr' defined in class path re                                   source [eam-server-beans.xml]: Could not resolve placeholder 'eam.hostd_restart_timeout'

      INFO   | jvm 1    | 2017/09/06 15:45:57 | WrapperSimpleApp Error:       at org.springframework.beans.factory.config.PropertyPlaceholderConfigurer.processProperties(PropertyPlaceholderConfigurer.java:268                                   )

      INFO   | jvm 1    | 2017/09/06 15:45:57 | WrapperSimpleApp Error:       at org.springframework.beans.factory.config.PropertyResourceConfigurer.postProcessBeanFactory(PropertyResourceConfigurer.java:75)

      INFO   | jvm 1    | 2017/09/06 15:45:57 | WrapperSimpleApp Error:       at org.springframework.context.support.AbstractApplicationContext.invokeBeanFactoryPostProcessors(AbstractApplicationContext.java:                                   554)

      INFO   | jvm 1    | 2017/09/06 15:45:57 | WrapperSimpleApp Error:       at org.springframework.context.support.AbstractApplicationContext.invokeBeanFactoryPostProcessors(AbstractApplicationContext.java:                                   528)

      INFO   | jvm 1    | 2017/09/06 15:45:57 | WrapperSimpleApp Error:       at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:363)

      INFO   | jvm 1    | 2017/09/06 15:45:57 | WrapperSimpleApp Error:       at org.springframework.context.support.ClassPathXmlApplicationContext.<init>(ClassPathXmlApplicationContext.java:139)

      INFO   | jvm 1    | 2017/09/06 15:45:57 | WrapperSimpleApp Error:       at org.springframework.context.support.ClassPathXmlApplicationContext.<init>(ClassPathXmlApplicationContext.java:83)

      INFO   | jvm 1    | 2017/09/06 15:45:57 | WrapperSimpleApp Error:       at com.vmware.eam.util.Main.main(Main.java:50)

      INFO   | jvm 1    | 2017/09/06 15:45:57 | WrapperSimpleApp Error:       at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

      INFO   | jvm 1    | 2017/09/06 15:45:57 | WrapperSimpleApp Error:       at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)

      INFO   | jvm 1    | 2017/09/06 15:45:57 | WrapperSimpleApp Error:       at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)

      INFO   | jvm 1    | 2017/09/06 15:45:57 | WrapperSimpleApp Error:       at java.lang.reflect.Method.invoke(Unknown Source)

      INFO   | jvm 1    | 2017/09/06 15:45:57 | WrapperSimpleApp Error:       at org.tanukisoftware.wrapper.WrapperSimpleApp.run(WrapperSimpleApp.java:290)

      INFO   | jvm 1    | 2017/09/06 15:45:57 | WrapperSimpleApp Error:       at java.lang.Thread.run(Unknown Source)

      STATUS | wrapper  | 2017/09/06 15:45:59 | <-- Wrapper Stopped

       

      So following my nose, a potentially relevant snip from the aforementioned eam-server-beans.xml found in /usr/lib/VMware-eam/conf is:

       

      <!-- ESX Agent Manager beans -->

         <bean id="esxAgentMgr"

               class="com.vmware.eam.EsxAgentManagerImpl"

               lazy-init="false"

               depends-on="springContext,

                        vcInventory,

                        vcListener,

                        httpNfcLeaseTimeoutHandler,

                        nameManager,

                        vcDatastoreManager,

                        vcEventManager,

                        vcKeyValueStore,

                        l10n">

            <constructor-arg ref="adapterServer" />

            <constructor-arg value="${vc.proxy.host}" />

            <constructor-arg value="${vc.proxy.port}" />

            <constructor-arg value="${vc.tunnelSdkUri}" />

            <constructor-arg ref="clientAuthenticator" />

            <constructor-arg value="${eam.clear_db_on_startup}" />

            <constructor-arg value="${eam.support_linked_clone}" />

            <constructor-arg value="${eam.debug_ref_count}" />

            <constructor-arg value="${vum.integration}" />

            <constructor-arg value="${Net.DVFilterBindIpAddress}" />

            <constructor-arg value="${Net.TrafficFilterIpAddress}" />

            <constructor-arg value="${Net.DVFilterVswifIpAddress}" />

            <constructor-arg ref="commonMapStore" />

            <constructor-arg ref="vpxdConnection" />

            <constructor-arg ref="vumConnection" />

            <constructor-arg ref="nameManager" />

            <constructor-arg ref="vibPackageDepots" />

           <property name="hostdRestartTimeout" value="${eam.hostd_restart_timeout}" />

         </bean>

       

      I'm struggling as I need to get my NSX piece back up and running (ironically so I can gracefully remove it), but believe I need the EAM to be working properly first.

       

      Any ideas before I open a ticket?

       

      Thanks.

       

      vDave

       

        • 2. Re: vSphere 6.0U3 EAM Error
          virtual_dave Novice

          Thanks for taking the time to post Rajeev.

           

          But i already went through that article before posting my question.  None of the symptoms listed seem to apply to my problem shown in the logs above.

           

          In a nutshell, my vmware-eam service on VCSA will not start.

          • 3. Re: vSphere 6.0U3 EAM Error
            cliffcahill Enthusiast

            I have seen similar before - Issue was Splunk was installed on the VM and was using the same port for forwarding etc.

            • 4. Re: vSphere 6.0U3 EAM Error
              airfrog7 Novice

              We have just had the exact same issue trying to upgrade vSphere 6.0U3 to 6.5. In our case the eam.properties file located at C:\ProgramData\VMware\vCenterServer\cfg\vmware-eam was completely blank. Luckily we had another vSphere server running 6.0U3, so we copied the contents of the eam.properties file from that server and pasted it in to the blank file on the faulty server. There is one URI at the bottom of the file you will need to change to match the hostname of the server. After that the Agent Manager service started OK.

              • 5. Re: vSphere 6.0U3 EAM Error
                MarkSen Lurker

                Hey,

                 

                By any chance, do you remember what you ended up doing to resolve the issue?

                 

                Thanks,

                • 6. Re: vSphere 6.0U3 EAM Error
                  Pattonville Lurker

                  What was the contents on your 6.0U3 eam.properties file you pasted in?

                   

                  Having same issue with EAM service not starting, Invalid bean definition with name 'esxAgentMgr' defined

                   

                  vcenter:~ # cat /etc/vmware-eam/eam.properties

                  eam.web.root=/usr/lib/vmware-eam/web

                  eam.web.root=/usr/lib/vmware-eam/web

                   

                  the files eam.properties just has one line dupped twice in it and we have no other vcenter server to crib from.

                  • 7. Re: vSphere 6.0U3 EAM Error
                    ImmedionChris Novice

                    I just encountered this same issue and was able to resolve it by copying the contents of /etc/vmware-eam/eam.properties from a 6.0U3 appliance in my lab. I was then able to start the vmware-eam service and proceed with my 6.5 upgrade.

                     

                    Below are the contents of the file I used. Maybe it will help someone else who doesn't have a spare appliance sitting around. Note that you need to change the the line vc.tunnelSdkUri=https://vcenter:8089/sdk/vimService to match your hostname.

                     

                     

                    #########################################################################
                    # Copyright 2013-2017 VMware, Inc.  All rights reserved. VMware Confidential
                    #########################################################################
                    vc.proxy.host=localhost
                    vc.proxy.port=80
                    # Hostname or IP of the EAM server
                    # Fill only if EAM is not running on the same host as VC
                    eam.host=
                    # EAM service port used to configure the HTTP connector of the application server.
                    eam.int.http.port=15005
                    # Port and scheme configuration which is used by the ESX 6.x hosts to reach EAM Vib
                    # file server.
                    eam.ext.port=443
                    eam.ext.scheme=https
                    # Port and scheme configuration which is used by the ESX 5.x hosts to reach EAM Vib
                    # file server.
                    eam.ext.port.deprecated=80
                    eam.ext.scheme.deprecated=http
                    eam.support_linked_clone=true
                    eam.clear_db_on_startup=false
                    eam.debug_ref_count=false
                    eam.recent_event_size=20
                    # Value is specified in minutes (set to 24h = 1440m)
                    eam.scan_for_unknown_agent_vms=1440
                    # The timeout to wait for hostd to restart on a host (set to 5m=300s)
                    eam.hostd_restart_timeout=300
                    # The following entries will be added verbatim to the advanced options
                    # of hosts on which EAM is enabled. All are optional.
                    Net.DVFilterBindIpAddress=169.254.0.1
                    Net.TrafficFilterIpAddress=
                    #The IP for the VSWIF NIC on the dvFilter switch (for ESX classic).
                    Net.DVFilterVswifIpAddress=169.254.0.2
                    # Resource bundle configuration
                    eam.resourcebundle.filename=eam-resourcebundle.jar
                    # VLSI embedded tcServer configuration
                    #
                    tcserver.tmp.dir=/var/tmp/vmware/eam/tomcat
                    eam.web.root=/usr/lib/vmware-eam/web
                    # EAM SSL configuration
                    #
                    eam.keystore.type=VKS
                    eam.key.alias=vpxd-extension
                    eam.keystore.storename=vpxd-extension
                    # CM configuration
                    #
                    cm.url=http://localhost:18090/cm/sdk/?hostid=fa3bd34a-4b3c-4ff0-ad4f-1c97ad6f93b5
                    cm.wait.attempts=360
                    cm.wait.intervalSeconds=5
                    # SSO configuration
                    #
                    sso.wait.attempts=360
                    sso.wait.intervalSeconds=5
                    # VC SSL configuration
                    #
                    vc.truststore.type=VKS
                    vc.truststore.storename=TRUSTED_ROOTS
                    vc.tunnelSdkUri.template=https://##{VC_HOST_NAME}##:8089/sdk/vimService
                    vc.tunnelSdkUri=https://vcenter:8089/sdk/vimService
                    drs.demandCapacityRatio=100
                    
                    1 person found this helpful
                    • 8. Re: vSphere 6.0U3 EAM Error
                      swalker_LST Novice

                      This was really helpful for me as I just ran into the same problem which was also blocking my 6.5 upgrade. Through discussion with VMware - in addition, line 46 in your example should be updated to include your specific hostID obtained using:

                      cat /etc/vmware/install-defaults/sca.hostid

                      After that I found that an additional line was required before the service would start correctly:

                       

                      eam.web.root=/usr/lib/vmware-eam/web
                      

                       

                      Finally, in case this was done using the root user you might find it necessary to set the permissions correctly for the properties file if it was deleted/recreated:

                      chown eam:cis eam.properties

                      chmod -R 644 eam.properties

                      Hope this helps if someone else runs into this issue during an upgrade..

                      1 person found this helpful