vm7user
Enthusiast
Enthusiast

Cannot upgrade VCSA 6.0 to 6.5 - Internal error occurs during Export of vSphere ESX Agent Manager

Jump to solution

Hello,

Cannot upgrade VCSA 6.0 to VCSA 6.5

always on stage 2 i get this error:


Error

Internal error occurs during Export of vSphere ESX Agent Manager.


Resolution

Send upgrade log files to VMware technical support team for further assistance.

i found article https://kb.vmware.com/kb/2147898, but this article applied only to vCenter Server 6.0 installed on Windows

1 Solution

Accepted Solutions
Charu
Enthusiast
Enthusiast

It is true that you cannot upgrade from vSphere 6.0 U3 to vSphere 6.5.  A new KB article has been published which explains the support policy for vSphere upgrade paths.  Please comment on this article if you have any additional questions. 

View solution in original post

48 Replies
muxonarization
Contributor
Contributor

Yes, I am not alone! See my findings:

Appliance 6.0U3 to 6.5a fails with Internal Error during Export of EAM

This even happens with a brend new VCVA6U3 when migrated to 65A.

0 Kudos
Xvor
Contributor
Contributor

Same here; 6.0u3 to 6.5 is a no go at the moment.

The external PSC is upgraded fine but vCenter component breaks at part 2 step 1.

I guess we shouldn't have patched to u3.

0 Kudos
YllowDnk
Enthusiast
Enthusiast

Hello,

same here, part 2 fails after 34 % with "Internal error occurs during Export of vSphere ESX Agent Manager"... Unable to upgrade 😞

0 Kudos
freeinthesky
Contributor
Contributor

yeah, exactly same error here Smiley Sad

"localized": "Internal error occurs during Export of vSphere ESX Agent Manager.",
            "translatable": "Internal error occurs during %(0)s of %(1)s."
        }
    ],
    "componentKey": "upgrade_framework",
    "problemId": null
},
"progress": 34
0 Kudos
Bouda0175
Enthusiast
Enthusiast

I have exactly same issue. I have VCSA 6.0 U3 tried to 6.5 A, but fails with the same error:

UpgradeFail.PNG

Problem is only with 6.0 U3. I tried upgrade on U2 and it works.Unfortunately, my production is on U3, so I am not able to upgrade.

My question is, when will be possible to do upgrade form 6.0 U3 to 6.5 A?

Why VMware doesn't support upgrade between latest versions of their major products?:smileyangry:

0 Kudos
freeinthesky
Contributor
Contributor

I tried both - 6.0 U3  to 6.5 and 6.0 U3  to 6.5a - same error. Looks like the problem is with 6.0 U3.

Still there,s no official word from VMware. Doesn't look that they plan to fix it...

0 Kudos
Harapng
Contributor
Contributor

Hi everyone.

Don't know if the solution is going to work for the appliance migration, but i was getting the same error during a windows vCenter 6.0U3 to appliance 6.5 migration and i solve it by:

1. Following the steps here Cannot remove or disable unwanted plug-ins from vCenter Server and vCenter Server Appliance (1025360...  to UnregisterExtension for the vSphere ESX Agent Manager.

2. On the windows vCenter machine, elevate a command prompt and execute from  C:\Program Files\VMware\vCenter Server\eam\wrapper\bin\ the UninstallApp-NT.bat to remove the ESX agent manager service.

3. Re-run the migration worked like a charm.

Hope it will work for you as well!

freeinthesky
Contributor
Contributor

Unfortunately  we use VCSA, so I could easy  remove Network & Security Plugin in  MOB, but can't  remove the ESX agent manager service.

That  .UninstallApp-NT.bat is available in Windows. Could not find any useful info on Google.

Actually I managed to remove eam(ESX Agent Manager) RPM file but service is still showing ( not running)

0 Kudos
Xvor
Contributor
Contributor

vCenter 6.5b is available, hopefully this resolves it:

VMware vSphere 6.5.0b Release Notes

vCenter Server upgrade might fail at pre-check phase of ESXi Agent Manager
ESXi Agent Manager might fail during the export phase. Failing at the export phase stops multiple services. This issue is resolved in this release by upgrade pre-check, without affecting vCenter Server or its health.

ESXi Agent Manager might fail during the upgrade with these possible symptoms:

  • EAM mob is not working.
  • The service is unresponsive on HTTP or API calls.
  • The user interface at Home > Administration > vCenter Server Extensions > vSphere ESX Agent Manager > Manage in vSphere Web client is not working.

If you encounter the following error, see http://kb.vmware.com/kb/2147898:
Internal error occurs during Export of vSphere ESX Agent Manager

This issue is resolved in this release.

0 Kudos
Xvor
Contributor
Contributor

UPDATE: just attempted VCSA u3 upgrade with 6.5b, still not working;

Same error.

0 Kudos
Bouda0175
Enthusiast
Enthusiast

UPDATE: just attempted VCSA u3 upgrade with 6.5b, still not working;

Same error.

I was so happy about this update, that upgrade process from 6.0 U3 is fixed. But when I read your post, I am not sure, If I should try upgrade porcedure on my infrastructure. I gues, that I would ended with the same result.

0 Kudos
jnewton03
Contributor
Contributor

I was able to get this working yesterday with 6.5a.  I have not yet tried 6.5b.

Here is what I did:

To remove unwanted plug-ins from the available plug-in list:

  1. In a web browser, navigate to http://vCenter_Server_name_or_IP/mob.

    Where vCenter_Server_name_or_IP/mob is the name of your vCenter Server or its IP address.

  2. Click Content.
  3. Click ExtensionManager.
  4. Select and copy the name of the plug-in you want to remove from the list of values under Properties. For a list of default plug-ins, see the Additional Information section of this article.
  5. Click UnregisterExtension. A new window appears.
  6. Paste the key of the plug-in and click Invoke Method. This removes the plug-in and results in void.  (I used com.vmware.vim.eam)
  7. Close the window.
  8. Refresh the Managed Object Type:ManagedObjectReference:ExtensionManager window to verify that the plug-in is removed successfully.
I then stopped the VMWare-EAM service on the source appliance by doing the following:

1.  SSH into the source appliance

2.  Go into shell

3.  issue a  service-control --stop vmware-eam

I then re-ran the upgrade and was successful.  Note that doing just one or the other of the above steps may have worked, but I have not tested it.

Jeriko2k
Contributor
Contributor

Just for fun, someone have done a test from a clean 6.0u3 to 6.5b? for me 6.5b fixed nothing also, same bug.

0 Kudos
freeinthesky
Contributor
Contributor

Was vmware-eam running before you stopped it ? Is it running now ?

In my case it's not starting - when I issue service-control --start vmware-eam  I get this error:

Starting VMware ESX Agent Manager...

WARN   | wrapper  | The 'EAM_WEB_DIR' environment variable was referenced but has not been defined.

WARN   | wrapper  | The 'EAM_WRAPPER_ADDITIONAL_FILE' environment variable was referenced but has not been defined.

Waiting for VMware ESX Agent Manager..................

WARNING: VMware ESX Agent Manager may have failed to start.

Last login: Tue Mar 14 15:25:28 NZDT 2017

Stderr = su: warning: cannot change directory to /home/eam: Permission denied

-bash: /home/eam/.bash_profile: Permission denied

FATAL  | wrapper  | Unable to resolve the working directory %EAM_WEB_DIR%: No such file or directory

FATAL  | wrapper  | Unable to set working directory to: /usr/lib/vmware-eam/wrapper/bin/%EAM_WEB_DIR% (No such file or directory)

0 Kudos
jnewton03
Contributor
Contributor

I did not check to see if it was running first.  This is an error you're getting on the source appliance running 6.0U3?  Are you using custom ports?  What does your eam.properties look like?

0 Kudos
freeinthesky
Contributor
Contributor

Correct: 6.0U3

I don't see any particular configuration in eam.properties (please see below)(maybe something is missing):

-------------------------------------------------------------------------------------------------------------------------------------------------------

#########################################################################

# Copyright 2013-2017 VMware, Inc.  All rights reserved. VMware Confidential

#########################################################################

vc.proxy.host=localhost

vc.proxy.port=##REVERSE_PROXY_HTTP_PORT##

# Hostname or IP of the EAM server

# Fill only if EAM is not running on the same host as VC

eam.host=

# EAM service port used to configure the HTTP connector of the application server.

eam.int.http.port=##EAM_INTERNAL_HTTP_PORT##

# Port and scheme configuration which is used by the ESX 6.x hosts to reach EAM Vib

# file server.

eam.ext.port=##EAM_EXTERNAL_PORT##

eam.ext.scheme=https

# Port and scheme configuration which is used by the ESX 5.x hosts to reach EAM Vib

# file server.

eam.ext.port.deprecated=##EAM_EXTERNAL_PORT_DEPRECATED##

eam.ext.scheme.deprecated=http

eam.support_linked_clone=true

eam.clear_db_on_startup=false

eam.debug_ref_count=false

eam.recent_event_size=20

# Value is specified in minutes (set to 24h = 1440m)

eam.scan_for_unknown_agent_vms=1440

# The timeout to wait for hostd to restart on a host (set to 5m=300s)

eam.hostd_restart_timeout=300

# The following entries will be added verbatim to the advanced options

# of hosts on which EAM is enabled. All are optional.

Net.DVFilterBindIpAddress=169.254.0.1

Net.TrafficFilterIpAddress=

#The IP for the VSWIF NIC on the dvFilter switch (for ESX classic).

Net.DVFilterVswifIpAddress=169.254.0.2

# Resource bundle configuration

eam.resourcebundle.filename=eam-resourcebundle.jar

# VLSI embedded tcServer configuration

#

tcserver.tmp.dir=##{TCSERVER_TMP_DIR}##

eam.web.root=##{EAM_WEB_ROOT}##

# EAM SSL configuration

#

eam.keystore.type=VKS

eam.key.alias=##{SSL_ALIAS}##

eam.keystore.storename=##{SSL_STORE_NAME}##

# CM configuration

#

cm.url=##{CM_URL}##

cm.wait.attempts=360

cm.wait.intervalSeconds=5

# SSO configuration

#

sso.wait.attempts=360

sso.wait.intervalSeconds=5

# VC SSL configuration

#

vc.truststore.type=VKS

vc.truststore.storename=TRUSTED_ROOTS

vc.tunnelSdkUri.template=https://##{VC_HOST_NAME}##:##{VC_SDK_TUNNEL_PORT}##/sdk/vimService

vc.tunnelSdkUri=

drs.demandCapacityRatio=100

eam.web.root=/usr/lib/vmware-eam/web

0 Kudos
Xvor
Contributor
Contributor

Not quite sure what to make of this but the VMware Product Interoperability Matrices now shows 6.0 U3 as incompatible for upgrade to 6.5.0 because: " 'back in time' upgrade not supported (vSphere 6.5.0 released 5 months before vSphere 6.0 U3)".

Great, so now what? We have to wait for 6.5 U1?

0 Kudos
kondrichRHI
Enthusiast
Enthusiast

We are also encountering this issue when migrating from vCenter Windows 6.0u3 to VCSA 6.5b.

UpgradeRunner.log shows:

"

2017-03-20T12:21:05.404Z INFO upgrade.states.component_states eam:Export: 2017-03-20T12:21:04.381Z INFO eam Going to access: http://localhost:80/eam/charset.jsp

2017-03-20T12:21:05.404Z INFO upgrade.states.component_states eam:Export: 2017-03-20T12:21:04.391Z WARNING eam vmware-eam service is not fully started, waiting till get up and running. Sleeping for5seconds. Error: HTTP Error 500: Internal Server Error

2017-03-20T12:21:06.410Z INFO upgrade.states.component_states eam:Export: didn't have progress. Next report will come after 60 seconds

"

I figured out that http://localhost:80/eam/charset.jsp gives an error when called on the source server in Internet Explorer:

"HTTP Status 500 - Unable to compile class for JSP

type Exception report

message Unable to compile class for JSP

description The server encountered an internal error that prevented it from fulfilling this request.

exception

org.apache.jasper.JasperException: Unable to compile class for JSP

  org.apache.jasper.JspCompilationContext.compile(JspCompilationContext.java:600)

  org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:363)

  org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:396)

  org.apache.jasper.servlet.JspServlet.service(JspServlet.java:340)

  javax.servlet.http.HttpServlet.service(HttpServlet.java:729)

root cause

java.lang.NullPointerException

  org.apache.jasper.compiler.Validator$ValidateVisitor.(Validator.java:515)

  org.apache.jasper.compiler.Validator.validateExDirectives(Validator.java:1853)

  org.apache.jasper.compiler.Compiler.generateJava(Compiler.java:217)

  org.apache.jasper.compiler.Compiler.compile(Compiler.java:356)

  org.apache.jasper.compiler.Compiler.compile(Compiler.java:336)

  org.apache.jasper.compiler.Compiler.compile(Compiler.java:323)

  org.apache.jasper.JspCompilationContext.compile(JspCompilationContext.java:585)

  org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:363)

  org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:396)

  org.apache.jasper.servlet.JspServlet.service(JspServlet.java:340)

  javax.servlet.http.HttpServlet.service(HttpServlet.java:729)

note The full stack trace of the root cause is available in the Pivotal tc Runtime 3.1.5.RELEASE/8.0.36.A.RELEASE logs."

I guess that's the n+1st problem with migration. 😞 Let's see what VMware support knows about that.

UPDATE:

I just figured out another issue:

Calling http://localhost/eam/index.jsp on an vCenter 6.0u3 (5112527) instance gives the mentioned JSP error.

Calling http://localhost/eam/index.jsp on an 6.0u2a (4541947) instance correctly gives "The vSphere ESX Agent Manager (vEAM) is successfully deployed.". Btw, charset.jsp is not existing on a 6.0u2a instance.

I verified this on different vCenter installations in our organization.

So it seems that there is a general 6.0u3 problem with EAM.

0 Kudos
jpiscaer
Enthusiast
Enthusiast

I can confirm; http://vcsa-ip:80/eam/charset.jsp throws an error.

Cheers, Joep Piscaer VMware vExpert 2009 Virtual Lifestyle: http://www.virtuallifestyle.nl Twitter: http://www.twitter.com/jpiscaer LinkedIn: http://www.linkedin.com/in/jpiscaer **If you found this information useful, please consider awarding points**
0 Kudos