VDP Plugin not loading in vSphere Web Client

[Dasutin]

I'm running VDP 6.1.2 and vCenter Appliance 6 U2.  When I try to launch the VDP plugin from the vSphere Client, I get the error "Unable to connect to the requested VDP Appliance.".  The logs in VDP look fine but when I check the virgo log in vCenter I get this error below.  I tried removing the plugin in mob and re-adding it, and uninstalling VDP completely.  The problem still remains.  This has happen with VDP 6.0 also.  Figure I would post it here since I can't find others with this error and nothing I do can fix it.

[2016-04-13T19:34:34.385-04:00] [INFO ] http-bio-9443-exec-11         org.springframework.flex.servlet.MessageBrokerHandlerAdapter      Channel endpoint vdp-amf received request. 
[2016-04-13T19:34:34.453-04:00] [ERROR] http-bio-9443-exec-11         com.emc.vdp2.api.impl.ActionApi                                   Trying to connect to: 192.168.***.*** 
[2016-04-13T19:34:34.527-04:00] [WARN ] http-bio-9443-exec-11         org.springframework.flex.core.DefaultExceptionLogger              The following exception occurred during request processing by the BlazeDS MessageBroker and will be serialized back to the client:  flex
.messaging.MessageException: java.lang.NoClassDefFoundError : Could not initialize class com.sun.jersey.core.header.MediaTypes
        at flex.messaging.services.remoting.adapters.JavaAdapter.invoke(JavaAdapter.java:444)
        at com.vmware.vise.messaging.remoting.JavaAdapterEx.invoke(JavaAdapterEx.java:50)
        at flex.messaging.services.RemotingService.serviceMessage(RemotingService.java:183)
        at flex.messaging.MessageBroker.routeMessageToService(MessageBroker.java:1400)
        at flex.messaging.endpoints.AbstractEndpoint.serviceMessage(AbstractEndpoint.java:1011)
        at flex.messaging.endpoints.AbstractEndpoint$$FastClassByCGLIB$$1a3ef066.invoke(<generated>)
        at net.sf.cglib.proxy.MethodProxy.invoke(MethodProxy.java:149)
        at org.springframework.aop.framework.Cglib2AopProxy$CglibMethodInvocation.invokeJoinpoint(Cglib2AopProxy.java:689)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
        at org.springframework.flex.core.MessageInterceptionAdvice.invoke(MessageInterceptionAdvice.java:66)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
        at org.springframework.aop.framework.adapter.ThrowsAdviceInterceptor.invoke(ThrowsAdviceInterceptor.java:124)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
        at org.springframework.aop.framework.Cglib2AopProxy$FixedChainStaticTargetInterceptor.intercept(Cglib2AopProxy.java:573)
        at flex.messaging.endpoints.AMFEndpoint$$EnhancerByCGLIB$$12bf7fa6.serviceMessage(<generated>)
        at flex.messaging.endpoints.amf.MessageBrokerFilter.invoke(MessageBrokerFilter.java:103)
        at flex.messaging.endpoints.amf.LegacyFilter.invoke(LegacyFilter.java:158)
        at flex.messaging.endpoints.amf.SessionFilter.invoke(SessionFilter.java:44)
        at flex.messaging.endpoints.amf.BatchProcessFilter.invoke(BatchProcessFilter.java:67)
        at flex.messaging.endpoints.amf.SerializationFilter.invoke(SerializationFilter.java:166)
        at flex.messaging.endpoints.BaseHTTPEndpoint.service(BaseHTTPEndpoint.java:291)
        at flex.messaging.endpoints.AMFEndpoint$$EnhancerByCGLIB$$12bf7fa6.service(<generated>)
        at org.springframework.flex.servlet.MessageBrokerHandlerAdapter.handle(MessageBrokerHandlerAdapter.java:109)
        at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:923)
        at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:852)
        at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:882)
        at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:789)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:755)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:848)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
        at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:220)
        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:122)
        at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:501)
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:170)
        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:98)
        at org.eclipse.virgo.web.tomcat.support.ApplicationNameTrackingValve.invoke(ApplicationNameTrackingValve.java:33)
        at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:950)
        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:408)
        at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1040)
        at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:607)
        at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:315)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
        at java.lang.Thread.run(Unknown Source)
Caused by: java.lang.NoClassDefFoundError: Could not initialize class com.sun.jersey.core.header.MediaTypes
        at com.sun.jersey.core.spi.factory.MessageBodyFactory.initReaders(MessageBodyFactory.java:182)
        at com.sun.jersey.core.spi.factory.MessageBodyFactory.initReaders(MessageBodyFactory.java:176)
        at com.sun.jersey.core.spi.factory.MessageBodyFactory.init(MessageBodyFactory.java:162)
        at com.sun.jersey.api.client.Client.init(Client.java:343)
        at com.sun.jersey.api.client.Client.access$000(Client.java:119)
        at com.sun.jersey.api.client.Client$1.f(Client.java:192)
        at com.sun.jersey.api.client.Client$1.f(Client.java:188)
        at com.sun.jersey.spi.inject.Errors.processWithErrors(Errors.java:193)
        at com.sun.jersey.api.client.Client.<init>(Client.java:188)
        at com.sun.jersey.api.client.Client.<init>(Client.java:171)
        at com.sun.jersey.api.client.Client.create(Client.java:683)
        at com.emc.vdp2.api.impl.BaseApi.buildWebResource(BaseApi.java:97)
        at com.emc.vdp2.api.impl.ActionApi.connectVDR(ActionApi.java:34)
        at sun.reflect.GeneratedMethodAccessor2408.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
        at java.lang.reflect.Method.invoke(Unknown Source)
        at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:319)
        at org.eclipse.gemini.blueprint.service.importer.support.internal.aop.ServiceInvoker.doInvoke(ServiceInvoker.java:56)
        at org.eclipse.gemini.blueprint.service.importer.support.internal.aop.ServiceInvoker.invoke(ServiceInvoker.java:60)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
        at org.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:131)
        at org.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:119)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
        at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202)
        at com.sun.proxy.$Proxy512.connectVDR(Unknown Source)
        at sun.reflect.GeneratedMethodAccessor2407.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
        at java.lang.reflect.Method.invoke(Unknown Source)
        at flex.messaging.services.remoting.adapters.JavaAdapter.invoke(JavaAdapter.java:418)
        ... 48 common frames omitted 
 

[UDPsystems]

Hi Dasutin.

Did you find a resolution? We are having the same, or at least similar, issue.

Hosts: ESXi 6.0 U1a

vCenter Appliance: 6.0.0 Build 3018523

vSphere Web Client: 6.0.0 Build 2997665

Client Integration Plug-In: 6.0.0 Build 2994041

VDP Plug-In 6.1.0.19

6 VDP appliances: Versions 6.0 and 6.1

All was working just fine. Came in yesterday and tried to log into one of the VDR appliances from the Web Client to add a new workload to a back up job, and got the error.

Unable to connect to the requested VDP Appliance. 

Would you like to be directed to the VDP Configuration utility to troubleshoot the issue?

Connecting to the VDR configuration utility works. Green board across all services and proxies. Further investigation reveals that all VDP appliances are impacted, but all backup jobs and integrity checks are running successfully. We know the backup jobs are running from the morning emails we have configured from each VDP appliance, and the restore points available from the Emergency Restore tab in the VDP Configuration Utility. We have checked all blogs and articles that google finds as hits for this error. This has led us to check time configuration, reboot the VDP appliances, disable and enabled the VDP plug-in. The issue occurs across all installed browsers. (Chrome, FF, IE) We have not found any errors in the logs on the VDP appliances... so far. However, we did note lots of errors in the vcenter virgo log which look very similar to Dasutin's. In particular I recognize this.

[2016-04-13T19:34:34.385-04:00] [INFO ] http-bio-9443-exec-11         org.springframework.flex.servlet.MessageBrokerHandlerAdapter      Channel endpoint vdp-amf received request.

[2016-04-13T19:34:34.453-04:00] [ERROR] http-bio-9443-exec-11         com.emc.vdp2.api.impl.ActionApi                                   Trying to connect to: 192.168.***.***

[2016-04-13T19:34:34.527-04:00] [WARN ] http-bio-9443-exec-11         org.springframework.flex.core.DefaultExceptionLogger              The following exception occurred during request processing by the BlazeDS MessageBroker and will be serialized back to the client:

I don't have access to the log at the moment but that looks very familiar from yesterday's troubleshooting.

Any help, insights or suggestions appreciated.

Respectfully,

J. Murdoch

[Dasutin]

I haven't got this resolved yet.  I have a support ticket open with VMware since last Friday.  I'm waiting for them to call me back.  So far they just took the logs from VDP but everything in VDP is fine according to it's logs.  Just like you I only see this in the virgo log in vCenter.  It seems that vCenter is the issue and not VDP.  

This is very frustrating and I'm without backups because I did a reinstall of VDP and it stopped the backups from occurring.

I'm going to keep pressing support to get this resolved.  Once I do I'll come back with an update on what they did to fix it.

[UDPsystems]

Dasutin,

Appreciate the response! Sorry to hear you are still fighting the issue.

I agree that the issue is with vCenter. I found no notable log entries in the VDPs, and I doubt all 6 had an issue at the same time. Especially as we are running two different VDP versions. I am waiting on storage changes before migrating the rest to 6.1 as I will be increasing the size. Right now we are still getting backups, so my pain is a little lower than yours. Other fires are taking priority at the moment. That said, the new workloads I need to add are Dev, so I really want those in nightly backups before they put to much work into them. For now I am taking snapshots. If I get free from other non-VMware issues I may manage to get a ticket open myself.

Please update if you find out any additional info. Thank you very much.

Respectfully,

J. Murdoch

[SindhuHanji]

Hi Daustin,

There is a bug with VDP, where it stops working if there are other plugins installed with jersey jar of different version than it uses. To make sure its the same reason causing this issue, can you please point me to vmware support ticket.

[UDPsystems]

Good morning SindhuHanji,

If this is the case, what would be the solution?

We recently installed the plug-ins for vRealize Orchestrator, vRealize Operations Manager. I disabled all of them during troubleshooting, but it sounds like you are suggesting that a common file was overwritten during the install, so just disabling would not work. Do we manually add the other version back, and if so how?

Respectfully,

J. Murdoch

[Dasutin]

My support number is 16961925904.

[Dasutin]

I got it working! I went through and looked at all the third-party plugins I had installed.  I uninstalled my TinTri Storage plugin and I was able to connect to the VDP plugin! I had a 1.0 version installed and after I uninstalled it, I installed the newest 2.0 version from TinTri (which was released late March).  I got the same problem with the VDP plugin crashing.  So again I uninstalled the 2.0 version and after a restart of the vSphere Web Client, the VDP plugin worked again. 

Thank you for sharing this information! I never would have realized another plugin would cause it to not load.

Now is this a bug that will be fixed with an update to VDP?  Or should I contact TinTri?

 UDPsystems - I too have Orchestrator and vRealize Operations Manager installed in my environment.  If you have any other plugins installed I would try and removing them and seeing if the VDP plugin loads. 

[SindhuHanji]

Yes, Tintri plugin uses jersey jar. Fix should be given for the VDP plugin, not the Tintri plugin. I hope the fix will be available for VDP in the next release.

[UDPsystems]

Dasutin,

We found that it was our Nimble plugin. Nimble stated that it was a known issue as well. They said that there was a fix for vCenter. We had a little trouble getting a case open with VMware, as once again our vendor did not update VMware with our last support renewal, thank you HP.

After some initial confusion with the wrong PR number, we did get a working fix though. Nimble and VDP plugins are playing happily together now. In case you or anyone else has not been directed to the fix, here is the download folder you want VMware to provide access to, "Tintri_Plugin_PR1513956"

The only files in the download folder are a Readme.txt with the instructions, and a file to use as a replacement on your vcenter appliance/server. "vdr-ui-war-6.1.2.war"

One more note, the Readme file path "/var/lib/vmware/vsphere-client/vc-packages/vsphere-client-serenity/com.vmware.vdp2-6.1.*/plugins" was wrong for our VCA version.

Our version has the path "etc/vmware/vsphere-client/vc-packages/vsphere-client-serenity/com.vmware.vdp2-6.1.*/plugins"

Respectfully,

J. Murdoch