Solved: How to troubleshoot data collection failures of th...

darrenoid · ‎05-20-2016

Hello,

We are having issues with the data collection for the vRO endpoint in vra 7. Data collection fails. We suspect this is related to the DEM worker, but we are unsure how to troubleshoot this. So far the only error we have is when running the data collection on the vRO endpoint:

In vRA console > infrastructure > monitoring, logs:

Endpoint Data Collection failed for endpoint Internal vRO Instance [Workflow Instance Id=31009] vRealize Orchestrator returned an error: Internal Server Error.

In C:\Program Files (x86)\VMware\vCAC\Distributed Execution Manager\DEM-Core2\Logs\DEM-Core2_Errors

[UTC:2016-05-20 21:24:05 Local:2016-05-20 14:24:05] [Error]: Thread-Id="11" - context="" token="" <?xml version="1.0" encoding="utf-16"?>

<boolean>false</boolean>

Endpoint Data Collection failed for endpoint Internal vRO Instance [Workflow Instance Id=31009]

System.Net.WebException: vRealize Orchestrator returned an error: Internal Server Error.

at DynamicOps.VcoModel.Common.VcoClient.CheckResponseStatus(IRestResponse response)

at DynamicOps.VcoModel.Common.VcoClient.DecodeJsonResponse(IRestResponse response)

at DynamicOps.VcoModel.Common.VcoVersionUtils.GetPluginsInformation(VcoClient client)

at DynamicOps.VcoModel.Activities.StoreVcoPluginsInformation.Execute(CodeActivityContext context)

at System.Activities.CodeActivity.InternalExecute(ActivityInstance instance, ActivityExecutor executor, BookmarkManager bookmarkManager)

at System.Activities.Runtime.ActivityExecutor.ExecuteActivityWorkItem.ExecuteBody(ActivityExecutor executor, BookmarkManager bookmarkManager, Location resultLocation)

Also, when running data collection on a compute resources VCNS inventory, the following error occurs:

[UTC:2016-05-20 22:54:53 Local:2016-05-20 15:54:53] [Error]: Thread-Id="12" - context="" token="" <?xml version="1.0" encoding="utf-16"?>

<boolean>false</boolean>

Workflow 'vSphereVCNSInventory' failed with the following exception:

DynamicOps.VcoModel.Common.VcoException: Endpoint not found. There is no vRealize Orchestrator endpoint configured with property __VMware.VCenterOrchestrator.Plugin.NSX.Build.

at DynamicOps.VcoModel.Common.EndpointSelector..ctor(ManagementModelEntities context, String selectionFilter)

at DynamicOps.VCNSModel.Activities.VcoNsxHelper.VcoNsxConnectionInfo.GetVcoEndpoint(ManagementModelEntities managementDataContext, Guid virtualMachineId)

at DynamicOps.VCNSModel.Activities.VcoNsxHelper.VcoNsxConnectionInfo.GetConnection(ManagementModelEntities managementDataContext, VCNSEndpoint endpoint, Guid virtualMachineId)

at DynamicOps.VCNSModel.Workflows.vSphereVCNSInventory_CompiledExpressionRoot.vSphereVCNSInventory_CompiledExpressionRoot_TypedDataContext3_ForReadOnly.__Expr21Get()

at DynamicOps.VCNSModel.Workflows.vSphereVCNSInventory_CompiledExpressionRoot.InvokeExpression(Int32 expressionId, IList`1 locations, ActivityContext activityContext)

at Microsoft.CSharp.Activities.CSharpValue`1.Execute(CodeActivityContext context)

at System.Activities.Runtime.ActivityExecutor.ExecuteInResolutionContext[T](ActivityInstance parentInstance, Activity`1 expressionActivity)

at System.Activities.InArgument`1.TryPopulateValue(LocationEnvironment targetEnvironment, ActivityInstance activityInstance, ActivityExecutor executor)

at System.Activities.ActivityInstance.InternalTryPopulateArgumentValueOrScheduleExpression(RuntimeArgument argument, Int32 nextArgumentIndex, ActivityExecutor executor, IDictionary`2 argumentValueOverrides, Location resultLocation, Boolean isDynamicUpdate)

at System.Activities.ActivityInstance.ResolveArguments(ActivityExecutor executor, IDictionary`2 argumentValueOverrides, Location resultLocation, Int32 startIndex)

at System.Activities.Runtime.ActivityExecutor.ExecuteActivityWorkItem.ExecuteBody(ActivityExecutor executor, BookmarkManager bookmarkManager, Location resultLocation)

We are using a distributed deployment of vRA 7 and have 2 appliances with the embedded vro. Both appliances show the vco-server service as running, and I can connect to each vco instance using the orchestrator client. Also the XaaS component can load the vro workflows. I am able to browse the inventory.

Also, we noticed this problem after one of our iaas servers died on us. We had to rebuild one from scratch and shiv it in.It was the manager node, so it has the DEO, DEM and manager services running on it. Not sure if this is related, just that we noticed it after. It could have been happening earlier too...

Any suggestions on where to look to narrow this down would be greatly appreciated.

Thank you,

Darrenoid

darrenoid · ‎05-23-2016

Thank you for your response.

I finally figured this out this morning. I was looking in the /var/log/vco/app-server/server.log on the vra appliances and found errors like this:

2016-05-21 08:31:33.345-0700 [http-nio-127.0.0.1-8280-exec-10] ERROR {} [[restServlet]] Servlet.service() for servlet [restServlet] in context with path [/vco] threw exception [Request processing failed; nested exception is java.lang.NullPointerException] with root cause

java.lang.NullPointerException

at com.vmware.o11n.web.util.CommonUtil.getServerConfiguration(CommonUtil.java:44)

at com.vmware.o11n.web.plugins.PluginController.listAllPlugins(PluginController.java:116)

at sun.reflect.GeneratedMethodAccessor347.invoke(Unknown Source)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:497)

at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:221)

at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:137)

at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:110)

at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:806)

at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:729)

at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:85)

at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:959)

at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:893)

at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:970)

at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:861)

at javax.servlet.http.HttpServlet.service(HttpServlet.java:622)

at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:846)

at javax.servlet.http.HttpServlet.service(HttpServlet.java:729)

at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:291)

at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)

at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)

at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:239)

at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)

at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:316)

at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:126)

at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:90)

at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330)

at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:114)

at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330)

at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:111)

at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330)

at org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:169)

at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330)

at org.springframework.security.web.authentication.www.BasicAuthenticationFilter.doFilterInternal(BasicAuthenticationFilter.java:158)

at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)

at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330)

at com.vmware.o11n.web.auth.http.TokenAuthenticationFilter.doFilter(TokenAuthenticationFilter.java:97)

at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330)

at org.springframework.security.web.header.HeaderWriterFilter.doFilterInternal(HeaderWriterFilter.java:64)

at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)

at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330)

at org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter.doFilterInternal(WebAsyncManagerIntegrationFilter.java:53)

at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)

at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330)

at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:91)

at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330)

at org.springframework.security.web.access.channel.ChannelProcessingFilter.doFilter(ChannelProcessingFilter.java:152)

at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330)

at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:213)

at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:176)

at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:346)

at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:262)

at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:239)

at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)

at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:85)

at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)

at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:239)

at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)

at org.springframework.web.filter.HiddenHttpMethodFilter.doFilterInternal(HiddenHttpMethodFilter.java:77)

at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)

at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:239)

at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)

at com.vmware.o11n.json.DefaultJsonVersionHeaderFilter.doFilter(DefaultJsonVersionHeaderFilter.java:95)

at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:239)

at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)

at com.vmware.o11n.web.cluster.RestActiveNodeFilter.doFilter(RestActiveNodeFilter.java:50)

at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:346)

at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:262)

at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:239)

at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)

at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:219)

at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:106)

at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:614)

at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:142)

at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:79)

at org.apache.catalina.valves.RemoteIpValve.invoke(RemoteIpValve.java:676)

at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:616)

at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:88)

at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:518)

at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1091)

at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:673)

at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1526)

at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1482)

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)

at java.lang.Thread.run(Thread.java:745)

I searched google for the first line in the stack trace "com.vmware.o11n.web.util.CommonUtil.getServerConfiguration" and stumbled on this recent KB article from VMWare:

https://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=21450...

That fixed it for me, now data collection on the vro endpoint works.

Regards,

Darrenoid

View solution in original post

emerson132 · ‎05-23-2016

Probably not much help but I was getting "Internal Server Error" issues in 6.2 with vRA/vRO after the appliance was updated. I managed to narrow it down to both the time being out of sync on the IaaS servers and the appliances, and also the person who re-installed the IaaS components on the server used the existing install file (old version) rather than re-download the IaaS components from the appliance and perform the upgrade.

darrenoid · ‎05-23-2016