Highlighted
Enthusiast
Enthusiast

vRO 7.4 is not logging workflow runs and complaining of Lucene60

Jump to solution

Just updated my home lab vra/vro from 7.3.1 to 7.4.0 .  Installation seemed to go flawlessly, except vro.

When I try to run a workflow, the workflow completes, but nothing is in the log.  When I look at the system log, I see:

2018-04-12 20:28:29.332-0500 [http-nio-127.0.0.1-8280-exec-6] WARN  {} [StubbornRetrier] Retry 3/6 failed because of: Could not load codec "Lucene60".  Did you forget to add lucene-backward-codecs.jar?

 

2018-04-12 20:28:29.336-0500 [http-nio-127.0.0.1-8280-exec-9] WARN  {} [FileLoggingServiceImpl] Unable to retrieve previous logs:

java.lang.RuntimeException: Cannot succeed after 6 retries.

at ch.dunes.util.StubbornRetrier.retryAtAllCosts(StubbornRetrier.java:49)

at com.vmware.o11n.service.logging.FileLoggingServiceImpl.getLogs(FileLoggingServiceImpl.java:122)

at com.vmware.o11n.service.logging.FileLoggingServiceImpl.getLogsBefore(FileLoggingServiceImpl.java:106)

at com.vmware.o11n.service.factory.VcoFactoryFacade.getLogsBefore(VcoFactoryFacade.java:3138)

at sun.reflect.GeneratedMethodAccessor246.invoke(Unknown Source)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:498)

at com.vmware.o11n.service.security.AccessRightsInterceptor.invoke(AccessRightsInterceptor.java:98)

at com.vmware.o11n.service.security.AccessRightsInterceptor.invoke(AccessRightsInterceptor.java:89)

at com.vmware.o11n.service.webremoting.VcoDelegatingWebFacade.invokeOperation(VcoDelegatingWebFacade.java:105)

at com.vmware.o11n.integration.initialization.VcoFactoryServiceFacadeProxy.invokeOperation(VcoFactoryServiceFacadeProxy.java:86)

at sun.reflect.GeneratedMethodAccessor241.invoke(Unknown Source)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:498)

at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:333)

at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:207)

at com.sun.proxy.$Proxy24.invokeOperation(Unknown Source)

at sun.reflect.GeneratedMethodAccessor240.invoke(Unknown Source)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:498)

at org.springframework.remoting.support.RemoteInvocation.invoke(RemoteInvocation.java:212)

at org.springframework.remoting.support.DefaultRemoteInvocationExecutor.invoke(DefaultRemoteInvocationExecutor.java:39)

at org.springframework.remoting.support.RemoteInvocationBasedExporter.invoke(RemoteInvocationBasedExporter.java:78)

at org.springframework.remoting.support.RemoteInvocationBasedExporter.invokeAndCreateResult(RemoteInvocationBasedExporter.java:114)

at org.springframework.remoting.httpinvoker.HttpInvokerServiceExporter.handleRequest(HttpInvokerServiceExporter.java:80)

at org.springframework.web.context.support.HttpRequestHandlerServlet.service(HttpRequestHandlerServlet.java:67)

at javax.servlet.http.HttpServlet.service(HttpServlet.java:742)

at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231)

at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)

at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)

at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)

at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)

at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:317)

at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:127)

at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:91)

at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)

at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:114)

at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)

at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:111)

at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)

at org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:170)

at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)

at org.springframework.security.web.authentication.www.BasicAuthenticationFilter.doFilterInternal(BasicAuthenticationFilter.java:215)

at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)

at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)

at com.vmware.o11n.web.auth.http.TokenAuthenticationFilter.doFilter(TokenAuthenticationFilter.java:67)

at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)

at org.springframework.security.web.header.HeaderWriterFilter.doFilterInternal(HeaderWriterFilter.java:64)

at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)

at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)

at org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter.doFilterInternal(WebAsyncManagerIntegrationFilter.java:56)

at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)

at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)

at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:105)

at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)

at org.springframework.security.web.access.channel.ChannelProcessingFilter.doFilter(ChannelProcessingFilter.java:157)

at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)

at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:214)

at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:177)

at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:347)

at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:263)

at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)

at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)

at com.vmware.o11n.service.spring.bootstrap.SecureSerializationFilter.doFilter(SecureSerializationFilter.java:30)

at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)

at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)

at net.sf.ehcache.constructs.web.filter.GzipFilter.doFilter(GzipFilter.java:95)

at net.sf.ehcache.constructs.web.filter.Filter.doFilter(Filter.java:86)

at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)

at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)

at com.vmware.o11n.service.spring.bootstrap.WebRemotingActiveNodeFilter.doFilter(WebRemotingActiveNodeFilter.java:38)

at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)

at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)

at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:198)

at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96)

at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:613)

at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:140)

at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:81)

at org.apache.catalina.valves.RemoteIpValve.invoke(RemoteIpValve.java:677)

at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:650)

at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:87)

at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:342)

at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:803)

at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66)

at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:790)

at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1459)

at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)

at java.lang.Thread.run(Thread.java:748)

Caused by: java.lang.IllegalArgumentException: Could not load codec "Lucene60".  Did you forget to add lucene-backward-codecs.jar?

at org.apache.lucene.index.SegmentInfos.readCodec(SegmentInfos.java:436)

at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:356)

at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:288)

at org.apache.lucene.index.StandardDirectoryReader$1.doBody(StandardDirectoryReader.java:57)

at org.apache.lucene.index.StandardDirectoryReader$1.doBody(StandardDirectoryReader.java:54)

at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:692)

at org.apache.lucene.index.StandardDirectoryReader.open(StandardDirectoryReader.java:77)

at org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:63)

at com.vmware.o11n.service.logging.FileLoggingServiceImpl.doSearchLogs(FileLoggingServiceImpl.java:312)

at com.vmware.o11n.service.logging.FileLoggingServiceImpl.findLogs(FileLoggingServiceImpl.java:195)

at com.vmware.o11n.service.logging.FileLoggingServiceImpl.access$000(FileLoggingServiceImpl.java:54)

at com.vmware.o11n.service.logging.FileLoggingServiceImpl$2.call(FileLoggingServiceImpl.java:127)

at com.vmware.o11n.service.logging.FileLoggingServiceImpl$2.call(FileLoggingServiceImpl.java:123)

at ch.dunes.util.StubbornRetrier.retryAtAllCosts(StubbornRetrier.java:27)

... 90 more

Caused by: java.lang.IllegalArgumentException: An SPI class of type org.apache.lucene.codecs.Codec with name "Lucene60" does not exist.  You need to add the corresponding JAR file supporting this SPI to your classpath.  The current classpath supports the following names: [Lucene62]

at org.apache.lucene.util.NamedSPILoader.lookup(NamedSPILoader.java:116)

at org.apache.lucene.codecs.Codec.forName(Codec.java:116)

at org.apache.lucene.index.SegmentInfos.readCodec(SegmentInfos.java:424)

... 103 more

2018-04-12 20:28:29.338-0500 [http-nio-127.0.0.1-8280-exec-8] WARN  {} [StubbornRetrier] Retry 5/6 failed because of: Could not load codec "Lucene60".  Did you forget to add lucene-backward-codecs.jar?

Any thoughts?

Tags (3)
24 Replies
Highlighted
Enthusiast
Enthusiast

Wow, this truely saved the day for me. Can we make a KB article out of this already? Hot patch the installer to remove the lucene indexes?

0 Kudos
Highlighted
VMware Employee
VMware Employee
0 Kudos
Highlighted
VMware Employee
VMware Employee

I believe there are more into this issue . The workaround helped bringing back the logging however the Events Tab still throws Error "Error while loading : Error while fetching : null .

Server.log is flooded with the following : 

2018-07-11 19:56:43.907+0000 [RollingLogFile-Thread] WARN {} [StubbornRetrier] Retry 1/3 failed because of: Lock held by this virtual machine: /storage/log/vmware/vco/app-server/scripting.log_lucene/write.lock

Waiting update if rebooting the vRO appliance Fix the issue. 

0 Kudos
Highlighted
Contributor
Contributor

I had similar issue but error in logs is different that what is mentioned here. I have tried the same workaround which did not fix the issue.

Error in catalina.out :

log4j:WARN Unable to get write.lock:

org.apache.lucene.store.LockObtainFailedException: Lock held by another program: /storage/log/vmware/vco/app-server/scripting.log_lucene/write.lock

        at org.apache.lucene.store.NativeFSLockFactory.obtainFSLock(NativeFSLockFactory.java:118)

        at org.apache.lucene.store.FSLockFactory.obtainLock(FSLockFactory.java:41)

        at org.apache.lucene.store.BaseDirectory.obtainLock(BaseDirectory.java:45)

        at org.apache.lucene.index.IndexWriter.<init>(IndexWriter.java:776)

        at com.vmware.o11n.service.logging.RollingLogFileAppender.initTry(RollingLogFileAppender.java:246)

        at com.vmware.o11n.service.logging.RollingLogFileAppender.access$400(RollingLogFileAppender.java:42)

        at com.vmware.o11n.service.logging.RollingLogFileAppender$2.call(RollingLogFileAppender.java:230)

        at com.vmware.o11n.service.logging.RollingLogFileAppender$2.call(RollingLogFileAppender.java:227)

        at ch.dunes.util.StubbornRetrier.retryAtAllCosts(StubbornRetrier.java:27)

        at com.vmware.o11n.service.logging.RollingLogFileAppender.init(RollingLogFileAppender.java:227)

        at com.vmware.o11n.service.logging.RollingLogFileAppender.access$300(RollingLogFileAppender.java:42)

        at com.vmware.o11n.service.logging.RollingLogFileAppender$RollingLogRunnable.run(RollingLogFileAppender.java:162)

        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)

        at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)

I could able to resolve this issue by rebooting faulty VRA/VRO appliance node.

0 Kudos
Highlighted
Enthusiast
Enthusiast

The above works for us as well... We have had this issue with every upgrade after 7.3, we are at 7.6.

0 Kudos