VMware Cloud Community
MikeNox
Enthusiast
Enthusiast
Jump to solution

vRO 7.4 is not logging workflow runs and complaining of Lucene60

Just updated my home lab vra/vro from 7.3.1 to 7.4.0 .  Installation seemed to go flawlessly, except vro.

When I try to run a workflow, the workflow completes, but nothing is in the log.  When I look at the system log, I see:

2018-04-12 20:28:29.332-0500 [http-nio-127.0.0.1-8280-exec-6] WARN  {} [StubbornRetrier] Retry 3/6 failed because of: Could not load codec "Lucene60".  Did you forget to add lucene-backward-codecs.jar?

 

2018-04-12 20:28:29.336-0500 [http-nio-127.0.0.1-8280-exec-9] WARN  {} [FileLoggingServiceImpl] Unable to retrieve previous logs:

java.lang.RuntimeException: Cannot succeed after 6 retries.

at ch.dunes.util.StubbornRetrier.retryAtAllCosts(StubbornRetrier.java:49)

at com.vmware.o11n.service.logging.FileLoggingServiceImpl.getLogs(FileLoggingServiceImpl.java:122)

at com.vmware.o11n.service.logging.FileLoggingServiceImpl.getLogsBefore(FileLoggingServiceImpl.java:106)

at com.vmware.o11n.service.factory.VcoFactoryFacade.getLogsBefore(VcoFactoryFacade.java:3138)

at sun.reflect.GeneratedMethodAccessor246.invoke(Unknown Source)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:498)

at com.vmware.o11n.service.security.AccessRightsInterceptor.invoke(AccessRightsInterceptor.java:98)

at com.vmware.o11n.service.security.AccessRightsInterceptor.invoke(AccessRightsInterceptor.java:89)

at com.vmware.o11n.service.webremoting.VcoDelegatingWebFacade.invokeOperation(VcoDelegatingWebFacade.java:105)

at com.vmware.o11n.integration.initialization.VcoFactoryServiceFacadeProxy.invokeOperation(VcoFactoryServiceFacadeProxy.java:86)

at sun.reflect.GeneratedMethodAccessor241.invoke(Unknown Source)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:498)

at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:333)

at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:207)

at com.sun.proxy.$Proxy24.invokeOperation(Unknown Source)

at sun.reflect.GeneratedMethodAccessor240.invoke(Unknown Source)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:498)

at org.springframework.remoting.support.RemoteInvocation.invoke(RemoteInvocation.java:212)

at org.springframework.remoting.support.DefaultRemoteInvocationExecutor.invoke(DefaultRemoteInvocationExecutor.java:39)

at org.springframework.remoting.support.RemoteInvocationBasedExporter.invoke(RemoteInvocationBasedExporter.java:78)

at org.springframework.remoting.support.RemoteInvocationBasedExporter.invokeAndCreateResult(RemoteInvocationBasedExporter.java:114)

at org.springframework.remoting.httpinvoker.HttpInvokerServiceExporter.handleRequest(HttpInvokerServiceExporter.java:80)

at org.springframework.web.context.support.HttpRequestHandlerServlet.service(HttpRequestHandlerServlet.java:67)

at javax.servlet.http.HttpServlet.service(HttpServlet.java:742)

at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231)

at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)

at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)

at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)

at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)

at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:317)

at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:127)

at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:91)

at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)

at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:114)

at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)

at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:111)

at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)

at org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:170)

at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)

at org.springframework.security.web.authentication.www.BasicAuthenticationFilter.doFilterInternal(BasicAuthenticationFilter.java:215)

at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)

at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)

at com.vmware.o11n.web.auth.http.TokenAuthenticationFilter.doFilter(TokenAuthenticationFilter.java:67)

at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)

at org.springframework.security.web.header.HeaderWriterFilter.doFilterInternal(HeaderWriterFilter.java:64)

at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)

at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)

at org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter.doFilterInternal(WebAsyncManagerIntegrationFilter.java:56)

at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)

at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)

at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:105)

at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)

at org.springframework.security.web.access.channel.ChannelProcessingFilter.doFilter(ChannelProcessingFilter.java:157)

at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)

at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:214)

at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:177)

at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:347)

at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:263)

at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)

at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)

at com.vmware.o11n.service.spring.bootstrap.SecureSerializationFilter.doFilter(SecureSerializationFilter.java:30)

at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)

at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)

at net.sf.ehcache.constructs.web.filter.GzipFilter.doFilter(GzipFilter.java:95)

at net.sf.ehcache.constructs.web.filter.Filter.doFilter(Filter.java:86)

at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)

at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)

at com.vmware.o11n.service.spring.bootstrap.WebRemotingActiveNodeFilter.doFilter(WebRemotingActiveNodeFilter.java:38)

at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)

at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)

at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:198)

at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96)

at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:613)

at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:140)

at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:81)

at org.apache.catalina.valves.RemoteIpValve.invoke(RemoteIpValve.java:677)

at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:650)

at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:87)

at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:342)

at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:803)

at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66)

at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:790)

at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1459)

at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)

at java.lang.Thread.run(Thread.java:748)

Caused by: java.lang.IllegalArgumentException: Could not load codec "Lucene60".  Did you forget to add lucene-backward-codecs.jar?

at org.apache.lucene.index.SegmentInfos.readCodec(SegmentInfos.java:436)

at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:356)

at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:288)

at org.apache.lucene.index.StandardDirectoryReader$1.doBody(StandardDirectoryReader.java:57)

at org.apache.lucene.index.StandardDirectoryReader$1.doBody(StandardDirectoryReader.java:54)

at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:692)

at org.apache.lucene.index.StandardDirectoryReader.open(StandardDirectoryReader.java:77)

at org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:63)

at com.vmware.o11n.service.logging.FileLoggingServiceImpl.doSearchLogs(FileLoggingServiceImpl.java:312)

at com.vmware.o11n.service.logging.FileLoggingServiceImpl.findLogs(FileLoggingServiceImpl.java:195)

at com.vmware.o11n.service.logging.FileLoggingServiceImpl.access$000(FileLoggingServiceImpl.java:54)

at com.vmware.o11n.service.logging.FileLoggingServiceImpl$2.call(FileLoggingServiceImpl.java:127)

at com.vmware.o11n.service.logging.FileLoggingServiceImpl$2.call(FileLoggingServiceImpl.java:123)

at ch.dunes.util.StubbornRetrier.retryAtAllCosts(StubbornRetrier.java:27)

... 90 more

Caused by: java.lang.IllegalArgumentException: An SPI class of type org.apache.lucene.codecs.Codec with name "Lucene60" does not exist.  You need to add the corresponding JAR file supporting this SPI to your classpath.  The current classpath supports the following names: [Lucene62]

at org.apache.lucene.util.NamedSPILoader.lookup(NamedSPILoader.java:116)

at org.apache.lucene.codecs.Codec.forName(Codec.java:116)

at org.apache.lucene.index.SegmentInfos.readCodec(SegmentInfos.java:424)

... 103 more

2018-04-12 20:28:29.338-0500 [http-nio-127.0.0.1-8280-exec-8] WARN  {} [StubbornRetrier] Retry 5/6 failed because of: Could not load codec "Lucene60".  Did you forget to add lucene-backward-codecs.jar?

Any thoughts?

Tags (3)
24 Replies
XModem
Enthusiast
Enthusiast
Jump to solution

Wow, this truely saved the day for me. Can we make a KB article out of this already? Hot patch the installer to remove the lucene indexes?

Reply
0 Kudos
iiliev
VMware Employee
VMware Employee
Jump to solution

Reply
0 Kudos
concord0007
VMware Employee
VMware Employee
Jump to solution

I believe there are more into this issue . The workaround helped bringing back the logging however the Events Tab still throws Error "Error while loading : Error while fetching : null .

Server.log is flooded with the following : 

2018-07-11 19:56:43.907+0000 [RollingLogFile-Thread] WARN {} [StubbornRetrier] Retry 1/3 failed because of: Lock held by this virtual machine: /storage/log/vmware/vco/app-server/scripting.log_lucene/write.lock

Waiting update if rebooting the vRO appliance Fix the issue. 

Reply
0 Kudos
Chakresh_HCSC
Contributor
Contributor
Jump to solution

I had similar issue but error in logs is different that what is mentioned here. I have tried the same workaround which did not fix the issue.

Error in catalina.out :

log4j:WARN Unable to get write.lock:

org.apache.lucene.store.LockObtainFailedException: Lock held by another program: /storage/log/vmware/vco/app-server/scripting.log_lucene/write.lock

        at org.apache.lucene.store.NativeFSLockFactory.obtainFSLock(NativeFSLockFactory.java:118)

        at org.apache.lucene.store.FSLockFactory.obtainLock(FSLockFactory.java:41)

        at org.apache.lucene.store.BaseDirectory.obtainLock(BaseDirectory.java:45)

        at org.apache.lucene.index.IndexWriter.<init>(IndexWriter.java:776)

        at com.vmware.o11n.service.logging.RollingLogFileAppender.initTry(RollingLogFileAppender.java:246)

        at com.vmware.o11n.service.logging.RollingLogFileAppender.access$400(RollingLogFileAppender.java:42)

        at com.vmware.o11n.service.logging.RollingLogFileAppender$2.call(RollingLogFileAppender.java:230)

        at com.vmware.o11n.service.logging.RollingLogFileAppender$2.call(RollingLogFileAppender.java:227)

        at ch.dunes.util.StubbornRetrier.retryAtAllCosts(StubbornRetrier.java:27)

        at com.vmware.o11n.service.logging.RollingLogFileAppender.init(RollingLogFileAppender.java:227)

        at com.vmware.o11n.service.logging.RollingLogFileAppender.access$300(RollingLogFileAppender.java:42)

        at com.vmware.o11n.service.logging.RollingLogFileAppender$RollingLogRunnable.run(RollingLogFileAppender.java:162)

        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)

        at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)

I could able to resolve this issue by rebooting faulty VRA/VRO appliance node.

Reply
0 Kudos
r0j
Enthusiast
Enthusiast
Jump to solution

The above works for us as well... We have had this issue with every upgrade after 7.3, we are at 7.6.

Reply
0 Kudos