NSX-T manager upgrade stuck in "Pre-upgrade checks failed for MP: Failed to execute Check for ,,,"

[AKTM]

Hi Everyone:

Failed to upgrade NSX-T manager stuck on pre-check and got 3 critical error on:

• Pre-upgrade checks failed for MP: Failed to execute Check for sufficient free space on /tmp partition. null
• Pre-upgrade checks failed for MP: Failed to execute Check for sufficient free space on /config partition. null
• Pre-upgrade checks failed for MP: Failed to execute Check for sufficient free space on /image partition. null

Upgrade path from 3.2.0 to 3.2.1.0/3.2.1.1

Check the catalina.log got  the below exception:

---------
2022-08-25T15:03:38.327Z INFO org.apache.catalina.startup.HostConfig deployWAR Deployment of web application archive [/opt/vmware/upgrade-coordinator-tomcat/webapps/upgrade-coordinator.war] has finished in [120,972] ms
2022-08-25T15:03:38.337Z INFO org.apache.coyote.AbstractProtocol start Starting ProtocolHandler ["http-nio-127.0.0.1-7442"]
2022-09-28T03:40:37.225Z SEVERE com.googlecode.jsonrpc4j.JsonRpcServer handleObject Error in JSON-RPC Service
java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.googlecode.jsonrpc4j.JsonRpcServer.invoke(JsonRpcServer.java:503)
at com.vmware.nsx.management.common.client.rpc.JsonRpcServer.invoke(JsonRpcServer.java:34)
at com.googlecode.jsonrpc4j.JsonRpcServer.handleObject(JsonRpcServer.java:374)
at com.googlecode.jsonrpc4j.JsonRpcServer.handleNode(JsonRpcServer.java:283)
at com.googlecode.jsonrpc4j.JsonRpcServer.handle(JsonRpcServer.java:220)
at com.googlecode.jsonrpc4j.JsonRpcServer.handle(JsonRpcServer.java:205)
at com.vmware.nsx.management.common.client.rpc.BaseRpcController.execute(BaseRpcController.java:65)
at com.vmware.nsx.management.client.rpc.RpcController.facade(RpcController.java:46)
at sun.reflect.GeneratedMethodAccessor274.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205)
at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:150)
at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:117)
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:895)
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:808)
at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87)
at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1067)
at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:963)
at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1006)
at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:909)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:681)
at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:883)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:764)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:327)
at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:115)
at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:81)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:336)
at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:121)
at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:115)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:336)
at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:105)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:336)
at org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:149)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:336)
at org.springframework.security.web.authentication.preauth.AbstractPreAuthenticatedProcessingFilter.doFilter(AbstractPreAuthenticatedProcessingFilter.java:141)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:336)
at org.springframework.security.web.header.HeaderWriterFilter.doHeadersAfter(HeaderWriterFilter.java:90)
at org.springframework.security.web.header.HeaderWriterFilter.doFilterInternal(HeaderWriterFilter.java:75)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:336)
at org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter.doFilterInternal(WebAsyncManagerIntegrationFilter.java:55)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:336)
at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:110)
at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:80)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:336)
at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:211)
at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:183)
at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:354)
at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:267)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:196)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:97)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:542)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:135)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:81)
at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:698)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:78)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:364)
at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:624)
at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65)
at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:831)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1650)
at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
at org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1191)
at org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:659)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:748)
Caused by: com.vmware.nsx.management.common.exceptions.BaseException: Pre-upgrade checks failed for MP: Failed to execute Check for sufficient free space on /tmp partition. null
at com.vmware.nsx.management.upgrade.service.impl.UpgradeServiceImpl.handleMandatoryCheckResult(UpgradeServiceImpl.java:1124)
at com.vmware.nsx.management.upgrade.service.impl.UpgradeServiceImpl.lambda$executeMandatoryPreUpgradeChecks$12(UpgradeServiceImpl.java:1102)
at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183)
at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:175)
at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384)
at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482)
at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472)
at java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:150)
at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:173)
at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:485)
at com.vmware.nsx.management.upgrade.service.impl.UpgradeServiceImpl.executeMandatoryPreUpgradeChecks(UpgradeServiceImpl.java:1102)
at com.vmware.nsx.management.upgrade.service.impl.UpgradeServiceImpl.triggerUpgrade(UpgradeServiceImpl.java:982)
at com.vmware.nsx.management.upgrade.service.impl.UpgradeServiceImpl.upgradeComponent(UpgradeServiceImpl.java:1004)
at com.vmware.nsx.management.upgrade.service.impl.UpgradeServiceImpl.continueUpgrade(UpgradeServiceImpl.java:366)
at com.vmware.nsx.management.upgrade.service.impl.UpgradeServiceImpl.continueUpgrade(UpgradeServiceImpl.java:256)
at com.vmware.nsx.management.upgrade.facade.UpgradeCoordinatorFacadeImpl.continueUpgrade_aroundBody40(UpgradeCoordinatorFacadeImpl.java:660)
at com.vmware.nsx.management.upgrade.facade.UpgradeCoordinatorFacadeImpl$AjcClosure41.run(UpgradeCoordinatorFacadeImpl.java:1)
at org.aspectj.runtime.reflect.JoinPointImpl.proceed(JoinPointImpl.java:149)
at com.vmware.nsx.management.upgrade.facade.FacadeInterceptorHelperImpl.proceed(FacadeInterceptorHelperImpl.java:76)
at com.vmware.nsx.management.common.api.FacadeInterceptorAspect.annotatedMethod(FacadeInterceptorAspect.java:77)
at com.vmware.nsx.management.upgrade.facade.UpgradeCoordinatorFacadeImpl.continueUpgrade(UpgradeCoordinatorFacadeImpl.java:633)
... 79 more
-------------------

Anyone occurred the same issue or how to fixing. Thanks.

[ssanthoshkum]

Hi,

      Is this a standalone NSX-T Upgrade ? or SDDC NSX-T Upgrade ?.

If its standalone NSX-T Upgrade clear the temp using following command

Empty the file /tmp/bar with the command: echo  > /tmp/bar

https://kb.vmware.com/s/article/80640

If the upgrade is via SDDC manager , Upgrade manually NSX-T Manager first.

https://kb.vmware.com/s/article/81165

HTH

Sandy.

[AKTM]

HI Team:

The NSX-T manager is standalone. The said KB is not working actually there is no bar file over the size issue. And even follow the KB create 0 size bar file then perform the pre-check still failed

Please have RD or support team check the attached file for more detail the exception issue. Thanks

[AKTM]

Add more info

Found the NSX-T manager got "REPO_SYNC Failed"  will it impact the Manager pre-check failed?

[AKTM]

Hi Team:

That KB doesn't fix the upgrade case.

The upgrade issue be fixed after deploying one more NSX-T manager and manual join to existing cluster.

Alex