ISO upload timeout issue with pyvcloud on vcloudDirector.

[sampatalhat]

Hi All,

We have configured VCD 10.2 in our environment. ISO upload stuck at with the following traceback in vcloud director appliance,

2021-02-22 06:16:42,054 | WARN | pool-jetty-485598 | AuthorizationMethodInterceptor | A security check was performed, but no required operation found for entityRef: com.vmware.vcloud.entity.org:e801cea9-ee34-43b0-8f0b-c1953f2d4c70. | requestId=a36a9491-b85a-4628-9861-b7a52ba759fa,request=GET https://172.18.3.21/api/org/e801cea9-ee34-43b0-8f0b-c1953f2d4c70,requestTime=1613974602001,remoteAdd... 31.0
2021-02-22 06:16:42,147 | WARN | pool-jetty-485422 | AuthorizationMethodInterceptor | A security check was performed, but no required operation found for entityRef: com.vmware.vcloud.entity.org:e801cea9-ee34-43b0-8f0b-c1953f2d4c70. | requestId=59eb55e9-8d22-4c14-b033-712a988c2628,request=GET https://172.18.3.21/api/org/e801cea9-ee34-43b0-8f0b-c1953f2d4c70,requestTime=1613974602095,remoteAdd... 31.0
2021-02-22 06:17:03,710 | WARN | pool-jetty-485634 | ImageTransferServer | Transfer session: 1d0e45c7-4a5f-4067-9922-6b7e8133856e. Resource file: file(baa82e8d-ae2a-47da-8e8a-a07c2084030a). Partial failure during upload from client to server file: VfsFile{fileObject=file:///opt/vmware/vcloud-director/data/transfer/1d0e45c7-4a5f-4067-9922-6b7e8133856e/file}. | requestId=b47b7df8-6764-4e8b-997e-36142509f87b,request=PUT https://172.18.3.21/transfer/1d0e45c7-4a5f-4067-9922-6b7e8133856e/file,requestTime=1613974593508,rem...
java.io.IOException: java.util.concurrent.TimeoutException: Idle timeout expired: 30001/30000 ms
at org.eclipse.jetty.server.HttpInput$ErrorState.noContent(HttpInput.java:857)
at org.eclipse.jetty.server.HttpInput.read(HttpInput.java:191)
at org.apache.commons.io.IOUtils.copyLarge(IOUtils.java:2389)
at org.apache.commons.io.IOUtils.copyLarge(IOUtils.java:2347)
at com.vmware.vcloud.common.vfs.chunked.VfsChunkedContent.addChunk(VfsChunkedContent.java:78)
at com.vmware.vcloud.common.vfs.chunked.VfsChunkedContent.addChunk(VfsChunkedContent.java:67)
at com.vmware.vcloud.transfer.impl.SimpleFileUploadStrategy$SimpleFileUploadConsumer.call(SimpleFileUploadStrategy.java:203)
at com.vmware.vcloud.transfer.impl.SimpleFileUploadStrategy$SimpleFileUploadConsumer.call(SimpleFileUploadStrategy.java:155)
at com.vmware.vcloud.transfer.ImageTransferServer.handlePutInternal(ImageTransferServer.java:789)
at com.vmware.vcloud.common.http.RestfulController.handleRequestInternal(RestfulController.java:52)
at org.springframework.web.servlet.mvc.AbstractController.handleRequest(AbstractController.java:174)
at org.springframework.web.servlet.mvc.SimpleControllerHandlerAdapter.handle(SimpleControllerHandlerAdapter.java:50)
at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:967)
at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:901)
at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:970)
at org.springframework.web.servlet.FrameworkServlet.doPut(FrameworkServlet.java:883)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:710)
at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:846)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:848)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1780)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:317)
at com.vmware.vcloud.security.filters.ValidityExceptionFilter.doFilterHttp(ValidityExceptionFilter.java:47)
at com.vmware.vcloud.security.filters.HttpFilterBean.doFilter(HttpFilterBean.java:35)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
at com.vmware.vcloud.security.filters.ConversationFilter$1.run(ConversationFilter.java:39)
at com.vmware.vcloud.security.filters.ConversationFilter$1.run(ConversationFilter.java:37)
at com.vmware.vcloud.common.persist.ConversationContextExecutor.execute(ConversationContextExecutor.java:89)
at com.vmware.vcloud.security.filters.ConversationFilter.doFilter(ConversationFilter.java:45)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
at com.vmware.vcloud.common.web.CorsFilter.doFilterInternal(CorsFilter.java:81)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:214)
at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:177)
at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:347)
at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:263)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
at com.vmware.vcloud.common.web.deployment.impl.RequestDiagnosticFilter.doFilter(RequestDiagnosticFilter.java:144)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1767)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:583)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:513)
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213)
at org.eclipse.jetty.server.handler.HandlerList.handle(HandlerList.java:52)
at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119)
at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:493)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
at org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:335)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
at org.eclipse.jetty.server.Server.handle(Server.java:539)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:333)
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108)
at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:259)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108)
at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
at com.vmware.vcloud.common.threading.ExtendedJettyQueuedThreadPool.runJob(ExtendedJettyQueuedThreadPool.java:33)
at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.util.concurrent.TimeoutException: Idle timeout expired: 30001/30000 ms
at org.eclipse.jetty.io.IdleTimeout.checkIdleTimeout(IdleTimeout.java:166)
at org.eclipse.jetty.io.IdleTimeout$1.run(IdleTimeout.java:50)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
... 1 more
Suppressed: java.lang.Throwable: HttpInput failure
at org.eclipse.jetty.server.HttpInput.failed(HttpInput.java:653)
at org.eclipse.jetty.server.HttpConnection$BlockingReadCallback.failed(HttpConnection.java:606)
at org.eclipse.jetty.io.FillInterest.onFail(FillInterest.java:138)
at org.eclipse.jetty.io.ssl.SslConnection.onFillInterestedFailed(SslConnection.java:286)
at org.eclipse.jetty.io.ssl.SslConnection$3.failed(SslConnection.java:123)
at org.eclipse.jetty.io.FillInterest.onFail(FillInterest.java:138)
at org.eclipse.jetty.io.AbstractEndPoint.onIdleExpired(AbstractEndPoint.java:173)
... 9 more
2021-02-22 06:17:09,342 | INFO | Backend-activity-pool-86812 | SpbmCollectorServiceImpl | Starting collection of Spbm info on VC 17b3e283-1aba-491a-8a8f-8aa5e2b50cd1 |

We have seen this issue many times in past, we tried with disabling proxy buffering in ./nginx/sites-available/vcd_ova_ui.conf    and nginx.conf file too, seen little bit improvement but still issue is seeing intermittently .

        proxy_buffering off;
        proxy_request_buffering off;