0 Replies Latest reply on Jun 7, 2020 12:01 PM by iNik26

    VCD 10.1.1 cannot create PVDC

    iNik26 Novice

      Hello,

       

      i've installed vcd 10.1.1 in lab to test integration with vSphere 7 and NSX-T 3.0 but i've some problem creating PVDC.

      I've configured NSX-T connection, network pool, imported t0 router and created external network. When i try to create PVCD the task fails with this error:

       

      com.vmware.ssdc.library.exceptions.InsufficientResourceException: The operation could not be performed, because there are no more host ID resources

       

      In vcloud-container-debug.log i've:

       

      =8b5b1e27-b181-4286-9cab-d876eefb491b,request=POST https://cloud.sddc.lab/api/admin/extension/providervdcsparams,requestTime=1591553695292,remoteAddress=10.26.12.1:51176,userAgent=Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:77.0) Gecko/...,accept=application/*+json;version 34.0

      2020-06-07 18:14:55,497 | WARN     | pool-jetty-75             | Conversation                   | Closing Hibernate session although it has not been flushed, allocated at=java.lang.Exception

        at com.vmware.vcloud.common.persist.Conversation.<init>(Conversation.java:142)

        at com.vmware.vcloud.common.persist.ConversationFactoryImpl.newConversation(ConversationFactoryImpl.java:256)

        at com.vmware.vcloud.common.persist.ConversationFactoryImpl.getConversation(ConversationFactoryImpl.java:193)

        at com.vmware.vcloud.common.persist.ConversationFactoryImpl.getConversation(ConversationFactoryImpl.java:165)

        at com.vmware.vcloud.common.persist.dao.AbstractHibernateDao.getConversation(AbstractHibernateDao.java:25)

        at com.vmware.vcloud.common.persist.dao.AbstractHibernateDaoBase.createCriteria(AbstractHibernateDaoBase.java:373)

        at com.vmware.vcloud.common.persist.dao.AbstractHibernateDaoBase.createCriteria(AbstractHibernateDaoBase.java:381)

        at com.vmware.vcloud.common.persist.dao.AbstractHibernateDaoBase.get(AbstractHibernateDaoBase.java:91)

        at com.vmware.ssdc.backendbase.security.SessionManagerImpl.getSession(SessionManagerImpl.java:146)

        at com.vmware.ssdc.backendbase.security.SessionManagerImpl.getSessionAndUpdateAccess(SessionManagerImpl.java:257)

        at com.vmware.vcloud.backendbase.management.system.SecurityServiceImpl.validateAuthenticationToken(SecurityServiceImpl.java:198)

        at com.vmware.vcloud.backendbase.management.system.SecurityServiceImpl.logInUser(SecurityServiceImpl.java:458)

        at com.vmware.vcloud.backendbase.management.system.SecurityServiceImpl.validateVcdJwt(SecurityServiceImpl.java:382)

        at sun.reflect.GeneratedMethodAccessor1020.invoke(Unknown Source)

        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

        at java.lang.reflect.Method.invoke(Method.java:498)

        at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:333)

        at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:190)

        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)

        at com.vmware.vcloud.common.validation.AbstractMethodInterceptor.invoke(AbstractMethodInterceptor.java:42)

        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)

        at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:213)

        at com.sun.proxy.$Proxy438.validateVcdJwt(Unknown Source)

        at com.vmware.vcloud.security.web.RestSessionFilter.validateVcdJwt(RestSessionFilter.java:291)

        at com.vmware.vcloud.security.web.RestSessionFilter.doFilter(RestSessionFilter.java:169)

        at com.vmware.vcloud.api.rest.diagnostics.RestApiDiagnosticsInterceptor.doFilter(RestApiDiagnosticsInterceptor.java:129)

        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)

        at com.vmware.vcloud.security.web.CellLivenessCheckFilter.doFilter(CellLivenessCheckFilter.java:54)

        at com.vmware.vcloud.api.rest.diagnostics.RestApiDiagnosticsInterceptor.doFilter(RestApiDiagnosticsInterceptor.java:129)

        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)

        at com.vmware.vcloud.security.web.SecurityContextCleanerFilter.doFilter(SecurityContextCleanerFilter.java:32)

        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)

        at com.vmware.vcloud.api.rest.versioning.AcceptHeaderFilter.doFilter(AcceptHeaderFilter.java:111)

        at com.vmware.vcloud.api.rest.diagnostics.RestApiDiagnosticsInterceptor.doFilter(RestApiDiagnosticsInterceptor.java:129)

        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)

        at com.vmware.vcloud.security.filters.ConversationFilter$1.run(ConversationFilter.java:39)

        at com.vmware.vcloud.security.filters.ConversationFilter$1.run(ConversationFilter.java:37)

        at com.vmware.vcloud.common.persist.ConversationContextExecutor.execute(ConversationContextExecutor.java:89)

        at com.vmware.vcloud.security.filters.ConversationFilter.doFilter(ConversationFilter.java:45)

        at com.vmware.vcloud.api.rest.diagnostics.RestApiDiagnosticsInterceptor.doFilter(RestApiDiagnosticsInterceptor.java:129)

        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)

        at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:214)

        at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:177)

        at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:347)

        at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:263)

        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)

        at com.vmware.vcloud.common.web.deployment.impl.RequestDiagnosticFilter.doFilter(RequestDiagnosticFilter.java:144)

        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1767)

        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:583)

        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)

        at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)

        at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)

        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)

        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:513)

        at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)

        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)

        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)

        at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213)

        at org.eclipse.jetty.server.handler.HandlerList.handle(HandlerList.java:52)

        at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119)

        at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:493)

        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)

        at org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:335)

        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)

        at org.eclipse.jetty.server.Server.handle(Server.java:539)

        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:333)

        at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)

        at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283)

        at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108)

        at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:259)

        at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283)

        at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108)

        at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)

        at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)

        at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)

        at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)

        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)

        at com.vmware.vcloud.common.threading.ExtendedJettyQueuedThreadPool.runJob(ExtendedJettyQueuedThreadPool.java:33)

        at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)

        at java.lang.Thread.run(Thread.java:748)

      [Conversation: 59835c82-6b73-4c17-b535-b2b1802ecfa7, transaction e42c7096-1464-488f-a66d-8233e079ef97, transactionDepth: 2] | requestId=8b5b1e27-b181-4286-9cab-d876eefb491b,request=POST https://cloud.sddc.lab/api/admin/extension/providervdcsparams,requestTime=1591553695292,remoteAddress=10.26.12.1:51176,userAgent=Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:77.0) Gecko/...,accept=application/*+json;version 34.0

      2020-06-07 18:14:55,498 | INFO     | pool-jetty-75             | Conversation                   | Transaction rolled back; session has been closed [Conversation: 59835c82-6b73-4c17-b535-b2b1802ecfa7, transaction null, transactionDepth: 0] | requestId=8b5b1e27-b181-4286-9cab-d876eefb491b,request=POST https://cloud.sddc.lab/api/admin/extension/providervdcsparams,requestTime=1591553695292,remoteAddress=10.26.12.1:51176,userAgent=Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:77.0) Gecko/...,accept=application/*+json;version 34.0

      2020-06-07 18:14:55,499 | ERROR    | pool-jetty-75             | CJob                           | Job (id=6c8a6c92-adad-489a-a624-a7393548db2a) failed, starting to update job. | requestId=8b5b1e27-b181-4286-9cab-d876eefb491b,request=POST https://cloud.sddc.lab/api/admin/extension/providervdcsparams,requestTime=1591553695292,remoteAddress=10.26.12.1:51176,userAgent=Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:77.0) Gecko/...,accept=application/*+json;version 34.0

      com.vmware.ssdc.library.exceptions.InsufficientResourceException: The operation could not be performed, because there are no more host ID resources

        at com.vmware.vcloud.net.impl.DefaultResourceManager.allocateResource(DefaultResourceManager.java:128)

        at com.vmware.vcloud.net.impl.DefaultResourceManager.allocateCHFHostId(DefaultResourceManager.java:134)

        at com.vmware.ssdc.backend.CManagedServer.insert(CManagedServer.java:494)

        at com.vmware.ssdc.backend.services.HostService.insertHostModel(HostService.java:116)

        at com.vmware.ssdc.backend.services.HostService.lambda$insertResourcePoolHosts$0(HostService.java:189)

        at java.util.ArrayList.forEach(ArrayList.java:1257)

        at com.vmware.ssdc.backend.services.HostService.insertResourcePoolHosts(HostService.java:187)

        at com.vmware.ssdc.backend.services.impl.ProviderVdcManagerImpl.insertProviderVdcModel(ProviderVdcManagerImpl.java:686)

        at com.vmware.vcloud.rcl.impl.RclServiceImpl.createProviderVdc(RclServiceImpl.java:841)

        at sun.reflect.GeneratedMethodAccessor3357.invoke(Unknown Source)

        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

        at java.lang.reflect.Method.invoke(Method.java:498)

        at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:333)

        at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:190)

        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)

        at org.springframework.aop.framework.adapter.MethodBeforeAdviceInterceptor.invoke(MethodBeforeAdviceInterceptor.java:56)

        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)

        at org.springframework.aop.framework.adapter.MethodBeforeAdviceInterceptor.invoke(MethodBeforeAdviceInterceptor.java:56)

        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)

        at com.vmware.vcloud.common.validation.AbstractMethodInterceptor.invoke(AbstractMethodInterceptor.java:42)

        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)

        at com.vmware.ssdc.backendbase.annotation.PresentationApiDiagnosticsInterceptor.invoke(PresentationApiDiagnosticsInterceptor.java:52)

        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)

        at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:213)

        at com.sun.proxy.$Proxy381.createProviderVdc(Unknown Source)

        at sun.reflect.GeneratedMethodAccessor3357.invoke(Unknown Source)

        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

        at java.lang.reflect.Method.invoke(Method.java:498)

        at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:333)

        at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:190)

        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)

        at com.vmware.vcloud.common.validation.AbstractMethodInterceptor.invoke(AbstractMethodInterceptor.java:42)

        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)

        at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:213)

        at com.sun.proxy.$Proxy639.createProviderVdc(Unknown Source)

        at com.vmware.vcloud.api.presentation.service.impl.RclServiceAdapterImpl.createProviderVdc(RclServiceAdapterImpl.java:91)

        at sun.reflect.GeneratedMethodAccessor3357.invoke(Unknown Source)

        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

        at java.lang.reflect.Method.invoke(Method.java:498)

        at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:333)

        at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:190)

        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)

        at com.vmware.vcloud.common.validation.AbstractMethodInterceptor.invoke(AbstractMethodInterceptor.java:42)

        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)

        at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:213)

        at com.sun.proxy.$Proxy640.createProviderVdc(Unknown Source)

        at com.vmware.vcloud.api.rest.handlers.admin.extension.ExtensionHandler.createProviderVdcParams(ExtensionHandler.java:1165)

        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)

        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

        at java.lang.reflect.Method.invoke(Method.java:498)

        at org.apache.cxf.service.invoker.AbstractInvoker.performInvocation(AbstractInvoker.java:180)

        at org.apache.cxf.service.invoker.AbstractInvoker.invoke(AbstractInvoker.java:96)

        at org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:191)

        at com.vmware.vcloud.api.rest.jaxrs.ContextClassLoaderInvoker.invoke(ContextClassLoaderInvoker.java:27)

        at org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:101)

        at org.apache.cxf.interceptor.ServiceInvokerInterceptor$1.run(ServiceInvokerInterceptor.java:59)

        at org.apache.cxf.interceptor.ServiceInvokerInterceptor.handleMessage(ServiceInvokerInterceptor.java:96)

        at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:309)

        at org.apache.cxf.transport.ChainInitiationObserver.onMessage(ChainInitiationObserver.java:121)

        at org.apache.cxf.transport.http.AbstractHTTPDestination.invoke(AbstractHTTPDestination.java:267)

        at org.apache.cxf.transport.servlet.ServletController.invokeDestination(ServletController.java:234)

        at org.apache.cxf.transport.servlet.ServletController.invoke(ServletController.java:208)

        at org.apache.cxf.transport.servlet.ServletController.invoke(ServletController.java:160)

        at org.apache.cxf.transport.servlet.CXFNonSpringServlet.invoke(CXFNonSpringServlet.java:189)

        at org.apache.cxf.transport.servlet.AbstractHTTPServlet.handleRequest(AbstractHTTPServlet.java:303)

        at org.apache.cxf.transport.servlet.AbstractHTTPServlet.doPost(AbstractHTTPServlet.java:222)

        at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)

        at org.apache.cxf.transport.servlet.AbstractHTTPServlet.service(AbstractHTTPServlet.java:278)

        at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:848)

        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1780)

        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:317)

        at com.vmware.vcloud.api.rest.diagnostics.DiagnosticFilter.doFilter(DiagnosticFilter.java:33)

        at com.vmware.vcloud.api.rest.diagnostics.RestApiDiagnosticsInterceptor.doFilter(RestApiDiagnosticsInterceptor.java:129)

        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)

        at com.vmware.vcloud.api.framework.web.ExtensibilityFilter.doFilter(ExtensibilityFilter.java:132)

        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)

        at com.vmware.vcloud.security.filters.ValidityExceptionFilter.doFilterHttp(ValidityExceptionFilter.java:47)

        at com.vmware.vcloud.security.filters.HttpFilterBean.doFilter(HttpFilterBean.java:35)

        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)

        at com.vmware.vcloud.security.web.RestSessionFilter.doFilter(RestSessionFilter.java:238)

        at com.vmware.vcloud.api.rest.diagnostics.RestApiDiagnosticsInterceptor.doFilter(RestApiDiagnosticsInterceptor.java:129)

        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)

        at com.vmware.vcloud.security.web.CellLivenessCheckFilter.doFilter(CellLivenessCheckFilter.java:54)

        at com.vmware.vcloud.api.rest.diagnostics.RestApiDiagnosticsInterceptor.doFilter(RestApiDiagnosticsInterceptor.java:129)

        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)

        at com.vmware.vcloud.security.web.SecurityContextCleanerFilter.doFilter(SecurityContextCleanerFilter.java:32)

        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)

        at com.vmware.vcloud.api.rest.versioning.AcceptHeaderFilter.doFilter(AcceptHeaderFilter.java:111)

        at com.vmware.vcloud.api.rest.diagnostics.RestApiDiagnosticsInterceptor.doFilter(RestApiDiagnosticsInterceptor.java:129)

        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)

        at com.vmware.vcloud.security.filters.ConversationFilter$1.run(ConversationFilter.java:39)

        at com.vmware.vcloud.security.filters.ConversationFilter$1.run(ConversationFilter.java:37)

        at com.vmware.vcloud.common.persist.ConversationContextExecutor.execute(ConversationContextExecutor.java:89)

        at com.vmware.vcloud.security.filters.ConversationFilter.doFilter(ConversationFilter.java:45)

        at com.vmware.vcloud.api.rest.diagnostics.RestApiDiagnosticsInterceptor.doFilter(RestApiDiagnosticsInterceptor.java:129)

        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)

        at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:214)

        at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:177)

        at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:347)

        at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:263)

        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)

        at com.vmware.vcloud.common.web.deployment.impl.RequestDiagnosticFilter.doFilter(RequestDiagnosticFilter.java:144)

        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1767)

        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:583)

        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)

        at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)

        at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)

        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)

        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:513)

        at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)

        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)

        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)

        at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213)

        at org.eclipse.jetty.server.handler.HandlerList.handle(HandlerList.java:52)

        at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119)

        at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:493)

        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)

        at org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:335)

        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)

        at org.eclipse.jetty.server.Server.handle(Server.java:539)

        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:333)

        at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)

        at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283)

        at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108)

        at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:259)

        at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283)

        at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108)

        at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)

        at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)

        at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)

        at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)

        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)

        at com.vmware.vcloud.common.threading.ExtendedJettyQueuedThreadPool.runJob(ExtendedJettyQueuedThreadPool.java:33)

        at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)

        at java.lang.Thread.run(Thread.java:748)

      2020-06-07 18:14:55,513 | DEBUG    | pool-jetty-75             | CJob                           | updateFailedJob(com.vmware.ssdc.library.exceptions.InsufficientResourceException) with locale=en_US | requestId=8b5b1e27-b181-4286-9cab-d876eefb491b,request=POST https://cloud.sddc.lab/api/admin/extension/providervdcsparams,requestTime=1591553695292,remoteAddress=10.26.12.1:51176,userAgent=Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:77.0) Gecko/...,accept=application/*+json;version 34.0

      com.vmware.ssdc.library.exceptions.InsufficientResourceException: The operation could not be performed, because there are no more host ID resources

        at com.vmware.vcloud.net.impl.DefaultResourceManager.allocateResource(DefaultResourceManager.java:128)

        at com.vmware.vcloud.net.impl.DefaultResourceManager.allocateCHFHostId(DefaultResourceManager.java:134)

        at com.vmware.ssdc.backend.CManagedServer.insert(CManagedServer.java:494)

        at com.vmware.ssdc.backend.services.HostService.insertHostModel(HostService.java:116)

        at com.vmware.ssdc.backend.services.HostService.lambda$insertResourcePoolHosts$0(HostService.java:189)

        at java.util.ArrayList.forEach(ArrayList.java:1257)

        at com.vmware.ssdc.backend.services.HostService.insertResourcePoolHosts(HostService.java:187)

        at com.vmware.ssdc.backend.services.impl.ProviderVdcManagerImpl.insertProviderVdcModel(ProviderVdcManagerImpl.java:686)

        at com.vmware.vcloud.rcl.impl.RclServiceImpl.createProviderVdc(RclServiceImpl.java:841)

        at sun.reflect.GeneratedMethodAccessor3357.invoke(Unknown Source)

        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

        at java.lang.reflect.Method.invoke(Method.java:498)

        at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:333)

        at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:190)

        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)

        at org.springframework.aop.framework.adapter.MethodBeforeAdviceInterceptor.invoke(MethodBeforeAdviceInterceptor.java:56)

        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)

        at org.springframework.aop.framework.adapter.MethodBeforeAdviceInterceptor.invoke(MethodBeforeAdviceInterceptor.java:56)

        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)

        at com.vmware.vcloud.common.validation.AbstractMethodInterceptor.invoke(AbstractMethodInterceptor.java:42)

        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)

        at com.vmware.ssdc.backendbase.annotation.PresentationApiDiagnosticsInterceptor.invoke(PresentationApiDiagnosticsInterceptor.java:52)

        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)

        at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:213)

        at com.sun.proxy.$Proxy381.createProviderVdc(Unknown Source)

        at sun.reflect.GeneratedMethodAccessor3357.invoke(Unknown Source)

        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

        at java.lang.reflect.Method.invoke(Method.java:498)

        at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:333)

        at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:190)

        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)

        at com.vmware.vcloud.common.validation.AbstractMethodInterceptor.invoke(AbstractMethodInterceptor.java:42)

        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)

        at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:213)

        at com.sun.proxy.$Proxy639.createProviderVdc(Unknown Source)

        at com.vmware.vcloud.api.presentation.service.impl.RclServiceAdapterImpl.createProviderVdc(RclServiceAdapterImpl.java:91)

        at sun.reflect.GeneratedMethodAccessor3357.invoke(Unknown Source)

        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

        at java.lang.reflect.Method.invoke(Method.java:498)

        at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:333)

        at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:190)

        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)

        at com.vmware.vcloud.common.validation.AbstractMethodInterceptor.invoke(AbstractMethodInterceptor.java:42)

        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)

        at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:213)

        at com.sun.proxy.$Proxy640.createProviderVdc(Unknown Source)

        at com.vmware.vcloud.api.rest.handlers.admin.extension.ExtensionHandler.createProviderVdcParams(ExtensionHandler.java:1165)

        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)

        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

        at java.lang.reflect.Method.invoke(Method.java:498)

        at org.apache.cxf.service.invoker.AbstractInvoker.performInvocation(AbstractInvoker.java:180)

        at org.apache.cxf.service.invoker.AbstractInvoker.invoke(AbstractInvoker.java:96)

        at org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:191)

        at com.vmware.vcloud.api.rest.jaxrs.ContextClassLoaderInvoker.invoke(ContextClassLoaderInvoker.java:27)

        at org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:101)

        at org.apache.cxf.interceptor.ServiceInvokerInterceptor$1.run(ServiceInvokerInterceptor.java:59)

        at org.apache.cxf.interceptor.ServiceInvokerInterceptor.handleMessage(ServiceInvokerInterceptor.java:96)

        at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:309)

        at org.apache.cxf.transport.ChainInitiationObserver.onMessage(ChainInitiationObserver.java:121)

        at org.apache.cxf.transport.http.AbstractHTTPDestination.invoke(AbstractHTTPDestination.java:267)

        at org.apache.cxf.transport.servlet.ServletController.invokeDestination(ServletController.java:234)

        at org.apache.cxf.transport.servlet.ServletController.invoke(ServletController.java:208)

        at org.apache.cxf.transport.servlet.ServletController.invoke(ServletController.java:160)

        at org.apache.cxf.transport.servlet.CXFNonSpringServlet.invoke(CXFNonSpringServlet.java:189)

        at org.apache.cxf.transport.servlet.AbstractHTTPServlet.handleRequest(AbstractHTTPServlet.java:303)

        at org.apache.cxf.transport.servlet.AbstractHTTPServlet.doPost(AbstractHTTPServlet.java:222)

        at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)

        at org.apache.cxf.transport.servlet.AbstractHTTPServlet.service(AbstractHTTPServlet.java:278)

        at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:848)

        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1780)

        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:317)

        at com.vmware.vcloud.api.rest.diagnostics.DiagnosticFilter.doFilter(DiagnosticFilter.java:33)

        at com.vmware.vcloud.api.rest.diagnostics.RestApiDiagnosticsInterceptor.doFilter(RestApiDiagnosticsInterceptor.java:129)

        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)

        at com.vmware.vcloud.api.framework.web.ExtensibilityFilter.doFilter(ExtensibilityFilter.java:132)

        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)

        at com.vmware.vcloud.security.filters.ValidityExceptionFilter.doFilterHttp(ValidityExceptionFilter.java:47)

        at com.vmware.vcloud.security.filters.HttpFilterBean.doFilter(HttpFilterBean.java:35)

        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)

        at com.vmware.vcloud.security.web.RestSessionFilter.doFilter(RestSessionFilter.java:238)

        at com.vmware.vcloud.api.rest.diagnostics.RestApiDiagnosticsInterceptor.doFilter(RestApiDiagnosticsInterceptor.java:129)

        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)

        at com.vmware.vcloud.security.web.CellLivenessCheckFilter.doFilter(CellLivenessCheckFilter.java:54)

        at com.vmware.vcloud.api.rest.diagnostics.RestApiDiagnosticsInterceptor.doFilter(RestApiDiagnosticsInterceptor.java:129)

        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)

        at com.vmware.vcloud.security.web.SecurityContextCleanerFilter.doFilter(SecurityContextCleanerFilter.java:32)

        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)

        at com.vmware.vcloud.api.rest.versioning.AcceptHeaderFilter.doFilter(AcceptHeaderFilter.java:111)

        at com.vmware.vcloud.api.rest.diagnostics.RestApiDiagnosticsInterceptor.doFilter(RestApiDiagnosticsInterceptor.java:129)

        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)

        at com.vmware.vcloud.security.filters.ConversationFilter$1.run(ConversationFilter.java:39)

        at com.vmware.vcloud.security.filters.ConversationFilter$1.run(ConversationFilter.java:37)

        at com.vmware.vcloud.common.persist.ConversationContextExecutor.execute(ConversationContextExecutor.java:89)

        at com.vmware.vcloud.security.filters.ConversationFilter.doFilter(ConversationFilter.java:45)

        at com.vmware.vcloud.api.rest.diagnostics.RestApiDiagnosticsInterceptor.doFilter(RestApiDiagnosticsInterceptor.java:129)

        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)

        at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:214)

        at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:177)

        at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:347)

        at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:263)

        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)

        at com.vmware.vcloud.common.web.deployment.impl.RequestDiagnosticFilter.doFilter(RequestDiagnosticFilter.java:144)

        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1767)

        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:583)

        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)

        at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)

        at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)

        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)

        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:513)

        at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)

        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)

        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)

        at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213)

        at org.eclipse.jetty.server.handler.HandlerList.handle(HandlerList.java:52)

        at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119)

        at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:493)

        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)

        at org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:335)

        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)

        at org.eclipse.jetty.server.Server.handle(Server.java:539)

        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:333)

        at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)

        at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283)

        at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108)

        at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:259)

        at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283)

        at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108)

        at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)

        at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)

        at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)

        at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)

        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)

        at com.vmware.vcloud.common.threading.ExtendedJettyQueuedThreadPool.runJob(ExtendedJettyQueuedThreadPool.java:33)

        at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)

        at java.lang.Thread.run(Thread.java:748)

       

       

      Any ideas about what's happening?

      Thanks for help!

       

      Kind regards,

      N.