Upgrade vRealize Automation 7.3.0 to 7.4 observations

[r0j]

We did a dry run upgrade of our Dev minimal install environment from 7.3-7.4.

This DEV environment was a fresh 7.3 build 6 months ago.

Initial identified issue was vRO stopped logging on workflows, which was resolved by the following (thank you poster).

https://communities.vmware.com/message/2762334#2762334

The upgrade proceeded to run extremely slow for about a day, before grenading itself.

The catalina.out filled itself to 5+GB, before blowing out the /storage/log partition on the appliance.

Deleted the catalina.out and rebooted.

Everything got ugly after that...

rabbitmq wouldn't start

composition-service - com.vmware.csp.component.cafe.composition wouldn't start.

Environment was fubar'd, had to roll back iaas and vra and databases.

Anyone have similar experiences?

[daphnissov]

Haven't done it myself (but is planned). I'm in the same boat with a newly-deployed 7.3 instance. I've never had any success with vRA upgrades, even on a simple minimal install. Fresh install is what I try to do now.

[r0j]

Thanks for the feedback and input daphnissov..

I am getting the very same impression, and feel that future versions should just incorporate this process, where appliances are just throwaways.

We use codestream to move content between vRA environments, however there are some deficiencies when moving tenant info / configurations and such, where improvements to that tool would be helpful with this.

[gradinka]

you can always leverage migration, and not upgrade.

that's also built-in

--

but I wonder what happened with this upgrade running for so long (that is not uncommon, depending on the amount of items in the environment).

can we get a log bundle with more details

[r0j]

So..... I attempted to do another 7.3-7.4 upgrade on my DEV environment on Friday afternoon.

Everything went smooth, no errors in any of the upgrade logs, and the environment was running great.

come on on monday everything in DEV vRA is down, and again the catalina.out log is 5+GB and the log partition is completely full.

delete the log and reboot, rabbitmq is dead and unrecoverable.

Can anyone provide clues as to wth is happening with my particular upgrade?

I pulled down the log bundle and catalina.out, and will be opening a support ticket.

For now I have to rollback my DEV. Again.

[sbeaver]

I just spent the weekend upgrading the environment.  I knew I was going to have some issues letting the appliance upgrade process also update the IaaS components in my distributed environment and ended up separating things out so the appliance upgrade would just update the appliances and then did the IaaS independently.  I do have a recommendation from my experience...  If you have multiple appliances in your setup. consolidate everything to the ,master node so that the database and the sync connector are the same and then split up again after the upgrade and for what it is worth, clear out all the old log files before you take you snapshot

[gradinka]

some clues in the catalina.out?

I guess we'll be looking into the SR anyways.

[r0j]

The catalina.out is in the log bundle I uploaded in SR # 18781199104.

we havnt rolled back dev yet in case we get someone on the line quick to look at this.

root /dev/sda1 also was at 100%

[sbeaver]

Have you rebooted that node lately since it was full?  I have a node that was 80% full and I could not find where but a reboot and all of a sudden it was back to like 10%

[r0j]

yes, rebooted multiple times, the reboot resolved the log partition not seeing the 5GB of free space, after deleting the catalina.out.

we rolled back to 7.3, and will work with vmware support to get through this.

thanks for the responses..

[r0j]

Cracked open the 5GB catalina.out, here is a sample of the repeated errors filling the log about 18-24 hours after the upgrade.

I have performed the upgrade 2x with the same results.

Definition of insanity?

Thoughts?

[UTC:2018-04-21 22:10:06,677 Local:2018-04-21 22:10:06,677] vcac: [component="cafe:notification" priority="WARN" thread="notification-service-subscrAmqpExec-5" tenant="" context="" parent="" token=""] org.springframework.integration.channel.MessagePublishingErrorHandler.handleError:123 - Error message was not delivered.

org.springframework.messaging.MessageDeliveryException: failed to send Message to channel 'errorChannel'; nested exception is org.springframework.amqp.AmqpConnectException: java.net.ConnectException: Connection refused (Connection refused)

at org.springframework.integration.channel.AbstractMessageChannel.send(AbstractMessageChannel.java:451) ~[spring-integration-core-4.3.13.RELEASE.jar:4.3.13.RELEASE]

at org.springframework.integration.channel.MessagePublishingErrorHandler.handleError(MessagePublishingErrorHandler.java:114) [spring-integration-core-4.3.13.RELEASE.jar:4.3.13.RELEASE]

at org.springframework.integration.util.ErrorHandlingTaskExecutor$1.run(ErrorHandlingTaskExecutor.java:58) [spring-integration-core-4.3.13.RELEASE.jar:4.3.13.RELEASE]

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_161]

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_161]

at java.lang.Thread.run(Thread.java:748) [?:1.8.0_161]

Caused by: org.springframework.amqp.AmqpConnectException: java.net.ConnectException: Connection refused (Connection refused)

at org.springframework.amqp.rabbit.support.RabbitExceptionTranslator.convertRabbitAccessException(RabbitExceptionTranslator.java:62) ~[spring-rabbit-1.6.11.RELEASE.jar:?]

at org.springframework.amqp.rabbit.connection.AbstractConnectionFactory.createBareConnection(AbstractConnectionFactory.java:306) ~[spring-rabbit-1.6.11.RELEASE.jar:?]

at org.springframework.amqp.rabbit.connection.CachingConnectionFactory.createConnection(CachingConnectionFactory.java:561) ~[spring-rabbit-1.6.11.RELEASE.jar:?]

at org.springframework.amqp.rabbit.core.RabbitTemplate.doExecute(RabbitTemplate.java:1431) ~[spring-rabbit-1.6.11.RELEASE.jar:?]

at org.springframework.amqp.rabbit.core.RabbitTemplate.access$2400(RabbitTemplate.java:132) ~[spring-rabbit-1.6.11.RELEASE.jar:?]

at org.springframework.amqp.rabbit.core.RabbitTemplate$10.doWithRetry(RabbitTemplate.java:1399) ~[spring-rabbit-1.6.11.RELEASE.jar:?]

at org.springframework.retry.support.RetryTemplate.doExecute(RetryTemplate.java:276) ~[spring-retry-1.1.3.RELEASE.jar:?]

at org.springframework.retry.support.RetryTemplate.execute(RetryTemplate.java:172) ~[spring-retry-1.1.3.RELEASE.jar:?]

at org.springframework.amqp.rabbit.core.RabbitTemplate.execute(RabbitTemplate.java:1395) ~[spring-rabbit-1.6.11.RELEASE.jar:?]

at org.springframework.amqp.rabbit.core.RabbitTemplate.send(RabbitTemplate.java:714) ~[spring-rabbit-1.6.11.RELEASE.jar:?]

at org.springframework.amqp.rabbit.core.RabbitTemplate.convertAndSend(RabbitTemplate.java:815) ~[spring-rabbit-1.6.11.RELEASE.jar:?]

at org.springframework.amqp.rabbit.core.RabbitTemplate.convertAndSend(RabbitTemplate.java:805) ~[spring-rabbit-1.6.11.RELEASE.jar:?]

at com.vmware.vcac.platform.service.integration.channel.ExchangeBindablePollableAmqpChannelProxy.doSend(ExchangeBindablePollableAmqpChannelProxy.java:89) ~[platform-integration-7.4.0-SNAPSHOT.jar:?]

at org.springframework.integration.channel.AbstractMessageChannel.send(AbstractMessageChannel.java:425) ~[spring-integration-core-4.3.13.RELEASE.jar:4.3.13.RELEASE]

... 5 more

Caused by: java.net.ConnectException: Connection refused (Connection refused)

at java.net.PlainSocketImpl.socketConnect(Native Method) ~[?:1.8.0_161]

at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) ~[?:1.8.0_161]

at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) ~[?:1.8.0_161]

at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) ~[?:1.8.0_161]

at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[?:1.8.0_161]

at java.net.Socket.connect(Socket.java:589) ~[?:1.8.0_161]

at com.rabbitmq.client.impl.FrameHandlerFactory.create(FrameHandlerFactory.java:47) ~[amqp-client-3.6.5.jar:?]

at com.rabbitmq.client.ConnectionFactory.newConnection(ConnectionFactory.java:822) ~[amqp-client-3.6.5.jar:?]

at com.rabbitmq.client.ConnectionFactory.newConnection(ConnectionFactory.java:778) ~[amqp-client-3.6.5.jar:?]

at com.rabbitmq.client.ConnectionFactory.newConnection(ConnectionFactory.java:898) ~[amqp-client-3.6.5.jar:?]

at org.springframework.amqp.rabbit.connection.AbstractConnectionFactory.createBareConnection(AbstractConnectionFactory.java:297) ~[spring-rabbit-1.6.11.RELEASE.jar:?]

at org.springframework.amqp.rabbit.connection.CachingConnectionFactory.createConnection(CachingConnectionFactory.java:561) ~[spring-rabbit-1.6.11.RELEASE.jar:?]

at org.springframework.amqp.rabbit.core.RabbitTemplate.doExecute(RabbitTemplate.java:1431) ~[spring-rabbit-1.6.11.RELEASE.jar:?]

at org.springframework.amqp.rabbit.core.RabbitTemplate.access$2400(RabbitTemplate.java:132) ~[spring-rabbit-1.6.11.RELEASE.jar:?]

at org.springframework.amqp.rabbit.core.RabbitTemplate$10.doWithRetry(RabbitTemplate.java:1399) ~[spring-rabbit-1.6.11.RELEASE.jar:?]

at org.springframework.retry.support.RetryTemplate.doExecute(RetryTemplate.java:276) ~[spring-retry-1.1.3.RELEASE.jar:?]

at org.springframework.retry.support.RetryTemplate.execute(RetryTemplate.java:172) ~[spring-retry-1.1.3.RELEASE.jar:?]

at org.springframework.amqp.rabbit.core.RabbitTemplate.execute(RabbitTemplate.java:1395) ~[spring-rabbit-1.6.11.RELEASE.jar:?]

at org.springframework.amqp.rabbit.core.RabbitTemplate.send(RabbitTemplate.java:714) ~[spring-rabbit-1.6.11.RELEASE.jar:?]

at org.springframework.amqp.rabbit.core.RabbitTemplate.convertAndSend(RabbitTemplate.java:815) ~[spring-rabbit-1.6.11.RELEASE.jar:?]

at org.springframework.amqp.rabbit.core.RabbitTemplate.convertAndSend(RabbitTemplate.java:805) ~[spring-rabbit-1.6.11.RELEASE.jar:?]

at com.vmware.vcac.platform.service.integration.channel.ExchangeBindablePollableAmqpChannelProxy.doSend(ExchangeBindablePollableAmqpChannelProxy.java:89) ~[platform-integration-7.4.0-SNAPSHOT.jar:?]

at org.springframework.integration.channel.AbstractMessageChannel.send(AbstractMessageChannel.java:425) ~[spring-integration-core-4.3.13.RELEASE.jar:4.3.13.RELEASE]

... 5 more

[UTC:2018-04-22 19:20:02,723 Local:2018-04-22 19:20:02,723] vcac: [component="cafe:event-broker" priority="ERROR" thread="tomcat-http--33" tenant="vsphere.local" context="ZcAo7KiR" parent="" token="ZcAo7KiR"] com.vmware.vcac.core.event.broker.service.impl.SubscribeServiceImpl.pollEvents:154 - Error on receive and convert from queue name :ebs.com.vmware.csp.iaas.blueprint.service.machine.lifecycle.provision__b46f8623-3da9-3f8f-b954-fbadd6895827

org.springframework.amqp.AmqpConnectException: java.net.ConnectException: Connection refused (Connection refused)

at org.springframework.amqp.rabbit.support.RabbitExceptionTranslator.convertRabbitAccessException(RabbitExceptionTranslator.java:62) ~[spring-rabbit-1.6.11.RELEASE.jar:?]

at org.springframework.amqp.rabbit.connection.AbstractConnectionFactory.createBareConnection(AbstractConnectionFactory.java:306) ~[spring-rabbit-1.6.11.RELEASE.jar:?]

at org.springframework.amqp.rabbit.connection.CachingConnectionFactory.createConnection(CachingConnectionFactory.java:561) ~[spring-rabbit-1.6.11.RELEASE.jar:?]

at org.springframework.amqp.rabbit.core.RabbitTemplate.doExecute(RabbitTemplate.java:1431) ~[spring-rabbit-1.6.11.RELEASE.jar:?]

at org.springframework.amqp.rabbit.core.RabbitTemplate.execute(RabbitTemplate.java:1412) ~[spring-rabbit-1.6.11.RELEASE.jar:?]

at org.springframework.amqp.rabbit.core.RabbitTemplate.doReceiveNoWait(RabbitTemplate.java:841) ~[spring-rabbit-1.6.11.RELEASE.jar:?]

at org.springframework.amqp.rabbit.core.RabbitTemplate.receiveAndConvert(RabbitTemplate.java:931) ~[spring-rabbit-1.6.11.RELEASE.jar:?]

at org.springframework.amqp.rabbit.core.RabbitTemplate.receiveAndConvert(RabbitTemplate.java:921) ~[spring-rabbit-1.6.11.RELEASE.jar:?]

at org.springframework.integration.amqp.channel.PollableAmqpChannel.doReceive(PollableAmqpChannel.java:210) ~[spring-integration-amqp-4.3.13.RELEASE.jar:4.3.13.RELEASE]

at org.springframework.integration.amqp.channel.PollableAmqpChannel.receive(PollableAmqpChannel.java:169) ~[spring-integration-amqp-4.3.13.RELEASE.jar:4.3.13.RELEASE]

at com.vmware.vcac.core.event.broker.service.impl.SubscribeServiceImpl.pollEvents(SubscribeServiceImpl.java:139) [classes/:?]

at com.vmware.vcac.core.event.broker.controller.SubscriptionController.pollEvents(SubscriptionController.java:263) [classes/:?]

at com.vmware.vcac.core.event.broker.controller.SubscriptionController$$FastClassBySpringCGLIB$$c3c3727.invoke(<generated>) [classes/:?]

at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204) [spring-core-4.3.13.RELEASE.jar:4.3.13.RELEASE]

at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:738) [spring-aop-4.3.13.RELEASE.jar:4.3.13.RELEASE]

at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157) [spring-aop-4.3.13.RELEASE.jar:4.3.13.RELEASE]

at org.springframework.security.access.intercept.aopalliance.MethodSecurityInterceptor.invoke(MethodSecurityInterceptor.java:69) [spring-security-core-4.2.1.RELEASE.jar:4.2.1.RELEASE]

at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) [spring-aop-4.3.13.RELEASE.jar:4.3.13.RELEASE]

at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:673) [spring-aop-4.3.13.RELEASE.jar:4.3.13.RELEASE]

at com.vmware.vcac.core.event.broker.controller.SubscriptionController$$EnhancerBySpringCGLIB$$5f26e10.pollEvents(<generated>) [classes/:?]

at sun.reflect.GeneratedMethodAccessor2051.invoke(Unknown Source) ~[?:?]

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_161]

at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_161]

at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205) [spring-web-4.3.13.RELEASE.jar:4.3.13.RELEASE]

at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:133) [spring-web-4.3.13.RELEASE.jar:4.3.13.RELEASE]

at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:97) [spring-webmvc-4.3.13.RELEASE.jar:4.3.13.RELEASE]

at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:827) [spring-webmvc-4.3.13.RELEASE.jar:4.3.13.RELEASE]

at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:738) [spring-webmvc-4.3.13.RELEASE.jar:4.3.13.RELEASE]

at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:85) [spring-webmvc-4.3.13.RELEASE.jar:4.3.13.RELEASE]

at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:967) [spring-webmvc-4.3.13.RELEASE.jar:4.3.13.RELEASE]

at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:901) [spring-webmvc-4.3.13.RELEASE.jar:4.3.13.RELEASE]

at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:970) [spring-webmvc-4.3.13.RELEASE.jar:4.3.13.RELEASE]

at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:861) [spring-webmvc-4.3.13.RELEASE.jar:4.3.13.RELEASE]

at javax.servlet.http.HttpServlet.service(HttpServlet.java:635) [servlet-api.jar:?]

at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:846) [spring-webmvc-4.3.13.RELEASE.jar:4.3.13.RELEASE]

at javax.servlet.http.HttpServlet.service(HttpServlet.java:742) [servlet-api.jar:?]

at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231) [catalina.jar:8.5.29]

at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [catalina.jar:8.5.29]

at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52) [tomcat-websocket.jar:8.5.29]

at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [catalina.jar:8.5.29]

at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [catalina.jar:8.5.29]

at org.apache.logging.log4j.web.Log4jServletFilter.doFilter(Log4jServletFilter.java:71) [platform-log4j-web-custom-7.4.0-SNAPSHOT.jar:?]

at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [catalina.jar:8.5.29]

at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [catalina.jar:8.5.29]

at com.vmware.vcac.platform.metrics.servlet.MetricsAbstractInstrumentedFilter.doFilter(MetricsAbstractInstrumentedFilter.java:95) [platform-metrics-servlet-7.4.0-SNAPSHOT.jar:?]

at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [catalina.jar:8.5.29]

at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [catalina.jar:8.5.29]

at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:317) [spring-security-web-4.2.1.RELEASE.jar:4.2.1.RELEASE]

at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:127) [spring-security-web-4.2.1.RELEASE.jar:4.2.1.RELEASE]

at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:91) [spring-security-web-4.2.1.RELEASE.jar:4.2.1.RELEASE]

at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) [spring-security-web-4.2.1.RELEASE.jar:4.2.1.RELEASE]

at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:114) [spring-security-web-4.2.1.RELEASE.jar:4.2.1.RELEASE]

at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) [spring-security-web-4.2.1.RELEASE.jar:4.2.1.RELEASE]

at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:111) [spring-security-web-4.2.1.RELEASE.jar:4.2.1.RELEASE]

at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) [spring-security-web-4.2.1.RELEASE.jar:4.2.1.RELEASE]

at org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:170) [spring-security-web-4.2.1.RELEASE.jar:4.2.1.RELEASE]

at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) [spring-security-web-4.2.1.RELEASE.jar:4.2.1.RELEASE]

at com.vmware.vcac.authentication.http.spring.BaseTokenAuthenticationFilter.doFilter(BaseTokenAuthenticationFilter.java:45) [platform-security-sso-client-7.4.0-SNAPSHOT.jar:?]

at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) [spring-security-web-4.2.1.RELEASE.jar:4.2.1.RELEASE]

at org.springframework.security.oauth2.provider.authentication.OAuth2AuthenticationProcessingFilter.doFilter(OAuth2AuthenticationProcessingFilter.java:176) [spring-security-oauth2-2.0.12.RELEASE.jar:?]

at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) [spring-security-web-4.2.1.RELEASE.jar:4.2.1.RELEASE]

at com.vmware.vcac.authentication.http.tenancy.TenancyContextFilter.doFilterWithTenancyContext(TenancyContextFilter.java:67) [platform-security-sso-client-7.4.0-SNAPSHOT.jar:?]

at com.vmware.vcac.authentication.http.tenancy.TenancyContextFilter.doFilter(TenancyContextFilter.java:54) [platform-security-sso-client-7.4.0-SNAPSHOT.jar:?]

at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) [spring-security-web-4.2.1.RELEASE.jar:4.2.1.RELEASE]

at org.springframework.security.web.header.HeaderWriterFilter.doFilterInternal(HeaderWriterFilter.java:64) [spring-security-web-4.2.1.RELEASE.jar:4.2.1.RELEASE]

at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-4.3.13.RELEASE.jar:4.3.13.RELEASE]

at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) [spring-security-web-4.2.1.RELEASE.jar:4.2.1.RELEASE]

at org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter.doFilterInternal(WebAsyncManagerIntegrationFilter.java:56) [spring-security-web-4.2.1.RELEASE.jar:4.2.1.RELEASE]

at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-4.3.13.RELEASE.jar:4.3.13.RELEASE]

at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) [spring-security-web-4.2.1.RELEASE.jar:4.2.1.RELEASE]

at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:105) [spring-security-web-4.2.1.RELEASE.jar:4.2.1.RELEASE]

at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) [spring-security-web-4.2.1.RELEASE.jar:4.2.1.RELEASE]

at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:214) [spring-security-web-4.2.1.RELEASE.jar:4.2.1.RELEASE]

at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:177) [spring-security-web-4.2.1.RELEASE.jar:4.2.1.RELEASE]

at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:347) [spring-web-4.3.13.RELEASE.jar:4.3.13.RELEASE]

at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:263) [spring-web-4.3.13.RELEASE.jar:4.3.13.RELEASE]

at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [catalina.jar:8.5.29]

at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [catalina.jar:8.5.29]

at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:197) [spring-web-4.3.13.RELEASE.jar:4.3.13.RELEASE]

at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-4.3.13.RELEASE.jar:4.3.13.RELEASE]

at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [catalina.jar:8.5.29]

at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [catalina.jar:8.5.29]

at com.vmware.vcac.platform.service.rest.filter.DefaultContentTypeFilter.doFilter(DefaultContentTypeFilter.java:69) [platform-service-config-7.4.0-SNAPSHOT.jar:?]

at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [catalina.jar:8.5.29]

at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [catalina.jar:8.5.29]

at com.vmware.vcac.platform.trace.servlet.TraceHttpRequestFilter.doFilter(TraceHttpRequestFilter.java:50) [platform-trace-servlet-7.4.0-SNAPSHOT.jar:?]

at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [catalina.jar:8.5.29]

at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [catalina.jar:8.5.29]

at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:198) [catalina.jar:8.5.29]

at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96) [catalina.jar:8.5.29]

at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:496) [catalina.jar:8.5.29]

at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:140) [catalina.jar:8.5.29]

at org.apache.catalina.valves.RemoteIpValve.invoke(RemoteIpValve.java:677) [catalina.jar:8.5.29]

at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:81) [catalina.jar:8.5.29]

at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:650) [catalina.jar:8.5.29]

at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:87) [catalina.jar:8.5.29]

at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:342) [catalina.jar:8.5.29]

at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:803) [tomcat-coyote.jar:8.5.29]

at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66) [tomcat-coyote.jar:8.5.29]

at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:790) [tomcat-coyote.jar:8.5.29]

at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1459) [tomcat-coyote.jar:8.5.29]

at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49) [tomcat-coyote.jar:8.5.29]

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_161]

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_161]

at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) [tomcat-util.jar:8.5.29]

at java.lang.Thread.run(Thread.java:748) [?:1.8.0_161]

Caused by: java.net.ConnectException: Connection refused (Connection refused)

at java.net.PlainSocketImpl.socketConnect(Native Method) ~[?:1.8.0_161]

at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) ~[?:1.8.0_161]

at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) ~[?:1.8.0_161]

at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) ~[?:1.8.0_161]

at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[?:1.8.0_161]

at java.net.Socket.connect(Socket.java:589) ~[?:1.8.0_161]

at com.rabbitmq.client.impl.FrameHandlerFactory.create(FrameHandlerFactory.java:47) ~[amqp-client-3.6.5.jar:?]

at com.rabbitmq.client.ConnectionFactory.newConnection(ConnectionFactory.java:822) ~[amqp-client-3.6.5.jar:?]

at com.rabbitmq.client.ConnectionFactory.newConnection(ConnectionFactory.java:778) ~[amqp-client-3.6.5.jar:?]

at com.rabbitmq.client.ConnectionFactory.newConnection(ConnectionFactory.java:898) ~[amqp-client-3.6.5.jar:?]

at org.springframework.amqp.rabbit.connection.AbstractConnectionFactory.createBareConnection(AbstractConnectionFactory.java:297) ~[spring-rabbit-1.6.11.RELEASE.jar:?]

... 104 more

[UTC:2018-04-22 19:20:02,726 Local:2018-04-22 19:20:02,726] vcac: [component="cafe:event-broker" priority="ERROR" thread="tomcat-http--33" tenant="vsphere.local" context="ZcAo7KiR" parent="" token="ZcAo7KiR"] com.vmware.vcac.core.event.broker.service.impl.SubscribeServiceImpl.pollEvents:154 - Error on receive and convert from queue name :ebs.com.vmware.csp.iaas.blueprint.service.machine.lifecycle.provision__b46f8623-3da9-3f8f-b954-fbadd6895827

org.springframework.amqp.AmqpConnectException: java.net.ConnectException: Connection refused (Connection refused)

at org.springframework.amqp.rabbit.support.RabbitExceptionTranslator.convertRabbitAccessException(RabbitExceptionTranslator.java:62) ~[spring-rabbit-1.6.11.RELEASE.jar:?]

at org.springframework.amqp.rabbit.connection.AbstractConnectionFactory.createBareConnection(AbstractConnectionFactory.java:306) ~[spring-rabbit-1.6.11.RELEASE.jar:?]

at org.springframework.amqp.rabbit.connection.CachingConnectionFactory.createConnection(CachingConnectionFactory.java:561) ~[spring-rabbit-1.6.11.RELEASE.jar:?]

at org.springframework.amqp.rabbit.core.RabbitTemplate.doExecute(RabbitTemplate.java:1431) ~[spring-rabbit-1.6.11.RELEASE.jar:?]

at org.springframework.amqp.rabbit.core.RabbitTemplate.execute(RabbitTemplate.java:1412) ~[spring-rabbit-1.6.11.RELEASE.jar:?]

at org.springframework.amqp.rabbit.core.RabbitTemplate.doReceiveNoWait(RabbitTemplate.java:841) ~[spring-rabbit-1.6.11.RELEASE.jar:?]

at org.springframework.amqp.rabbit.core.RabbitTemplate.receiveAndConvert(RabbitTemplate.java:931) ~[spring-rabbit-1.6.11.RELEASE.jar:?]

at org.springframework.amqp.rabbit.core.RabbitTemplate.receiveAndConvert(RabbitTemplate.java:921) ~[spring-rabbit-1.6.11.RELEASE.jar:?]

at org.springframework.integration.amqp.channel.PollableAmqpChannel.doReceive(PollableAmqpChannel.java:210) ~[spring-integration-amqp-4.3.13.RELEASE.jar:4.3.13.RELEASE]

at org.springframework.integration.amqp.channel.PollableAmqpChannel.receive(PollableAmqpChannel.java:169) ~[spring-integration-amqp-4.3.13.RELEASE.jar:4.3.13.RELEASE]

at com.vmware.vcac.core.event.broker.service.impl.SubscribeServiceImpl.pollEvents(SubscribeServiceImpl.java:139) [classes/:?]

at com.vmware.vcac.core.event.broker.controller.SubscriptionController.pollEvents(SubscriptionController.java:263) [classes/:?]

at com.vmware.vcac.core.event.broker.controller.SubscriptionController$$FastClassBySpringCGLIB$$c3c3727.invoke(<generated>) [classes/:?]

at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204) [spring-core-4.3.13.RELEASE.jar:4.3.13.RELEASE]

at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:738) [spring-aop-4.3.13.RELEASE.jar:4.3.13.RELEASE]

at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157) [spring-aop-4.3.13.RELEASE.jar:4.3.13.RELEASE]

at org.springframework.security.access.intercept.aopalliance.MethodSecurityInterceptor.invoke(MethodSecurityInterceptor.java:69) [spring-security-core-4.2.1.RELEASE.jar:4.2.1.RELEASE]

at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) [spring-aop-4.3.13.RELEASE.jar:4.3.13.RELEASE]

at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:673) [spring-aop-4.3.13.RELEASE.jar:4.3.13.RELEASE]

at com.vmware.vcac.core.event.broker.controller.SubscriptionController$$EnhancerBySpringCGLIB$$5f26e10.pollEvents(<generated>) [classes/:?]

at sun.reflect.GeneratedMethodAccessor2051.invoke(Unknown Source) ~[?:?]

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_161]

at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_161]

at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205) [spring-web-4.3.13.RELEASE.jar:4.3.13.RELEASE]

at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:133) [spring-web-4.3.13.RELEASE.jar:4.3.13.RELEASE]

at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:97) [spring-webmvc-4.3.13.RELEASE.jar:4.3.13.RELEASE]

at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:827) [spring-webmvc-4.3.13.RELEASE.jar:4.3.13.RELEASE]

at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:738) [spring-webmvc-4.3.13.RELEASE.jar:4.3.13.RELEASE]

at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:85) [spring-webmvc-4.3.13.RELEASE.jar:4.3.13.RELEASE]

at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:967) [spring-webmvc-4.3.13.RELEASE.jar:4.3.13.RELEASE]

at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:901) [spring-webmvc-4.3.13.RELEASE.jar:4.3.13.RELEASE]

at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:970) [spring-webmvc-4.3.13.RELEASE.jar:4.3.13.RELEASE]

at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:861) [spring-webmvc-4.3.13.RELEASE.jar:4.3.13.RELEASE]

at javax.servlet.http.HttpServlet.service(HttpServlet.java:635) [servlet-api.jar:?]

at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:846) [spring-webmvc-4.3.13.RELEASE.jar:4.3.13.RELEASE]

at javax.servlet.http.HttpServlet.service(HttpServlet.java:742) [servlet-api.jar:?]

at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231) [catalina.jar:8.5.29]

at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [catalina.jar:8.5.29]

at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52) [tomcat-websocket.jar:8.5.29]

at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [catalina.jar:8.5.29]

at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [catalina.jar:8.5.29]

at org.apache.logging.log4j.web.Log4jServletFilter.doFilter(Log4jServletFilter.java:71) [platform-log4j-web-custom-7.4.0-SNAPSHOT.jar:?]

at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [catalina.jar:8.5.29]

at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [catalina.jar:8.5.29]

at com.vmware.vcac.platform.metrics.servlet.MetricsAbstractInstrumentedFilter.doFilter(MetricsAbstractInstrumentedFilter.java:95) [platform-metrics-servlet-7.4.0-SNAPSHOT.jar:?]

at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [catalina.jar:8.5.29]

at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [catalina.jar:8.5.29]

at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:317) [spring-security-web-4.2.1.RELEASE.jar:4.2.1.RELEASE]

at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:127) [spring-security-web-4.2.1.RELEASE.jar:4.2.1.RELEASE]

at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:91) [spring-security-web-4.2.1.RELEASE.jar:4.2.1.RELEASE]

at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) [spring-security-web-4.2.1.RELEASE.jar:4.2.1.RELEASE]

at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:114) [spring-security-web-4.2.1.RELEASE.jar:4.2.1.RELEASE]

at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) [spring-security-web-4.2.1.RELEASE.jar:4.2.1.RELEASE]

at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:111) [spring-security-web-4.2.1.RELEASE.jar:4.2.1.RELEASE]

at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) [spring-security-web-4.2.1.RELEASE.jar:4.2.1.RELEASE]

at org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:170) [spring-security-web-4.2.1.RELEASE.jar:4.2.1.RELEASE]

at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) [spring-security-web-4.2.1.RELEASE.jar:4.2.1.RELEASE]

at com.vmware.vcac.authentication.http.spring.BaseTokenAuthenticationFilter.doFilter(BaseTokenAuthenticationFilter.java:45) [platform-security-sso-client-7.4.0-SNAPSHOT.jar:?]

at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) [spring-security-web-4.2.1.RELEASE.jar:4.2.1.RELEASE]

at org.springframework.security.oauth2.provider.authentication.OAuth2AuthenticationProcessingFilter.doFilter(OAuth2AuthenticationProcessingFilter.java:176) [spring-security-oauth2-2.0.12.RELEASE.jar:?]

at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) [spring-security-web-4.2.1.RELEASE.jar:4.2.1.RELEASE]

at com.vmware.vcac.authentication.http.tenancy.TenancyContextFilter.doFilterWithTenancyContext(TenancyContextFilter.java:67) [platform-security-sso-client-7.4.0-SNAPSHOT.jar:?]

at com.vmware.vcac.authentication.http.tenancy.TenancyContextFilter.doFilter(TenancyContextFilter.java:54) [platform-security-sso-client-7.4.0-SNAPSHOT.jar:?]

at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) [spring-security-web-4.2.1.RELEASE.jar:4.2.1.RELEASE]

at org.springframework.security.web.header.HeaderWriterFilter.doFilterInternal(HeaderWriterFilter.java:64) [spring-security-web-4.2.1.RELEASE.jar:4.2.1.RELEASE]

at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-4.3.13.RELEASE.jar:4.3.13.RELEASE]

at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) [spring-security-web-4.2.1.RELEASE.jar:4.2.1.RELEASE]

at org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter.doFilterInternal(WebAsyncManagerIntegrationFilter.java:56) [spring-security-web-4.2.1.RELEASE.jar:4.2.1.RELEASE]

at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-4.3.13.RELEASE.jar:4.3.13.RELEASE]

at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) [spring-security-web-4.2.1.RELEASE.jar:4.2.1.RELEASE]

at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:105) [spring-security-web-4.2.1.RELEASE.jar:4.2.1.RELEASE]

at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) [spring-security-web-4.2.1.RELEASE.jar:4.2.1.RELEASE]

at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:214) [spring-security-web-4.2.1.RELEASE.jar:4.2.1.RELEASE]

at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:177) [spring-security-web-4.2.1.RELEASE.jar:4.2.1.RELEASE]

at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:347) [spring-web-4.3.13.RELEASE.jar:4.3.13.RELEASE]

at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:263) [spring-web-4.3.13.RELEASE.jar:4.3.13.RELEASE]

at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [catalina.jar:8.5.29]

at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [catalina.jar:8.5.29]

at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:197) [spring-web-4.3.13.RELEASE.jar:4.3.13.RELEASE]

at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-4.3.13.RELEASE.jar:4.3.13.RELEASE]

at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [catalina.jar:8.5.29]

at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [catalina.jar:8.5.29]

at com.vmware.vcac.platform.service.rest.filter.DefaultContentTypeFilter.doFilter(DefaultContentTypeFilter.java:69) [platform-service-config-7.4.0-SNAPSHOT.jar:?]

at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [catalina.jar:8.5.29]

at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [catalina.jar:8.5.29]

at com.vmware.vcac.platform.trace.servlet.TraceHttpRequestFilter.doFilter(TraceHttpRequestFilter.java:50) [platform-trace-servlet-7.4.0-SNAPSHOT.jar:?]

at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [catalina.jar:8.5.29]

at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [catalina.jar:8.5.29]

at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:198) [catalina.jar:8.5.29]

at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96) [catalina.jar:8.5.29]

at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:496) [catalina.jar:8.5.29]

at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:140) [catalina.jar:8.5.29]

at org.apache.catalina.valves.RemoteIpValve.invoke(RemoteIpValve.java:677) [catalina.jar:8.5.29]

at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:81) [catalina.jar:8.5.29]

at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:650) [catalina.jar:8.5.29]

at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:87) [catalina.jar:8.5.29]

at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:342) [catalina.jar:8.5.29]

at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:803) [tomcat-coyote.jar:8.5.29]

at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66) [tomcat-coyote.jar:8.5.29]

at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:790) [tomcat-coyote.jar:8.5.29]

at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1459) [tomcat-coyote.jar:8.5.29]

at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49) [tomcat-coyote.jar:8.5.29]

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_161]

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_161]

at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) [tomcat-util.jar:8.5.29]

at java.lang.Thread.run(Thread.java:748) [?:1.8.0_161]

Caused by: java.net.ConnectException: Connection refused (Connection refused)

at java.net.PlainSocketImpl.socketConnect(Native Method) ~[?:1.8.0_161]

at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) ~[?:1.8.0_161]

at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) ~[?:1.8.0_161]

at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) ~[?:1.8.0_161]

at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[?:1.8.0_161]

at java.net.Socket.connect(Socket.java:589) ~[?:1.8.0_161]

at com.rabbitmq.client.impl.FrameHandlerFactory.create(FrameHandlerFactory.java:47) ~[amqp-client-3.6.5.jar:?]

at com.rabbitmq.client.ConnectionFactory.newConnection(ConnectionFactory.java:822) ~[amqp-client-3.6.5.jar:?]

at com.rabbitmq.client.ConnectionFactory.newConnection(ConnectionFactory.java:778) ~[amqp-client-3.6.5.jar:?]

at com.rabbitmq.client.ConnectionFactory.newConnection(ConnectionFactory.java:898) ~[amqp-client-3.6.5.jar:?]

at org.springframework.amqp.rabbit.connection.AbstractConnectionFactory.createBareConnection(AbstractConnectionFactory.java:297) ~[spring-rabbit-1.6.11.RELEASE.jar:?]

... 104 more

[UTC:2018-04-22 19:20:02,730 Local:2018-04-22 19:20:02,730] vcac: [component="cafe:event-broker" priority="ERROR" thread="tomcat-http--33" tenant="vsphere.local" context="ZcAo7KiR" parent="" token="ZcAo7KiR"] com.vmware.vcac.core.event.broker.service.impl.SubscribeServiceImpl.pollEvents:154 - Error on receive and convert from queue name :ebs.com.vmware.csp.iaas.blueprint.service.machine.lifecycle.provision__b46f8623-3da9-3f8f-b954-fbadd6895827

org.springframework.amqp.AmqpConnectException: java.net.ConnectException: Connection refused (Connection refused)

at org.springframework.amqp.rabbit.support.RabbitExceptionTranslator.convertRabbitAccessException(RabbitExceptionTranslator.java:62) ~[spring-rabbit-1.6.11.RELEASE.jar:?]

at org.springframework.amqp.rabbit.connection.AbstractConnectionFactory.createBareConnection(AbstractConnectionFactory.java:306) ~[spring-rabbit-1.6.11.RELEASE.jar:?]

at org.springframework.amqp.rabbit.connection.CachingConnectionFactory.createConnection(CachingConnectionFactory.java:561) ~[spring-rabbit-1.6.11.RELEASE.jar:?]

at org.springframework.amqp.rabbit.core.RabbitTemplate.doExecute(RabbitTemplate.java:1431) ~[spring-rabbit-1.6.11.RELEASE.jar:?]

at org.springframework.amqp.rabbit.core.RabbitTemplate.execute(RabbitTemplate.java:1412) ~[spring-rabbit-1.6.11.RELEASE.jar:?]

at org.springframework.amqp.rabbit.core.RabbitTemplate.doReceiveNoWait(RabbitTemplate.java:841) ~[spring-rabbit-1.6.11.RELEASE.jar:?]

at org.springframework.amqp.rabbit.core.RabbitTemplate.receiveAndConvert(RabbitTemplate.java:931) ~[spring-rabbit-1.6.11.RELEASE.jar:?]

at org.springframework.amqp.rabbit.core.RabbitTemplate.receiveAndConvert(RabbitTemplate.java:921) ~[spring-rabbit-1.6.11.RELEASE.jar:?]

at org.springframework.integration.amqp.channel.PollableAmqpChannel.doReceive(PollableAmqpChannel.java:210) ~[spring-integration-amqp-4.3.13.RELEASE.jar:4.3.13.RELEASE]

at org.springframework.integration.amqp.channel.PollableAmqpChannel.receive(PollableAmqpChannel.java:169) ~[spring-integration-amqp-4.3.13.RELEASE.jar:4.3.13.RELEASE]

at com.vmware.vcac.core.event.broker.service.impl.SubscribeServiceImpl.pollEvents(SubscribeServiceImpl.java:139) [classes/:?]

at com.vmware.vcac.core.event.broker.controller.SubscriptionController.pollEvents(SubscriptionController.java:263) [classes/:?]

at com.vmware.vcac.core.event.broker.controller.SubscriptionController$$FastClassBySpringCGLIB$$c3c3727.invoke(<generated>) [classes/:?]

at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204) [spring-core-4.3.13.RELEASE.jar:4.3.13.RELEASE]

at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:738) [spring-aop-4.3.13.RELEASE.jar:4.3.13.RELEASE]

at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157) [spring-aop-4.3.13.RELEASE.jar:4.3.13.RELEASE]

at org.springframework.security.access.intercept.aopalliance.MethodSecurityInterceptor.invoke(MethodSecurityInterceptor.java:69) [spring-security-core-4.2.1.RELEASE.jar:4.2.1.RELEASE]

at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) [spring-aop-

[r0j · Accepted solution]

opened support ticket 18781199104 on this.

The following resolved our issue.

Pre-upgrade task

https://kb.vmware.com/s/article/2151841

Post-upgrade tasks

1. Take a backup copy of the  /etc/init.d/vrhb-service

   cp /etc/init.d/vrhb-service /etc/init.d/vrhb-service.bak

2. Stop the health service

   service vrhb-service stop

3. Open /etc/init.d/vrhb-service in vi

   vi /etc/init.d/vrhb-service

4. Find all instances of "-XX:+HeapDumpOnOutOfMemoryError" (without the quotes) and delete (you

should find this twice)

Note: this entry did not exist pre-upgrade

5. Delete your currently configuration

   rm -rf /var/lib/vrhb/service-host/sandbox

   rm -rf /var/lib/vrhb/vra-test-host/sandbox

6. Start the health service

   service vrhb-service start

7. Recreate your health check configuration

Fix vRO workflow logging

https://communities.vmware.com/message/2762334#2762334

[r0j]

With all the 7.3-7.4 upgrades and rollbacks on our DEV environment, and working with vmware support, the IaaS pre-requisites (jre 1.8 u1.61, .NET 4.5.2) got missed on one run.

You get the following message on the update status page in the vami, which appears to show the IaaS components upgraded successfully to version 7.3.0.

IaaS upgrade 100% completed

Node img-tun-vra7w3.caf.cloud

Database 7.3.0.10750

Website 7.3.0.10750

ModelManagerData 7.3.0.10750

ModelManagerWeb 7.3.0.10750

ManagerService 7.3.0.10750

ManagementAgent 7.3.0.10750

DemOrchestrator 7.3.0.10750

DemWorker       7.3.0.10750

WAPI     7.3.0.10750

vSphereAgent   7.3.0.10750

Upgrade completed successfully.

Upon further inspection of the Management Agent setup install logs, you will find the install failed.

Quickly resolved by installing the pre-requisite .NET and JRE components on the Iaas node(s), and running the upgrade script in /usr/lib/vcac/tools/upgrade/ from the console on the vRA appliance.

A self induced error, worth mentioning...

:smileylaugh:

[Czernobog]

Thanks for posting the observation. I've also tried to upgrade a few weeks ago, with live support from engineering. We spent around 6 hours tripping over various issues during the upgrade process (who's better at testing the product releases than VMware's clients, amirite??) and after finally getting the environment running I have noticed that vIDM won't sync my AD Directory. Turns out that you can no longer control the ldap query sizes from vRA, but instead IDM uses parameters set in the AD, which, when different from default values, won't let you sync large directories. This is (was?) of course not documented, had to roll back too:|

[gradinka]

these are some settings on AD side which are now being honored:

MaxPageSize

MaxValRange

the MaxValRange being set to zero (MaxValRange = 0) is the one that can cause trouble, as IDM will now take it into account and sync 'zero' users 😕