14 Replies Latest reply on May 25, 2018 3:07 AM by gradinka

    Upgrade vRealize Automation 7.3.0 to 7.4 observations

    r0j Novice

      We did a dry run upgrade of our Dev minimal install environment from 7.3-7.4.

      This DEV environment was a fresh 7.3 build 6 months ago.

       

      Initial identified issue was vRO stopped logging on workflows, which was resolved by the following (thank you poster).

      https://communities.vmware.com/message/2762334#2762334

       

      The upgrade proceeded to run extremely slow for about a day, before grenading itself.

      The catalina.out filled itself to 5+GB, before blowing out the /storage/log partition on the appliance.

      Deleted the catalina.out and rebooted.

      Everything got ugly after that...

      rabbitmq wouldn't start

      composition-service - com.vmware.csp.component.cafe.composition wouldn't start.

       

      Environment was fubar'd, had to roll back iaas and vra and databases.

       

      Anyone have similar experiences?

        • 1. Re: Upgrade vRealize Automation 7.3.0 to 7.4 observations
          daphnissov Guru
          vExpertCommunity Warriors

          Haven't done it myself (but is planned). I'm in the same boat with a newly-deployed 7.3 instance. I've never had any success with vRA upgrades, even on a simple minimal install. Fresh install is what I try to do now.

          • 2. Re: Upgrade vRealize Automation 7.3.0 to 7.4 observations
            r0j Novice

            Thanks for the feedback and input daphnissov..

             

            I am getting the very same impression, and feel that future versions should just incorporate this process, where appliances are just throwaways.

             

            We use codestream to move content between vRA environments, however there are some deficiencies when moving tenant info / configurations and such, where improvements to that tool would be helpful with this.

             

             

            • 3. Re: Upgrade vRealize Automation 7.3.0 to 7.4 observations
              gradinka Master
              VMware Employees

              you can always leverage migration, and not upgrade.

              that's also built-in

               

              --

              but I wonder what happened with this upgrade running for so long (that is not uncommon, depending on the amount of items in the environment).

              can we get a log bundle with more details

              • 4. Re: Upgrade vRealize Automation 7.3.0 to 7.4 observations
                r0j Novice

                So..... I attempted to do another 7.3-7.4 upgrade on my DEV environment on Friday afternoon.

                 

                Everything went smooth, no errors in any of the upgrade logs, and the environment was running great.

                 

                come on on monday everything in DEV vRA is down, and again the catalina.out log is 5+GB and the log partition is completely full.

                delete the log and reboot, rabbitmq is dead and unrecoverable.

                 

                Can anyone provide clues as to wth is happening with my particular upgrade?

                 

                I pulled down the log bundle and catalina.out, and will be opening a support ticket.

                 

                For now I have to rollback my DEV. Again.

                • 5. Re: Upgrade vRealize Automation 7.3.0 to 7.4 observations
                  sbeaver Guru
                  User ModeratorsvExpert

                  I just spent the weekend upgrading the environment.  I knew I was going to have some issues letting the appliance upgrade process also update the IaaS components in my distributed environment and ended up separating things out so the appliance upgrade would just update the appliances and then did the IaaS independently.  I do have a recommendation from my experience...  If you have multiple appliances in your setup. consolidate everything to the ,master node so that the database and the sync connector are the same and then split up again after the upgrade and for what it is worth, clear out all the old log files before you take you snapshot

                  • 6. Re: Upgrade vRealize Automation 7.3.0 to 7.4 observations
                    gradinka Master
                    VMware Employees

                    some clues in the catalina.out?

                    I guess we'll be looking into the SR anyways.

                    • 7. Re: Upgrade vRealize Automation 7.3.0 to 7.4 observations
                      r0j Novice

                      The catalina.out is in the log bundle I uploaded in SR # 18781199104.

                       

                      we havnt rolled back dev yet in case we get someone on the line quick to look at this.

                       

                      root /dev/sda1 also was at 100%

                      • 8. Re: Upgrade vRealize Automation 7.3.0 to 7.4 observations
                        sbeaver Guru
                        User ModeratorsvExpert

                        Have you rebooted that node lately since it was full?  I have a node that was 80% full and I could not find where but a reboot and all of a sudden it was back to like 10%

                        • 9. Re: Upgrade vRealize Automation 7.3.0 to 7.4 observations
                          r0j Novice

                          yes, rebooted multiple times, the reboot resolved the log partition not seeing the 5GB of free space, after deleting the catalina.out.

                           

                          we rolled back to 7.3, and will work with vmware support to get through this.

                           

                          thanks for the responses..

                          • 10. Re: Upgrade vRealize Automation 7.3.0 to 7.4 observations
                            r0j Novice

                            Cracked open the 5GB catalina.out, here is a sample of the repeated errors filling the log about 18-24 hours after the upgrade.

                            I have performed the upgrade 2x with the same results.

                            Definition of insanity?

                            Thoughts?

                             

                            [UTC:2018-04-21 22:10:06,677 Local:2018-04-21 22:10:06,677] vcac: [component="cafe:notification" priority="WARN" thread="notification-service-subscrAmqpExec-5" tenant="" context="" parent="" token=""] org.springframework.integration.channel.MessagePublishingErrorHandler.handleError:123 - Error message was not delivered.

                            org.springframework.messaging.MessageDeliveryException: failed to send Message to channel 'errorChannel'; nested exception is org.springframework.amqp.AmqpConnectException: java.net.ConnectException: Connection refused (Connection refused)

                            at org.springframework.integration.channel.AbstractMessageChannel.send(AbstractMessageChannel.java:451) ~[spring-integration-core-4.3.13.RELEASE.jar:4.3.13.RELEASE]

                            at org.springframework.integration.channel.MessagePublishingErrorHandler.handleError(MessagePublishingErrorHandler.java:114) [spring-integration-core-4.3.13.RELEASE.jar:4.3.13.RELEASE]

                            at org.springframework.integration.util.ErrorHandlingTaskExecutor$1.run(ErrorHandlingTaskExecutor.java:58) [spring-integration-core-4.3.13.RELEASE.jar:4.3.13.RELEASE]

                            at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_161]

                            at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_161]

                            at java.lang.Thread.run(Thread.java:748) [?:1.8.0_161]

                            Caused by: org.springframework.amqp.AmqpConnectException: java.net.ConnectException: Connection refused (Connection refused)

                            at org.springframework.amqp.rabbit.support.RabbitExceptionTranslator.convertRabbitAccessException(RabbitExceptionTranslator.java:62) ~[spring-rabbit-1.6.11.RELEASE.jar:?]

                            at org.springframework.amqp.rabbit.connection.AbstractConnectionFactory.createBareConnection(AbstractConnectionFactory.java:306) ~[spring-rabbit-1.6.11.RELEASE.jar:?]

                            at org.springframework.amqp.rabbit.connection.CachingConnectionFactory.createConnection(CachingConnectionFactory.java:561) ~[spring-rabbit-1.6.11.RELEASE.jar:?]

                            at org.springframework.amqp.rabbit.core.RabbitTemplate.doExecute(RabbitTemplate.java:1431) ~[spring-rabbit-1.6.11.RELEASE.jar:?]

                            at org.springframework.amqp.rabbit.core.RabbitTemplate.access$2400(RabbitTemplate.java:132) ~[spring-rabbit-1.6.11.RELEASE.jar:?]

                            at org.springframework.amqp.rabbit.core.RabbitTemplate$10.doWithRetry(RabbitTemplate.java:1399) ~[spring-rabbit-1.6.11.RELEASE.jar:?]

                            at org.springframework.retry.support.RetryTemplate.doExecute(RetryTemplate.java:276) ~[spring-retry-1.1.3.RELEASE.jar:?]

                            at org.springframework.retry.support.RetryTemplate.execute(RetryTemplate.java:172) ~[spring-retry-1.1.3.RELEASE.jar:?]

                            at org.springframework.amqp.rabbit.core.RabbitTemplate.execute(RabbitTemplate.java:1395) ~[spring-rabbit-1.6.11.RELEASE.jar:?]

                            at org.springframework.amqp.rabbit.core.RabbitTemplate.send(RabbitTemplate.java:714) ~[spring-rabbit-1.6.11.RELEASE.jar:?]

                            at org.springframework.amqp.rabbit.core.RabbitTemplate.convertAndSend(RabbitTemplate.java:815) ~[spring-rabbit-1.6.11.RELEASE.jar:?]

                            at org.springframework.amqp.rabbit.core.RabbitTemplate.convertAndSend(RabbitTemplate.java:805) ~[spring-rabbit-1.6.11.RELEASE.jar:?]

                            at com.vmware.vcac.platform.service.integration.channel.ExchangeBindablePollableAmqpChannelProxy.doSend(ExchangeBindablePollableAmqpChannelProxy.java:89) ~[platform-integration-7.4.0-SNAPSHOT.jar:?]

                            at org.springframework.integration.channel.AbstractMessageChannel.send(AbstractMessageChannel.java:425) ~[spring-integration-core-4.3.13.RELEASE.jar:4.3.13.RELEASE]

                            ... 5 more

                            Caused by: java.net.ConnectException: Connection refused (Connection refused)

                            at java.net.PlainSocketImpl.socketConnect(Native Method) ~[?:1.8.0_161]

                            at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) ~[?:1.8.0_161]

                            at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) ~[?:1.8.0_161]

                            at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) ~[?:1.8.0_161]

                            at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[?:1.8.0_161]

                            at java.net.Socket.connect(Socket.java:589) ~[?:1.8.0_161]

                            at com.rabbitmq.client.impl.FrameHandlerFactory.create(FrameHandlerFactory.java:47) ~[amqp-client-3.6.5.jar:?]

                            at com.rabbitmq.client.ConnectionFactory.newConnection(ConnectionFactory.java:822) ~[amqp-client-3.6.5.jar:?]

                            at com.rabbitmq.client.ConnectionFactory.newConnection(ConnectionFactory.java:778) ~[amqp-client-3.6.5.jar:?]

                            at com.rabbitmq.client.ConnectionFactory.newConnection(ConnectionFactory.java:898) ~[amqp-client-3.6.5.jar:?]

                            at org.springframework.amqp.rabbit.connection.AbstractConnectionFactory.createBareConnection(AbstractConnectionFactory.java:297) ~[spring-rabbit-1.6.11.RELEASE.jar:?]

                            at org.springframework.amqp.rabbit.connection.CachingConnectionFactory.createConnection(CachingConnectionFactory.java:561) ~[spring-rabbit-1.6.11.RELEASE.jar:?]

                            at org.springframework.amqp.rabbit.core.RabbitTemplate.doExecute(RabbitTemplate.java:1431) ~[spring-rabbit-1.6.11.RELEASE.jar:?]

                            at org.springframework.amqp.rabbit.core.RabbitTemplate.access$2400(RabbitTemplate.java:132) ~[spring-rabbit-1.6.11.RELEASE.jar:?]

                            at org.springframework.amqp.rabbit.core.RabbitTemplate$10.doWithRetry(RabbitTemplate.java:1399) ~[spring-rabbit-1.6.11.RELEASE.jar:?]

                            at org.springframework.retry.support.RetryTemplate.doExecute(RetryTemplate.java:276) ~[spring-retry-1.1.3.RELEASE.jar:?]

                            at org.springframework.retry.support.RetryTemplate.execute(RetryTemplate.java:172) ~[spring-retry-1.1.3.RELEASE.jar:?]

                            at org.springframework.amqp.rabbit.core.RabbitTemplate.execute(RabbitTemplate.java:1395) ~[spring-rabbit-1.6.11.RELEASE.jar:?]

                            at org.springframework.amqp.rabbit.core.RabbitTemplate.send(RabbitTemplate.java:714) ~[spring-rabbit-1.6.11.RELEASE.jar:?]

                            at org.springframework.amqp.rabbit.core.RabbitTemplate.convertAndSend(RabbitTemplate.java:815) ~[spring-rabbit-1.6.11.RELEASE.jar:?]

                            at org.springframework.amqp.rabbit.core.RabbitTemplate.convertAndSend(RabbitTemplate.java:805) ~[spring-rabbit-1.6.11.RELEASE.jar:?]

                            at com.vmware.vcac.platform.service.integration.channel.ExchangeBindablePollableAmqpChannelProxy.doSend(ExchangeBindablePollableAmqpChannelProxy.java:89) ~[platform-integration-7.4.0-SNAPSHOT.jar:?]

                            at org.springframework.integration.channel.AbstractMessageChannel.send(AbstractMessageChannel.java:425) ~[spring-integration-core-4.3.13.RELEASE.jar:4.3.13.RELEASE]

                            ... 5 more

                            [UTC:2018-04-22 19:20:02,723 Local:2018-04-22 19:20:02,723] vcac: [component="cafe:event-broker" priority="ERROR" thread="tomcat-http--33" tenant="vsphere.local" context="ZcAo7KiR" parent="" token="ZcAo7KiR"] com.vmware.vcac.core.event.broker.service.impl.SubscribeServiceImpl.pollEvents:154 - Error on receive and convert from queue name :ebs.com.vmware.csp.iaas.blueprint.service.machine.lifecycle.provision__b46f8623-3da9-3f8f-b954-fbadd6895827

                            org.springframework.amqp.AmqpConnectException: java.net.ConnectException: Connection refused (Connection refused)

                            at org.springframework.amqp.rabbit.support.RabbitExceptionTranslator.convertRabbitAccessException(RabbitExceptionTranslator.java:62) ~[spring-rabbit-1.6.11.RELEASE.jar:?]

                            at org.springframework.amqp.rabbit.connection.AbstractConnectionFactory.createBareConnection(AbstractConnectionFactory.java:306) ~[spring-rabbit-1.6.11.RELEASE.jar:?]

                            at org.springframework.amqp.rabbit.connection.CachingConnectionFactory.createConnection(CachingConnectionFactory.java:561) ~[spring-rabbit-1.6.11.RELEASE.jar:?]

                            at org.springframework.amqp.rabbit.core.RabbitTemplate.doExecute(RabbitTemplate.java:1431) ~[spring-rabbit-1.6.11.RELEASE.jar:?]

                            at org.springframework.amqp.rabbit.core.RabbitTemplate.execute(RabbitTemplate.java:1412) ~[spring-rabbit-1.6.11.RELEASE.jar:?]

                            at org.springframework.amqp.rabbit.core.RabbitTemplate.doReceiveNoWait(RabbitTemplate.java:841) ~[spring-rabbit-1.6.11.RELEASE.jar:?]

                            at org.springframework.amqp.rabbit.core.RabbitTemplate.receiveAndConvert(RabbitTemplate.java:931) ~[spring-rabbit-1.6.11.RELEASE.jar:?]

                            at org.springframework.amqp.rabbit.core.RabbitTemplate.receiveAndConvert(RabbitTemplate.java:921) ~[spring-rabbit-1.6.11.RELEASE.jar:?]

                            at org.springframework.integration.amqp.channel.PollableAmqpChannel.doReceive(PollableAmqpChannel.java:210) ~[spring-integration-amqp-4.3.13.RELEASE.jar:4.3.13.RELEASE]

                            at org.springframework.integration.amqp.channel.PollableAmqpChannel.receive(PollableAmqpChannel.java:169) ~[spring-integration-amqp-4.3.13.RELEASE.jar:4.3.13.RELEASE]

                            at com.vmware.vcac.core.event.broker.service.impl.SubscribeServiceImpl.pollEvents(SubscribeServiceImpl.java:139) [classes/:?]

                            at com.vmware.vcac.core.event.broker.controller.SubscriptionController.pollEvents(SubscriptionController.java:263) [classes/:?]

                            at com.vmware.vcac.core.event.broker.controller.SubscriptionController$$FastClassBySpringCGLIB$$c3c3727.invoke(<generated>) [classes/:?]

                            at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204) [spring-core-4.3.13.RELEASE.jar:4.3.13.RELEASE]

                            at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:738) [spring-aop-4.3.13.RELEASE.jar:4.3.13.RELEASE]

                            at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157) [spring-aop-4.3.13.RELEASE.jar:4.3.13.RELEASE]

                            at org.springframework.security.access.intercept.aopalliance.MethodSecurityInterceptor.invoke(MethodSecurityInterceptor.java:69) [spring-security-core-4.2.1.RELEASE.jar:4.2.1.RELEASE]

                            at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) [spring-aop-4.3.13.RELEASE.jar:4.3.13.RELEASE]

                            at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:673) [spring-aop-4.3.13.RELEASE.jar:4.3.13.RELEASE]

                            at com.vmware.vcac.core.event.broker.controller.SubscriptionController$$EnhancerBySpringCGLIB$$5f26e10.pollEvents(<generated>) [classes/:?]

                            at sun.reflect.GeneratedMethodAccessor2051.invoke(Unknown Source) ~[?:?]

                            at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_161]

                            at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_161]

                            at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205) [spring-web-4.3.13.RELEASE.jar:4.3.13.RELEASE]

                            at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:133) [spring-web-4.3.13.RELEASE.jar:4.3.13.RELEASE]

                            at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:97) [spring-webmvc-4.3.13.RELEASE.jar:4.3.13.RELEASE]

                            at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:827) [spring-webmvc-4.3.13.RELEASE.jar:4.3.13.RELEASE]

                            at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:738) [spring-webmvc-4.3.13.RELEASE.jar:4.3.13.RELEASE]

                            at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:85) [spring-webmvc-4.3.13.RELEASE.jar:4.3.13.RELEASE]

                            at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:967) [spring-webmvc-4.3.13.RELEASE.jar:4.3.13.RELEASE]

                            at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:901) [spring-webmvc-4.3.13.RELEASE.jar:4.3.13.RELEASE]

                            at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:970) [spring-webmvc-4.3.13.RELEASE.jar:4.3.13.RELEASE]

                            at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:861) [spring-webmvc-4.3.13.RELEASE.jar:4.3.13.RELEASE]

                            at javax.servlet.http.HttpServlet.service(HttpServlet.java:635) [servlet-api.jar:?]

                            at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:846) [spring-webmvc-4.3.13.RELEASE.jar:4.3.13.RELEASE]

                            at javax.servlet.http.HttpServlet.service(HttpServlet.java:742) [servlet-api.jar:?]

                            at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231) [catalina.jar:8.5.29]

                            at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [catalina.jar:8.5.29]

                            at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52) [tomcat-websocket.jar:8.5.29]

                            at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [catalina.jar:8.5.29]

                            at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [catalina.jar:8.5.29]

                            at org.apache.logging.log4j.web.Log4jServletFilter.doFilter(Log4jServletFilter.java:71) [platform-log4j-web-custom-7.4.0-SNAPSHOT.jar:?]

                            at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [catalina.jar:8.5.29]

                            at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [catalina.jar:8.5.29]

                            at com.vmware.vcac.platform.metrics.servlet.MetricsAbstractInstrumentedFilter.doFilter(MetricsAbstractInstrumentedFilter.java:95) [platform-metrics-servlet-7.4.0-SNAPSHOT.jar:?]

                            at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [catalina.jar:8.5.29]

                            at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [catalina.jar:8.5.29]

                            at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:317) [spring-security-web-4.2.1.RELEASE.jar:4.2.1.RELEASE]

                            at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:127) [spring-security-web-4.2.1.RELEASE.jar:4.2.1.RELEASE]

                            at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:91) [spring-security-web-4.2.1.RELEASE.jar:4.2.1.RELEASE]

                            at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) [spring-security-web-4.2.1.RELEASE.jar:4.2.1.RELEASE]

                            at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:114) [spring-security-web-4.2.1.RELEASE.jar:4.2.1.RELEASE]

                            at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) [spring-security-web-4.2.1.RELEASE.jar:4.2.1.RELEASE]

                            at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:111) [spring-security-web-4.2.1.RELEASE.jar:4.2.1.RELEASE]

                            at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) [spring-security-web-4.2.1.RELEASE.jar:4.2.1.RELEASE]

                            at org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:170) [spring-security-web-4.2.1.RELEASE.jar:4.2.1.RELEASE]

                            at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) [spring-security-web-4.2.1.RELEASE.jar:4.2.1.RELEASE]

                            at com.vmware.vcac.authentication.http.spring.BaseTokenAuthenticationFilter.doFilter(BaseTokenAuthenticationFilter.java:45) [platform-security-sso-client-7.4.0-SNAPSHOT.jar:?]

                            at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) [spring-security-web-4.2.1.RELEASE.jar:4.2.1.RELEASE]

                            at org.springframework.security.oauth2.provider.authentication.OAuth2AuthenticationProcessingFilter.doFilter(OAuth2AuthenticationProcessingFilter.java:176) [spring-security-oauth2-2.0.12.RELEASE.jar:?]

                            at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) [spring-security-web-4.2.1.RELEASE.jar:4.2.1.RELEASE]

                            at com.vmware.vcac.authentication.http.tenancy.TenancyContextFilter.doFilterWithTenancyContext(TenancyContextFilter.java:67) [platform-security-sso-client-7.4.0-SNAPSHOT.jar:?]

                            at com.vmware.vcac.authentication.http.tenancy.TenancyContextFilter.doFilter(TenancyContextFilter.java:54) [platform-security-sso-client-7.4.0-SNAPSHOT.jar:?]

                            at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) [spring-security-web-4.2.1.RELEASE.jar:4.2.1.RELEASE]

                            at org.springframework.security.web.header.HeaderWriterFilter.doFilterInternal(HeaderWriterFilter.java:64) [spring-security-web-4.2.1.RELEASE.jar:4.2.1.RELEASE]

                            at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-4.3.13.RELEASE.jar:4.3.13.RELEASE]

                            at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) [spring-security-web-4.2.1.RELEASE.jar:4.2.1.RELEASE]

                            at org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter.doFilterInternal(WebAsyncManagerIntegrationFilter.java:56) [spring-security-web-4.2.1.RELEASE.jar:4.2.1.RELEASE]

                            at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-4.3.13.RELEASE.jar:4.3.13.RELEASE]

                            at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) [spring-security-web-4.2.1.RELEASE.jar:4.2.1.RELEASE]

                            at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:105) [spring-security-web-4.2.1.RELEASE.jar:4.2.1.RELEASE]

                            at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) [spring-security-web-4.2.1.RELEASE.jar:4.2.1.RELEASE]

                            at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:214) [spring-security-web-4.2.1.RELEASE.jar:4.2.1.RELEASE]

                            at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:177) [spring-security-web-4.2.1.RELEASE.jar:4.2.1.RELEASE]

                            at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:347) [spring-web-4.3.13.RELEASE.jar:4.3.13.RELEASE]

                            at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:263) [spring-web-4.3.13.RELEASE.jar:4.3.13.RELEASE]

                            at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [catalina.jar:8.5.29]

                            at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [catalina.jar:8.5.29]

                            at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:197) [spring-web-4.3.13.RELEASE.jar:4.3.13.RELEASE]

                            at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-4.3.13.RELEASE.jar:4.3.13.RELEASE]

                            at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [catalina.jar:8.5.29]

                            at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [catalina.jar:8.5.29]

                            at com.vmware.vcac.platform.service.rest.filter.DefaultContentTypeFilter.doFilter(DefaultContentTypeFilter.java:69) [platform-service-config-7.4.0-SNAPSHOT.jar:?]

                            at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [catalina.jar:8.5.29]

                            at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [catalina.jar:8.5.29]

                            at com.vmware.vcac.platform.trace.servlet.TraceHttpRequestFilter.doFilter(TraceHttpRequestFilter.java:50) [platform-trace-servlet-7.4.0-SNAPSHOT.jar:?]

                            at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [catalina.jar:8.5.29]

                            at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [catalina.jar:8.5.29]

                            at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:198) [catalina.jar:8.5.29]

                            at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96) [catalina.jar:8.5.29]

                            at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:496) [catalina.jar:8.5.29]

                            at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:140) [catalina.jar:8.5.29]

                            at org.apache.catalina.valves.RemoteIpValve.invoke(RemoteIpValve.java:677) [catalina.jar:8.5.29]

                            at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:81) [catalina.jar:8.5.29]

                            at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:650) [catalina.jar:8.5.29]

                            at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:87) [catalina.jar:8.5.29]

                            at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:342) [catalina.jar:8.5.29]

                            at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:803) [tomcat-coyote.jar:8.5.29]

                            at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66) [tomcat-coyote.jar:8.5.29]

                            at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:790) [tomcat-coyote.jar:8.5.29]

                            at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1459) [tomcat-coyote.jar:8.5.29]

                            at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49) [tomcat-coyote.jar:8.5.29]

                            at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_161]

                            at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_161]

                            at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) [tomcat-util.jar:8.5.29]

                            at java.lang.Thread.run(Thread.java:748) [?:1.8.0_161]

                            Caused by: java.net.ConnectException: Connection refused (Connection refused)

                            at java.net.PlainSocketImpl.socketConnect(Native Method) ~[?:1.8.0_161]

                            at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) ~[?:1.8.0_161]

                            at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) ~[?:1.8.0_161]

                            at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) ~[?:1.8.0_161]

                            at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[?:1.8.0_161]

                            at java.net.Socket.connect(Socket.java:589) ~[?:1.8.0_161]

                            at com.rabbitmq.client.impl.FrameHandlerFactory.create(FrameHandlerFactory.java:47) ~[amqp-client-3.6.5.jar:?]

                            at com.rabbitmq.client.ConnectionFactory.newConnection(ConnectionFactory.java:822) ~[amqp-client-3.6.5.jar:?]

                            at com.rabbitmq.client.ConnectionFactory.newConnection(ConnectionFactory.java:778) ~[amqp-client-3.6.5.jar:?]

                            at com.rabbitmq.client.ConnectionFactory.newConnection(ConnectionFactory.java:898) ~[amqp-client-3.6.5.jar:?]

                            at org.springframework.amqp.rabbit.connection.AbstractConnectionFactory.createBareConnection(AbstractConnectionFactory.java:297) ~[spring-rabbit-1.6.11.RELEASE.jar:?]

                            ... 104 more

                            [UTC:2018-04-22 19:20:02,726 Local:2018-04-22 19:20:02,726] vcac: [component="cafe:event-broker" priority="ERROR" thread="tomcat-http--33" tenant="vsphere.local" context="ZcAo7KiR" parent="" token="ZcAo7KiR"] com.vmware.vcac.core.event.broker.service.impl.SubscribeServiceImpl.pollEvents:154 - Error on receive and convert from queue name :ebs.com.vmware.csp.iaas.blueprint.service.machine.lifecycle.provision__b46f8623-3da9-3f8f-b954-fbadd6895827

                            org.springframework.amqp.AmqpConnectException: java.net.ConnectException: Connection refused (Connection refused)

                            at org.springframework.amqp.rabbit.support.RabbitExceptionTranslator.convertRabbitAccessException(RabbitExceptionTranslator.java:62) ~[spring-rabbit-1.6.11.RELEASE.jar:?]

                            at org.springframework.amqp.rabbit.connection.AbstractConnectionFactory.createBareConnection(AbstractConnectionFactory.java:306) ~[spring-rabbit-1.6.11.RELEASE.jar:?]

                            at org.springframework.amqp.rabbit.connection.CachingConnectionFactory.createConnection(CachingConnectionFactory.java:561) ~[spring-rabbit-1.6.11.RELEASE.jar:?]

                            at org.springframework.amqp.rabbit.core.RabbitTemplate.doExecute(RabbitTemplate.java:1431) ~[spring-rabbit-1.6.11.RELEASE.jar:?]

                            at org.springframework.amqp.rabbit.core.RabbitTemplate.execute(RabbitTemplate.java:1412) ~[spring-rabbit-1.6.11.RELEASE.jar:?]

                            at org.springframework.amqp.rabbit.core.RabbitTemplate.doReceiveNoWait(RabbitTemplate.java:841) ~[spring-rabbit-1.6.11.RELEASE.jar:?]

                            at org.springframework.amqp.rabbit.core.RabbitTemplate.receiveAndConvert(RabbitTemplate.java:931) ~[spring-rabbit-1.6.11.RELEASE.jar:?]

                            at org.springframework.amqp.rabbit.core.RabbitTemplate.receiveAndConvert(RabbitTemplate.java:921) ~[spring-rabbit-1.6.11.RELEASE.jar:?]

                            at org.springframework.integration.amqp.channel.PollableAmqpChannel.doReceive(PollableAmqpChannel.java:210) ~[spring-integration-amqp-4.3.13.RELEASE.jar:4.3.13.RELEASE]

                            at org.springframework.integration.amqp.channel.PollableAmqpChannel.receive(PollableAmqpChannel.java:169) ~[spring-integration-amqp-4.3.13.RELEASE.jar:4.3.13.RELEASE]

                            at com.vmware.vcac.core.event.broker.service.impl.SubscribeServiceImpl.pollEvents(SubscribeServiceImpl.java:139) [classes/:?]

                            at com.vmware.vcac.core.event.broker.controller.SubscriptionController.pollEvents(SubscriptionController.java:263) [classes/:?]

                            at com.vmware.vcac.core.event.broker.controller.SubscriptionController$$FastClassBySpringCGLIB$$c3c3727.invoke(<generated>) [classes/:?]

                            at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204) [spring-core-4.3.13.RELEASE.jar:4.3.13.RELEASE]

                            at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:738) [spring-aop-4.3.13.RELEASE.jar:4.3.13.RELEASE]

                            at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157) [spring-aop-4.3.13.RELEASE.jar:4.3.13.RELEASE]

                            at org.springframework.security.access.intercept.aopalliance.MethodSecurityInterceptor.invoke(MethodSecurityInterceptor.java:69) [spring-security-core-4.2.1.RELEASE.jar:4.2.1.RELEASE]

                            at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) [spring-aop-4.3.13.RELEASE.jar:4.3.13.RELEASE]

                            at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:673) [spring-aop-4.3.13.RELEASE.jar:4.3.13.RELEASE]

                            at com.vmware.vcac.core.event.broker.controller.SubscriptionController$$EnhancerBySpringCGLIB$$5f26e10.pollEvents(<generated>) [classes/:?]

                            at sun.reflect.GeneratedMethodAccessor2051.invoke(Unknown Source) ~[?:?]

                            at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_161]

                            at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_161]

                            at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205) [spring-web-4.3.13.RELEASE.jar:4.3.13.RELEASE]

                            at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:133) [spring-web-4.3.13.RELEASE.jar:4.3.13.RELEASE]

                            at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:97) [spring-webmvc-4.3.13.RELEASE.jar:4.3.13.RELEASE]

                            at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:827) [spring-webmvc-4.3.13.RELEASE.jar:4.3.13.RELEASE]

                            at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:738) [spring-webmvc-4.3.13.RELEASE.jar:4.3.13.RELEASE]

                            at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:85) [spring-webmvc-4.3.13.RELEASE.jar:4.3.13.RELEASE]

                            at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:967) [spring-webmvc-4.3.13.RELEASE.jar:4.3.13.RELEASE]

                            at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:901) [spring-webmvc-4.3.13.RELEASE.jar:4.3.13.RELEASE]

                            at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:970) [spring-webmvc-4.3.13.RELEASE.jar:4.3.13.RELEASE]

                            at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:861) [spring-webmvc-4.3.13.RELEASE.jar:4.3.13.RELEASE]

                            at javax.servlet.http.HttpServlet.service(HttpServlet.java:635) [servlet-api.jar:?]

                            at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:846) [spring-webmvc-4.3.13.RELEASE.jar:4.3.13.RELEASE]

                            at javax.servlet.http.HttpServlet.service(HttpServlet.java:742) [servlet-api.jar:?]

                            at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231) [catalina.jar:8.5.29]

                            at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [catalina.jar:8.5.29]

                            at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52) [tomcat-websocket.jar:8.5.29]

                            at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [catalina.jar:8.5.29]

                            at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [catalina.jar:8.5.29]

                            at org.apache.logging.log4j.web.Log4jServletFilter.doFilter(Log4jServletFilter.java:71) [platform-log4j-web-custom-7.4.0-SNAPSHOT.jar:?]

                            at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [catalina.jar:8.5.29]

                            at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [catalina.jar:8.5.29]

                            at com.vmware.vcac.platform.metrics.servlet.MetricsAbstractInstrumentedFilter.doFilter(MetricsAbstractInstrumentedFilter.java:95) [platform-metrics-servlet-7.4.0-SNAPSHOT.jar:?]

                            at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [catalina.jar:8.5.29]

                            at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [catalina.jar:8.5.29]

                            at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:317) [spring-security-web-4.2.1.RELEASE.jar:4.2.1.RELEASE]

                            at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:127) [spring-security-web-4.2.1.RELEASE.jar:4.2.1.RELEASE]

                            at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:91) [spring-security-web-4.2.1.RELEASE.jar:4.2.1.RELEASE]

                            at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) [spring-security-web-4.2.1.RELEASE.jar:4.2.1.RELEASE]

                            at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:114) [spring-security-web-4.2.1.RELEASE.jar:4.2.1.RELEASE]

                            at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) [spring-security-web-4.2.1.RELEASE.jar:4.2.1.RELEASE]

                            at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:111) [spring-security-web-4.2.1.RELEASE.jar:4.2.1.RELEASE]

                            at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) [spring-security-web-4.2.1.RELEASE.jar:4.2.1.RELEASE]

                            at org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:170) [spring-security-web-4.2.1.RELEASE.jar:4.2.1.RELEASE]

                            at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) [spring-security-web-4.2.1.RELEASE.jar:4.2.1.RELEASE]

                            at com.vmware.vcac.authentication.http.spring.BaseTokenAuthenticationFilter.doFilter(BaseTokenAuthenticationFilter.java:45) [platform-security-sso-client-7.4.0-SNAPSHOT.jar:?]

                            at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) [spring-security-web-4.2.1.RELEASE.jar:4.2.1.RELEASE]

                            at org.springframework.security.oauth2.provider.authentication.OAuth2AuthenticationProcessingFilter.doFilter(OAuth2AuthenticationProcessingFilter.java:176) [spring-security-oauth2-2.0.12.RELEASE.jar:?]

                            at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) [spring-security-web-4.2.1.RELEASE.jar:4.2.1.RELEASE]

                            at com.vmware.vcac.authentication.http.tenancy.TenancyContextFilter.doFilterWithTenancyContext(TenancyContextFilter.java:67) [platform-security-sso-client-7.4.0-SNAPSHOT.jar:?]

                            at com.vmware.vcac.authentication.http.tenancy.TenancyContextFilter.doFilter(TenancyContextFilter.java:54) [platform-security-sso-client-7.4.0-SNAPSHOT.jar:?]

                            at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) [spring-security-web-4.2.1.RELEASE.jar:4.2.1.RELEASE]

                            at org.springframework.security.web.header.HeaderWriterFilter.doFilterInternal(HeaderWriterFilter.java:64) [spring-security-web-4.2.1.RELEASE.jar:4.2.1.RELEASE]

                            at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-4.3.13.RELEASE.jar:4.3.13.RELEASE]

                            at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) [spring-security-web-4.2.1.RELEASE.jar:4.2.1.RELEASE]

                            at org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter.doFilterInternal(WebAsyncManagerIntegrationFilter.java:56) [spring-security-web-4.2.1.RELEASE.jar:4.2.1.RELEASE]

                            at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-4.3.13.RELEASE.jar:4.3.13.RELEASE]

                            at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) [spring-security-web-4.2.1.RELEASE.jar:4.2.1.RELEASE]

                            at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:105) [spring-security-web-4.2.1.RELEASE.jar:4.2.1.RELEASE]

                            at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) [spring-security-web-4.2.1.RELEASE.jar:4.2.1.RELEASE]

                            at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:214) [spring-security-web-4.2.1.RELEASE.jar:4.2.1.RELEASE]

                            at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:177) [spring-security-web-4.2.1.RELEASE.jar:4.2.1.RELEASE]

                            at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:347) [spring-web-4.3.13.RELEASE.jar:4.3.13.RELEASE]

                            at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:263) [spring-web-4.3.13.RELEASE.jar:4.3.13.RELEASE]

                            at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [catalina.jar:8.5.29]

                            at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [catalina.jar:8.5.29]

                            at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:197) [spring-web-4.3.13.RELEASE.jar:4.3.13.RELEASE]

                            at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-4.3.13.RELEASE.jar:4.3.13.RELEASE]

                            at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [catalina.jar:8.5.29]

                            at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [catalina.jar:8.5.29]

                            at com.vmware.vcac.platform.service.rest.filter.DefaultContentTypeFilter.doFilter(DefaultContentTypeFilter.java:69) [platform-service-config-7.4.0-SNAPSHOT.jar:?]

                            at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [catalina.jar:8.5.29]

                            at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [catalina.jar:8.5.29]

                            at com.vmware.vcac.platform.trace.servlet.TraceHttpRequestFilter.doFilter(TraceHttpRequestFilter.java:50) [platform-trace-servlet-7.4.0-SNAPSHOT.jar:?]

                            at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [catalina.jar:8.5.29]

                            at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [catalina.jar:8.5.29]

                            at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:198) [catalina.jar:8.5.29]

                            at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96) [catalina.jar:8.5.29]

                            at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:496) [catalina.jar:8.5.29]

                            at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:140) [catalina.jar:8.5.29]

                            at org.apache.catalina.valves.RemoteIpValve.invoke(RemoteIpValve.java:677) [catalina.jar:8.5.29]

                            at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:81) [catalina.jar:8.5.29]

                            at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:650) [catalina.jar:8.5.29]

                            at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:87) [catalina.jar:8.5.29]

                            at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:342) [catalina.jar:8.5.29]

                            at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:803) [tomcat-coyote.jar:8.5.29]

                            at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66) [tomcat-coyote.jar:8.5.29]

                            at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:790) [tomcat-coyote.jar:8.5.29]

                            at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1459) [tomcat-coyote.jar:8.5.29]

                            at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49) [tomcat-coyote.jar:8.5.29]

                            at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_161]

                            at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_161]

                            at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) [tomcat-util.jar:8.5.29]

                            at java.lang.Thread.run(Thread.java:748) [?:1.8.0_161]

                            Caused by: java.net.ConnectException: Connection refused (Connection refused)

                            at java.net.PlainSocketImpl.socketConnect(Native Method) ~[?:1.8.0_161]

                            at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) ~[?:1.8.0_161]

                            at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) ~[?:1.8.0_161]

                            at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) ~[?:1.8.0_161]

                            at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[?:1.8.0_161]

                            at java.net.Socket.connect(Socket.java:589) ~[?:1.8.0_161]

                            at com.rabbitmq.client.impl.FrameHandlerFactory.create(FrameHandlerFactory.java:47) ~[amqp-client-3.6.5.jar:?]

                            at com.rabbitmq.client.ConnectionFactory.newConnection(ConnectionFactory.java:822) ~[amqp-client-3.6.5.jar:?]

                            at com.rabbitmq.client.ConnectionFactory.newConnection(ConnectionFactory.java:778) ~[amqp-client-3.6.5.jar:?]

                            at com.rabbitmq.client.ConnectionFactory.newConnection(ConnectionFactory.java:898) ~[amqp-client-3.6.5.jar:?]

                            at org.springframework.amqp.rabbit.connection.AbstractConnectionFactory.createBareConnection(AbstractConnectionFactory.java:297) ~[spring-rabbit-1.6.11.RELEASE.jar:?]

                            ... 104 more

                            [UTC:2018-04-22 19:20:02,730 Local:2018-04-22 19:20:02,730] vcac: [component="cafe:event-broker" priority="ERROR" thread="tomcat-http--33" tenant="vsphere.local" context="ZcAo7KiR" parent="" token="ZcAo7KiR"] com.vmware.vcac.core.event.broker.service.impl.SubscribeServiceImpl.pollEvents:154 - Error on receive and convert from queue name :ebs.com.vmware.csp.iaas.blueprint.service.machine.lifecycle.provision__b46f8623-3da9-3f8f-b954-fbadd6895827

                            org.springframework.amqp.AmqpConnectException: java.net.ConnectException: Connection refused (Connection refused)

                            at org.springframework.amqp.rabbit.support.RabbitExceptionTranslator.convertRabbitAccessException(RabbitExceptionTranslator.java:62) ~[spring-rabbit-1.6.11.RELEASE.jar:?]

                            at org.springframework.amqp.rabbit.connection.AbstractConnectionFactory.createBareConnection(AbstractConnectionFactory.java:306) ~[spring-rabbit-1.6.11.RELEASE.jar:?]

                            at org.springframework.amqp.rabbit.connection.CachingConnectionFactory.createConnection(CachingConnectionFactory.java:561) ~[spring-rabbit-1.6.11.RELEASE.jar:?]

                            at org.springframework.amqp.rabbit.core.RabbitTemplate.doExecute(RabbitTemplate.java:1431) ~[spring-rabbit-1.6.11.RELEASE.jar:?]

                            at org.springframework.amqp.rabbit.core.RabbitTemplate.execute(RabbitTemplate.java:1412) ~[spring-rabbit-1.6.11.RELEASE.jar:?]

                            at org.springframework.amqp.rabbit.core.RabbitTemplate.doReceiveNoWait(RabbitTemplate.java:841) ~[spring-rabbit-1.6.11.RELEASE.jar:?]

                            at org.springframework.amqp.rabbit.core.RabbitTemplate.receiveAndConvert(RabbitTemplate.java:931) ~[spring-rabbit-1.6.11.RELEASE.jar:?]

                            at org.springframework.amqp.rabbit.core.RabbitTemplate.receiveAndConvert(RabbitTemplate.java:921) ~[spring-rabbit-1.6.11.RELEASE.jar:?]

                            at org.springframework.integration.amqp.channel.PollableAmqpChannel.doReceive(PollableAmqpChannel.java:210) ~[spring-integration-amqp-4.3.13.RELEASE.jar:4.3.13.RELEASE]

                            at org.springframework.integration.amqp.channel.PollableAmqpChannel.receive(PollableAmqpChannel.java:169) ~[spring-integration-amqp-4.3.13.RELEASE.jar:4.3.13.RELEASE]

                            at com.vmware.vcac.core.event.broker.service.impl.SubscribeServiceImpl.pollEvents(SubscribeServiceImpl.java:139) [classes/:?]

                            at com.vmware.vcac.core.event.broker.controller.SubscriptionController.pollEvents(SubscriptionController.java:263) [classes/:?]

                            at com.vmware.vcac.core.event.broker.controller.SubscriptionController$$FastClassBySpringCGLIB$$c3c3727.invoke(<generated>) [classes/:?]

                            at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204) [spring-core-4.3.13.RELEASE.jar:4.3.13.RELEASE]

                            at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:738) [spring-aop-4.3.13.RELEASE.jar:4.3.13.RELEASE]

                            at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157) [spring-aop-4.3.13.RELEASE.jar:4.3.13.RELEASE]

                            at org.springframework.security.access.intercept.aopalliance.MethodSecurityInterceptor.invoke(MethodSecurityInterceptor.java:69) [spring-security-core-4.2.1.RELEASE.jar:4.2.1.RELEASE]

                            at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) [spring-aop-

                            • 11. Re: Upgrade vRealize Automation 7.3.0 to 7.4 observations
                              r0j Novice

                              opened support ticket 18781199104 on this.

                               

                              The following resolved our issue.

                               

                              Pre-upgrade task

                              https://kb.vmware.com/s/article/2151841

                               

                              Post-upgrade tasks

                              1. Take a backup copy of the  /etc/init.d/vrhb-service

                                 cp /etc/init.d/vrhb-service /etc/init.d/vrhb-service.bak

                              2. Stop the health service

                                 service vrhb-service stop

                              3. Open /etc/init.d/vrhb-service in vi

                                 vi /etc/init.d/vrhb-service

                              4. Find all instances of "-XX:+HeapDumpOnOutOfMemoryError" (without the quotes) and delete (you

                              should find this twice)

                              Note: this entry did not exist pre-upgrade

                              5. Delete your currently configuration

                                 rm -rf /var/lib/vrhb/service-host/sandbox

                                 rm -rf /var/lib/vrhb/vra-test-host/sandbox

                              6. Start the health service

                                 service vrhb-service start

                              7. Recreate your health check configuration

                               

                              Fix vRO workflow logging

                              https://communities.vmware.com/message/2762334#2762334

                              • 12. Re: Upgrade vRealize Automation 7.3.0 to 7.4 observations
                                r0j Novice

                                With all the 7.3-7.4 upgrades and rollbacks on our DEV environment, and working with vmware support, the IaaS pre-requisites (jre 1.8 u1.61, .NET 4.5.2) got missed on one run.

                                 

                                You get the following message on the update status page in the vami, which appears to show the IaaS components upgraded successfully to version 7.3.0.

                                 

                                IaaS upgrade 100% completed

                                 

                                Node img-tun-vra7w3.caf.cloud

                                Database 7.3.0.10750

                                Website 7.3.0.10750

                                ModelManagerData 7.3.0.10750

                                ModelManagerWeb 7.3.0.10750

                                ManagerService 7.3.0.10750

                                ManagementAgent 7.3.0.10750

                                DemOrchestrator 7.3.0.10750

                                DemWorker       7.3.0.10750

                                WAPI     7.3.0.10750

                                vSphereAgent   7.3.0.10750

                                 

                                Upgrade completed successfully.

                                 

                                Upon further inspection of the Management Agent setup install logs, you will find the install failed.

                                 

                                Quickly resolved by installing the pre-requisite .NET and JRE components on the Iaas node(s), and running the upgrade script in /usr/lib/vcac/tools/upgrade/ from the console on the vRA appliance.

                                 

                                A self induced error, worth mentioning...

                                 

                                • 13. Re: Upgrade vRealize Automation 7.3.0 to 7.4 observations
                                  Czernobog Hot Shot

                                  Thanks for posting the observation. I've also tried to upgrade a few weeks ago, with live support from engineering. We spent around 6 hours tripping over various issues during the upgrade process (who's better at testing the product releases than VMware's clients, amirite??) and after finally getting the environment running I have noticed that vIDM won't sync my AD Directory. Turns out that you can no longer control the ldap query sizes from vRA, but instead IDM uses parameters set in the AD, which, when different from default values, won't let you sync large directories. This is (was?) of course not documented, had to roll back too:|

                                  • 14. Re: Upgrade vRealize Automation 7.3.0 to 7.4 observations
                                    gradinka Master
                                    VMware Employees

                                    these are some settings on AD side which are now being honored:

                                    MaxPageSize

                                    MaxValRange

                                     

                                    the MaxValRange being set to zero (MaxValRange = 0) is the one that can cause trouble, as IDM will now take it into account and sync 'zero' users :/