7 Replies Latest reply on Jul 25, 2019 7:42 AM by aenagy

    VRA 7.6 VAMI out of order after upgrade from 7.4

    _mduchaine_ Novice

      Hi everyone,

      I recently upgraded my 7.4 lab environment to 7.6.

      Upgrade went without a hitch - or so it seems - and I've been able to use the environment normally.

      The only problem I got is with the VAMI - it's become totally unresponsive. The only thing I get out of it is the list of services and their state.

      Anything else the interface just sits there either waiting or reporting nothing.

      Summary shows nothing except the error

      License keeps waiting,

      Doesn't show any status for VCO, yet it's running. I checked.

      VAMI logs don't report anything useful. vami-ovf.log is the only one that regularly updates and keeps reporting PIDs processed externally.

       

      Has anyone faced this issue before, maybe point me in the right direction to possible causes?

      Thanks in advance

        • 1. Re: VRA 7.6 VAMI out of order after upgrade from 7.4
          daphnissov Guru
          vExpertCommunity Warriors

          Clear cache and cookies. Verify in a private/incognito window to rule out browser-related things.

          • 2. Re: VRA 7.6 VAMI out of order after upgrade from 7.4
            _mduchaine_ Novice

            Whish it would have been that simple ...

            Tried clearing browser cache and icognito, tried with chrome, ie and edge. Installed firefox ...

            All same behavior./result.

            • 4. Re: VRA 7.6 VAMI out of order after upgrade from 7.4
              nboshnakov Novice
              VMware Employees

              Can you show the error messages that you get in /var/log/messages when you refresh on of the tabs?

              Also, do you use full fqdn to access VAMI or a shortname ? Can you try using IP address?

              • 5. Re: VRA 7.6 VAMI out of order after upgrade from 7.4
                _mduchaine_ Novice

                Hi nboshnakov,

                Nice catch! Using the IP instead of the FQDN, all tabs of the VAMI seem to work fine.

                Also just the hostname "vra-01a" without domain works.

                Issue is apparently only if use FQDN vra-01a.corp.local

                Looked at the log files /var/log/messages and I see this very suspect line:

                vra-01a vami /opt/vmware/share/htdocs/service/cafe/config.py[6594]: info Host of referer: vra-01a.corp.local not in ['192.168.110.78', 'vra-01a', 'vra-01a']

                Are we looking at a certificate issue perhaps?

                 

                I've included an extract from the /var/log/messages, from the moment I logged-in.

                ip of the vra appliance is 192.168.110.78, iaas server is .75, client from which I'm connecting is 192.168.110.10

                 

                ..........................................

                 

                2019-07-17T21:48:57.861757+00:00 vra-01a vami /opt/vmware/share/htdocs/service/cafe/summary-page.py[6593]: info Executing shell command...

                2019-07-17T21:48:58.128341+00:00 vra-01a vami /opt/vmware/share/htdocs/service/cafe/summary-page.py[6593]: info Host of referer: vra-01a.corp.local not in ['192.168.110.78', 'vra-01a', 'vra-01a']

                2019-07-17T21:48:59.077613+00:00 vra-01a vami /opt/vmware/share/htdocs/service/cafe/config.py[6594]: info Executing shell command...

                2019-07-17T21:48:59.776085+00:00 vra-01a vami /opt/vmware/share/htdocs/service/cafe/config.py[6594]: info Host of referer: vra-01a.corp.local not in ['192.168.110.78', 'vra-01a', 'vra-01a']

                2019-07-17T21:49:00.465538+00:00 vra-01a vami /opt/vmware/share/htdocs/service/cafe/summary-page.py[6625]: info Executing shell command...

                2019-07-17T21:49:00.677052+00:00 vra-01a vami /opt/vmware/share/htdocs/service/cafe/summary-page.py[6622]: info Executing shell command...

                2019-07-17T21:49:00.773158+00:00 vra-01a vami /opt/vmware/share/htdocs/service/cafe/summary-page.py[6626]: info Executing shell command...

                2019-07-17T21:49:00.808813+00:00 vra-01a vami /opt/vmware/share/htdocs/service/cafe/summary-page.py[6623]: info Executing shell command...

                2019-07-17T21:49:01.304056+00:00 vra-01a vami /opt/vmware/share/htdocs/service/cafe/summary-page.py[6625]: info Host of referer: vra-01a.corp.local not in ['192.168.110.78', 'vra-01a', 'vra-01a']

                2019-07-17T21:49:01.304141+00:00 vra-01a vami /opt/vmware/share/htdocs/service/cafe/summary-page.py[6626]: info Host of referer: vra-01a.corp.local not in ['192.168.110.78', 'vra-01a', 'vra-01a']

                2019-07-17T21:49:01.343941+00:00 vra-01a vami /opt/vmware/share/htdocs/service/cafe/summary-page.py[6624]: info Executing shell command...

                2019-07-17T21:49:01.424507+00:00 vra-01a vami /opt/vmware/share/htdocs/service/cafe/summary-page.py[6622]: info Host of referer: vra-01a.corp.local not in ['192.168.110.78', 'vra-01a', 'vra-01a']

                2019-07-17T21:49:01.752683+00:00 vra-01a vami /opt/vmware/share/htdocs/service/cafe/summary-page.py[6623]: info Host of referer: vra-01a.corp.local not in ['192.168.110.78', 'vra-01a', 'vra-01a']

                2019-07-17T21:49:01.866909+00:00 vra-01a /usr/sbin/cron[6772]: (root) CMD (/usr/bin/flock -n -o /tmp/.cluster-event -c "find /tmp -maxdepth 1 -regex '/tmp/cluster-event-[0-9]*-.+' | sort -n | head -n 1 | xargs -r -I {} bash -c '{} 2>&1 | logger -t {}; rm {}'")

                2019-07-17T21:49:01.888576+00:00 vra-01a /usr/sbin/cron[6773]: (root) CMD (/usr/bin/python /usr/lib/vcac/tools/vami/bin/ma-heartbeat.py > /dev/null 2>&1)

                2019-07-17T21:49:02.089703+00:00 vra-01a vami /opt/vmware/share/htdocs/service/cafe/summary-page.py[6624]: info Host of referer: vra-01a.corp.local not in ['192.168.110.78', 'vra-01a', 'vra-01a']

                2019-07-17T21:49:02.412152+00:00 vra-01a vami /opt/vmware/share/htdocs/service/cafe/summary-page.py[6643]: info Executing shell command...

                2019-07-17T21:49:03.196090+00:00 vra-01a vami /opt/vmware/share/htdocs/service/cafe/summary-page.py[6643]: info Host of referer: vra-01a.corp.local not in ['192.168.110.78', 'vra-01a', 'vra-01a']

                2019-07-17T21:49:03.733788+00:00 vra-01a vami /opt/vmware/share/htdocs/service/cafe/summary-page.py[6741]: info Executing shell command...

                2019-07-17T21:49:03.997824+00:00 vra-01a vami /opt/vmware/share/htdocs/service/cafe/config.py[6644]: info Processing request GET /config/nodes/1C124FE1-0FC3-44A0-8C1B-48344008B80F/commands/next-command, referer: None

                2019-07-17T21:49:04.195447+00:00 vra-01a vami /opt/vmware/share/htdocs/service/cafe/summary-page.py[6747]: info Executing shell command...

                2019-07-17T21:49:04.362323+00:00 vra-01a vami /opt/vmware/share/htdocs/service/cafe/config.py[6644]: info Legacy authentication token received from ::ffff:192.168.110.75

                2019-07-17T21:49:04.382926+00:00 vra-01a [vra-command-agent][3995]: [015123] "GET http://localhost/api/node/1C124FE1-0FC3-44A0-8C1B-48344008B80F/command/processable HTTP/1.1" from @ - 200 15B in 5.885009ms

                2019-07-17T21:49:04.385814+00:00 vra-01a vami /opt/vmware/share/htdocs/service/cafe/config.py[6644]: info No command for node 1C124FE1-0FC3-44A0-8C1B-48344008B80F

                2019-07-17T21:49:04.543330+00:00 vra-01a vami /opt/vmware/share/htdocs/service/cafe/summary-page.py[6781]: info Executing shell command...

                2019-07-17T21:49:04.552485+00:00 vra-01a vami /opt/vmware/share/htdocs/service/cafe/summary-page.py[6741]: info Host of referer: vra-01a.corp.local not in ['192.168.110.78', 'vra-01a', 'vra-01a']

                2019-07-17T21:49:04.606173+00:00 vra-01a vami /opt/vmware/share/htdocs/service/cafe/summary-page.py[6761]: info Executing shell command...

                2019-07-17T21:49:04.692539+00:00 vra-01a vami /opt/vmware/share/htdocs/service/cafe/config.py[6648]: info Processing request PUT /config/nodes/1C124FE1-0FC3-44A0-8C1B-48344008B80F/ping, referer: None

                2019-07-17T21:49:04.880076+00:00 vra-01a vami /opt/vmware/share/htdocs/service/cafe/config.py[6648]: info Legacy authentication token received from ::ffff:192.168.110.75

                2019-07-17T21:49:05.029646+00:00 vra-01a vami /opt/vmware/share/htdocs/service/cafe/summary-page.py[6785]: info Executing shell command...

                2019-07-17T21:49:05.040260+00:00 vra-01a vami /opt/vmware/share/htdocs/service/cafe/summary-page.py[6747]: info Host of referer: vra-01a.corp.local not in ['192.168.110.78', 'vra-01a', 'vra-01a']

                2019-07-17T21:49:05.472072+00:00 vra-01a vami /opt/vmware/share/htdocs/service/cafe/summary-page.py[6761]: info Host of referer: vra-01a.corp.local not in ['192.168.110.78', 'vra-01a', 'vra-01a']

                2019-07-17T21:49:05.620162+00:00 vra-01a vami /opt/vmware/share/htdocs/service/cafe/summary-page.py[6781]: info Host of referer: vra-01a.corp.local not in ['192.168.110.78', 'vra-01a', 'vra-01a']

                2019-07-17T21:49:05.751628+00:00 vra-01a vami /opt/vmware/share/htdocs/service/cafe/summary-page.py[6811]: info Executing shell command...

                2019-07-17T21:49:05.780077+00:00 vra-01a vami /opt/vmware/share/htdocs/service/cafe/summary-page.py[6785]: info Host of referer: vra-01a.corp.local not in ['192.168.110.78', 'vra-01a', 'vra-01a']

                2019-07-17T21:49:06.460160+00:00 vra-01a vami /opt/vmware/share/htdocs/service/cafe/summary-page.py[6811]: info Host of referer: vra-01a.corp.local not in ['192.168.110.78', 'vra-01a', 'vra-01a']

                2019-07-17T21:49:06.923197+00:00 vra-01a vami /opt/vmware/share/htdocs/service/cafe/summary-page.py[6906]: info Executing shell command...

                2019-07-17T21:49:07.008367+00:00 vra-01a vami /opt/vmware/share/htdocs/service/cafe/summary-page.py[6952]: info Executing shell command...

                2019-07-17T21:49:07.364771+00:00 vra-01a vami /opt/vmware/share/htdocs/service/cafe/summary-page.py[6974]: info Executing shell command...

                • 6. Re: VRA 7.6 VAMI out of order after upgrade from 7.4
                  _mduchaine_ Novice

                  Some additional info:

                   

                  [master] vra-01a:/etc # vra-command list-nodes

                  Node:

                    NodeHost: vra-01a

                    NodeId: cafe.node.719751461.21656

                    NodeType: VA

                  Node:

                    NodeHost: iaas-01a.corp.local

                    NodeId: 1C124FE1-0FC3-44A0-8C1B-48344008B80F

                    NodeType: IAAS

                  [master] vra-01a:/etc #

                   

                  and /etc/hosts:

                   

                  [master] vra-01a:/etc # cat /etc/hosts

                  # VAMI_EDIT_BEGIN

                  # Generated by Studio VAMI service. Do not modify manually.

                  127.0.0.1   vra-01a localhost

                  ::1   vra-01a localhost ipv6-localhost ipv6-loopback

                  # VAMI_EDIT_END

                  127.0.0.1 vra-01a.corp.local load-balancer-host

                   

                  not sure why the last line was moved outside the VAMI_EDIT section, it used to be there with VRA 7.4

                   

                  etc/hosts.orig (taken  priior to the upgrade):

                   

                  # VAMI_EDIT_BEGIN

                  # Generated by Studio VAMI service. Do not modify manually.

                  127.0.0.1   vra-01a localhost

                  ::1   vra-01a localhost ipv6-localhost ipv6-loopback

                  127.0.0.1 vra-01a.corp.local load-balancer-host

                  # VAMI_EDIT_END

                   

                  Thx for your help,

                  Mike

                   

                  [master] vra-01a:/etc # cat /etc/hosts## hosts         This file describes a number of hostname-to-address#               mappings for the TCP/IP subsystem.  It is mostly#               used at boot time, when no name servers are running.#               On small systems, this file can be used instead of a#               "named" name server.# Syntax:## IP-Address  Full-Qualified-Hostname  Short-Hostname#

                  # special IPv6 addresses
                  fe00::0         ipv6-localnet
                  ff00::0         ipv6-mcastprefixff02::2         ipv6-allroutersff02::3         ipv6-allhosts# VAMI_EDIT_BEGIN# Generated by Studio VAMI service. Do not modify manually.127.0.0.1   vra-01a localhost::1   vra-01a localhost ipv6-localhost ipv6-loopback# VAMI_EDIT_END

                  127.0.0.1 vra-01a.corp.local load-balancer-host

                  • 7. Re: VRA 7.6 VAMI out of order after upgrade from 7.4
                    aenagy Hot Shot

                    _mduchaine_:

                     

                    Something I learned from GSS while trying to upgrade one of my client's vRA instances from 7.3 to 7.6 is that if you are experiencing similar symptoms (VAMI pages not rendering/populating especially the Services tab or services not starting) is to check vIDM/Horizon (/var/log/vmware/horizon/workspace.log, /var/log/vmware/horizon/connector.log) and RabbitMQ (Messaging tab) as these are among the first services started on the Cafe node(s).

                     

                    In the case of vIDM/Horizon I found that the upgrade process seemed to consistently break connections to the Active Directory domain controllers. It is possible that there were problems with the Active Directory LDAP integration before the upgrade. We did find that "IdP Hostname" in "Identity Providers" was set to one of our Cafe appliances instead of the NLB -- easy to fix. In any case fixing the broken connectors is a non-trivial task as it required GSS several hours to remove the problem Cafe appliance and re-add it. Having repeated this upgrade several times I found that sometimes the local vRA groups would lose their membership information which mean't logging in with a local account. In these cases the best thing to do is roll-back the environment and retry the upgrade.

                     

                    BTW, some of the tabs and such have been moved around in vRA 7.6 but that doesn't explain the blank pages you posted.

                     

                    Another trick I learned from GSS is how to re-use the PEM files containing the certificate chain for the web console and apply them to the VAMI. This trick only makes sense if you are using trusted and signed certificates, and that your browser trusts the certificate chain.

                     

                    cp /opt/vmware/etc/lighttpd/server.pem /opt/vmware/etc/lighttpd/server.pem.backup
                    cp /etc/apache2/server.pem /opt/vmware/etc/lighttpd/server.pem
                    service vami-lighttp restart
                    service haproxy restart
                    

                     

                    This is modified from:

                     

                    Replace the vRealize Automation Appliance Management Site Certificate

                    https://docs.vmware.com/en/vRealize-Automation/7.6/com.vmware.vra.prepare.use.doc/GUID-E6515C12-61DA-4F67-8F44-C740B778743B.html 

                     

                    Good luck.