VMware Cloud Community
_mduchaine_
Enthusiast
Enthusiast

VRA 7.6 VAMI out of order after upgrade from 7.4

Hi everyone,

I recently upgraded my 7.4 lab environment to 7.6.

Upgrade went without a hitch - or so it seems - and I've been able to use the environment normally.

The only problem I got is with the VAMI - it's become totally unresponsive. The only thing I get out of it is the list of services and their state.

Anything else the interface just sits there either waiting or reporting nothing.

pastedImage_1.png

Summary shows nothing except the error

pastedImage_2.png

License keeps waiting,

pastedImage_3.png

Doesn't show any status for VCO, yet it's running. I checked.

VAMI logs don't report anything useful. vami-ovf.log is the only one that regularly updates and keeps reporting PIDs processed externally.

pastedImage_4.png

Has anyone faced this issue before, maybe point me in the right direction to possible causes?

Thanks in advance

Tags (3)
7 Replies
daphnissov
Immortal
Immortal

Clear cache and cookies. Verify in a private/incognito window to rule out browser-related things.

Reply
0 Kudos
_mduchaine_
Enthusiast
Enthusiast

Whish it would have been that simple ...

Tried clearing browser cache and icognito, tried with chrome, ie and edge. Installed firefox ...

All same behavior./result.:smileyconfused:

Reply
0 Kudos
daphnissov
Immortal
Immortal

Reboot the appliance?

Reply
0 Kudos
nboshnakov
VMware Employee
VMware Employee

Can you show the error messages that you get in /var/log/messages when you refresh on of the tabs?

Also, do you use full fqdn to access VAMI or a shortname ? Can you try using IP address?

_mduchaine_
Enthusiast
Enthusiast

Hi nboshnakov,

Nice catch! Using the IP instead of the FQDN, all tabs of the VAMI seem to work fine.

Also just the hostname "vra-01a" without domain works.

Issue is apparently only if use FQDN vra-01a.corp.local

Looked at the log files /var/log/messages and I see this very suspect line:

vra-01a vami /opt/vmware/share/htdocs/service/cafe/config.py[6594]: info Host of referer: vra-01a.corp.local not in ['192.168.110.78', 'vra-01a', 'vra-01a']

Are we looking at a certificate issue perhaps?

I've included an extract from the /var/log/messages, from the moment I logged-in.

ip of the vra appliance is 192.168.110.78, iaas server is .75, client from which I'm connecting is 192.168.110.10

..........................................

2019-07-17T21:48:57.861757+00:00 vra-01a vami /opt/vmware/share/htdocs/service/cafe/summary-page.py[6593]: info Executing shell command...

2019-07-17T21:48:58.128341+00:00 vra-01a vami /opt/vmware/share/htdocs/service/cafe/summary-page.py[6593]: info Host of referer: vra-01a.corp.local not in ['192.168.110.78', 'vra-01a', 'vra-01a']

2019-07-17T21:48:59.077613+00:00 vra-01a vami /opt/vmware/share/htdocs/service/cafe/config.py[6594]: info Executing shell command...

2019-07-17T21:48:59.776085+00:00 vra-01a vami /opt/vmware/share/htdocs/service/cafe/config.py[6594]: info Host of referer: vra-01a.corp.local not in ['192.168.110.78', 'vra-01a', 'vra-01a']

2019-07-17T21:49:00.465538+00:00 vra-01a vami /opt/vmware/share/htdocs/service/cafe/summary-page.py[6625]: info Executing shell command...

2019-07-17T21:49:00.677052+00:00 vra-01a vami /opt/vmware/share/htdocs/service/cafe/summary-page.py[6622]: info Executing shell command...

2019-07-17T21:49:00.773158+00:00 vra-01a vami /opt/vmware/share/htdocs/service/cafe/summary-page.py[6626]: info Executing shell command...

2019-07-17T21:49:00.808813+00:00 vra-01a vami /opt/vmware/share/htdocs/service/cafe/summary-page.py[6623]: info Executing shell command...

2019-07-17T21:49:01.304056+00:00 vra-01a vami /opt/vmware/share/htdocs/service/cafe/summary-page.py[6625]: info Host of referer: vra-01a.corp.local not in ['192.168.110.78', 'vra-01a', 'vra-01a']

2019-07-17T21:49:01.304141+00:00 vra-01a vami /opt/vmware/share/htdocs/service/cafe/summary-page.py[6626]: info Host of referer: vra-01a.corp.local not in ['192.168.110.78', 'vra-01a', 'vra-01a']

2019-07-17T21:49:01.343941+00:00 vra-01a vami /opt/vmware/share/htdocs/service/cafe/summary-page.py[6624]: info Executing shell command...

2019-07-17T21:49:01.424507+00:00 vra-01a vami /opt/vmware/share/htdocs/service/cafe/summary-page.py[6622]: info Host of referer: vra-01a.corp.local not in ['192.168.110.78', 'vra-01a', 'vra-01a']

2019-07-17T21:49:01.752683+00:00 vra-01a vami /opt/vmware/share/htdocs/service/cafe/summary-page.py[6623]: info Host of referer: vra-01a.corp.local not in ['192.168.110.78', 'vra-01a', 'vra-01a']

2019-07-17T21:49:01.866909+00:00 vra-01a /usr/sbin/cron[6772]: (root) CMD (/usr/bin/flock -n -o /tmp/.cluster-event -c "find /tmp -maxdepth 1 -regex '/tmp/cluster-event-[0-9]*-.+' | sort -n | head -n 1 | xargs -r -I {} bash -c '{} 2>&1 | logger -t {}; rm {}'")

2019-07-17T21:49:01.888576+00:00 vra-01a /usr/sbin/cron[6773]: (root) CMD (/usr/bin/python /usr/lib/vcac/tools/vami/bin/ma-heartbeat.py > /dev/null 2>&1)

2019-07-17T21:49:02.089703+00:00 vra-01a vami /opt/vmware/share/htdocs/service/cafe/summary-page.py[6624]: info Host of referer: vra-01a.corp.local not in ['192.168.110.78', 'vra-01a', 'vra-01a']

2019-07-17T21:49:02.412152+00:00 vra-01a vami /opt/vmware/share/htdocs/service/cafe/summary-page.py[6643]: info Executing shell command...

2019-07-17T21:49:03.196090+00:00 vra-01a vami /opt/vmware/share/htdocs/service/cafe/summary-page.py[6643]: info Host of referer: vra-01a.corp.local not in ['192.168.110.78', 'vra-01a', 'vra-01a']

2019-07-17T21:49:03.733788+00:00 vra-01a vami /opt/vmware/share/htdocs/service/cafe/summary-page.py[6741]: info Executing shell command...

2019-07-17T21:49:03.997824+00:00 vra-01a vami /opt/vmware/share/htdocs/service/cafe/config.py[6644]: info Processing request GET /config/nodes/1C124FE1-0FC3-44A0-8C1B-48344008B80F/commands/next-command, referer: None

2019-07-17T21:49:04.195447+00:00 vra-01a vami /opt/vmware/share/htdocs/service/cafe/summary-page.py[6747]: info Executing shell command...

2019-07-17T21:49:04.362323+00:00 vra-01a vami /opt/vmware/share/htdocs/service/cafe/config.py[6644]: info Legacy authentication token received from ::ffff:192.168.110.75

2019-07-17T21:49:04.382926+00:00 vra-01a [vra-command-agent][3995]: [015123] "GET http://localhost/api/node/1C124FE1-0FC3-44A0-8C1B-48344008B80F/command/processable HTTP/1.1" from @ - 200 15B in 5.885009ms

2019-07-17T21:49:04.385814+00:00 vra-01a vami /opt/vmware/share/htdocs/service/cafe/config.py[6644]: info No command for node 1C124FE1-0FC3-44A0-8C1B-48344008B80F

2019-07-17T21:49:04.543330+00:00 vra-01a vami /opt/vmware/share/htdocs/service/cafe/summary-page.py[6781]: info Executing shell command...

2019-07-17T21:49:04.552485+00:00 vra-01a vami /opt/vmware/share/htdocs/service/cafe/summary-page.py[6741]: info Host of referer: vra-01a.corp.local not in ['192.168.110.78', 'vra-01a', 'vra-01a']

2019-07-17T21:49:04.606173+00:00 vra-01a vami /opt/vmware/share/htdocs/service/cafe/summary-page.py[6761]: info Executing shell command...

2019-07-17T21:49:04.692539+00:00 vra-01a vami /opt/vmware/share/htdocs/service/cafe/config.py[6648]: info Processing request PUT /config/nodes/1C124FE1-0FC3-44A0-8C1B-48344008B80F/ping, referer: None

2019-07-17T21:49:04.880076+00:00 vra-01a vami /opt/vmware/share/htdocs/service/cafe/config.py[6648]: info Legacy authentication token received from ::ffff:192.168.110.75

2019-07-17T21:49:05.029646+00:00 vra-01a vami /opt/vmware/share/htdocs/service/cafe/summary-page.py[6785]: info Executing shell command...

2019-07-17T21:49:05.040260+00:00 vra-01a vami /opt/vmware/share/htdocs/service/cafe/summary-page.py[6747]: info Host of referer: vra-01a.corp.local not in ['192.168.110.78', 'vra-01a', 'vra-01a']

2019-07-17T21:49:05.472072+00:00 vra-01a vami /opt/vmware/share/htdocs/service/cafe/summary-page.py[6761]: info Host of referer: vra-01a.corp.local not in ['192.168.110.78', 'vra-01a', 'vra-01a']

2019-07-17T21:49:05.620162+00:00 vra-01a vami /opt/vmware/share/htdocs/service/cafe/summary-page.py[6781]: info Host of referer: vra-01a.corp.local not in ['192.168.110.78', 'vra-01a', 'vra-01a']

2019-07-17T21:49:05.751628+00:00 vra-01a vami /opt/vmware/share/htdocs/service/cafe/summary-page.py[6811]: info Executing shell command...

2019-07-17T21:49:05.780077+00:00 vra-01a vami /opt/vmware/share/htdocs/service/cafe/summary-page.py[6785]: info Host of referer: vra-01a.corp.local not in ['192.168.110.78', 'vra-01a', 'vra-01a']

2019-07-17T21:49:06.460160+00:00 vra-01a vami /opt/vmware/share/htdocs/service/cafe/summary-page.py[6811]: info Host of referer: vra-01a.corp.local not in ['192.168.110.78', 'vra-01a', 'vra-01a']

2019-07-17T21:49:06.923197+00:00 vra-01a vami /opt/vmware/share/htdocs/service/cafe/summary-page.py[6906]: info Executing shell command...

2019-07-17T21:49:07.008367+00:00 vra-01a vami /opt/vmware/share/htdocs/service/cafe/summary-page.py[6952]: info Executing shell command...

2019-07-17T21:49:07.364771+00:00 vra-01a vami /opt/vmware/share/htdocs/service/cafe/summary-page.py[6974]: info Executing shell command...

Reply
0 Kudos
_mduchaine_
Enthusiast
Enthusiast

Some additional info:

[master] vra-01a:/etc # vra-command list-nodes

Node:

  NodeHost: vra-01a

  NodeId: cafe.node.719751461.21656

  NodeType: VA

Node:

  NodeHost: iaas-01a.corp.local

  NodeId: 1C124FE1-0FC3-44A0-8C1B-48344008B80F

  NodeType: IAAS

[master] vra-01a:/etc #

and /etc/hosts:

[master] vra-01a:/etc # cat /etc/hosts

# VAMI_EDIT_BEGIN

# Generated by Studio VAMI service. Do not modify manually.

127.0.0.1   vra-01a localhost

::1   vra-01a localhost ipv6-localhost ipv6-loopback

# VAMI_EDIT_END

127.0.0.1 vra-01a.corp.local load-balancer-host

not sure why the last line was moved outside the VAMI_EDIT section, it used to be there with VRA 7.4

etc/hosts.orig (taken  priior to the upgrade):

# VAMI_EDIT_BEGIN

# Generated by Studio VAMI service. Do not modify manually.

127.0.0.1   vra-01a localhost

::1   vra-01a localhost ipv6-localhost ipv6-loopback

127.0.0.1 vra-01a.corp.local load-balancer-host

# VAMI_EDIT_END

Thx for your help,

Mike

[master] vra-01a:/etc # cat /etc/hosts## hosts         This file describes a number of hostname-to-address#               mappings for the TCP/IP subsystem.  It is mostly#               used at boot time, when no name servers are running.#               On small systems, this file can be used instead of a#               "named" name server.# Syntax:## IP-Address  Full-Qualified-Hostname  Short-Hostname#

# special IPv6 addresses
fe00::0         ipv6-localnet
ff00::0         ipv6-mcastprefixff02::2         ipv6-allroutersff02::3         ipv6-allhosts# VAMI_EDIT_BEGIN# Generated by Studio VAMI service. Do not modify manually.127.0.0.1   vra-01a localhost::1   vra-01a localhost ipv6-localhost ipv6-loopback# VAMI_EDIT_END

127.0.0.1 vra-01a.corp.local load-balancer-host

Reply
0 Kudos
aenagy
Hot Shot
Hot Shot

_mduchaine_​:

Something I learned from GSS while trying to upgrade one of my client's vRA instances from 7.3 to 7.6 is that if you are experiencing similar symptoms (VAMI pages not rendering/populating especially the Services tab or services not starting) is to check vIDM/Horizon (/var/log/vmware/horizon/workspace.log, /var/log/vmware/horizon/connector.log) and RabbitMQ (Messaging tab) as these are among the first services started on the Cafe node(s).

In the case of vIDM/Horizon I found that the upgrade process seemed to consistently break connections to the Active Directory domain controllers. It is possible that there were problems with the Active Directory LDAP integration before the upgrade. We did find that "IdP Hostname" in "Identity Providers" was set to one of our Cafe appliances instead of the NLB -- easy to fix. In any case fixing the broken connectors is a non-trivial task as it required GSS several hours to remove the problem Cafe appliance and re-add it. Having repeated this upgrade several times I found that sometimes the local vRA groups would lose their membership information which mean't logging in with a local account. In these cases the best thing to do is roll-back the environment and retry the upgrade.

BTW, some of the tabs and such have been moved around in vRA 7.6 but that doesn't explain the blank pages you posted.

Another trick I learned from GSS is how to re-use the PEM files containing the certificate chain for the web console and apply them to the VAMI. This trick only makes sense if you are using trusted and signed certificates, and that your browser trusts the certificate chain.

cp /opt/vmware/etc/lighttpd/server.pem /opt/vmware/etc/lighttpd/server.pem.backup

cp /etc/apache2/server.pem /opt/vmware/etc/lighttpd/server.pem

service vami-lighttp restart

service haproxy restart

This is modified from:

Replace the vRealize Automation Appliance Management Site Certificate

https://docs.vmware.com/en/vRealize-Automation/7.6/com.vmware.vra.prepare.use.doc/GUID-E6515C12-61DA... 

Good luck.

Reply
0 Kudos