EUC CST Tech Notes - Upgrading VMware Identity Manager On-Premises Virtual Appliances - Updated for 20.01!

EUC CST Tech Notes - Upgrading VMware Identity Manager On-Premises Virtual Appliances - Updated for 20.01!

This document is to assist with upgrading on-premises VMware Identity Manager appliances, including both the full on-premises appliance (commonly known as the Single Virtual Appliance - or SVA) and the on-premises standalone connector for a SaaS VMware Identity Manager tenant.  It should be used with the official VMware Identity Manager upgrade documentation (NOTE: Select the version of the appliance you are upgrading to from the drop down menu).

Please note there are links embedded throughout the document below in reference to the documented procedures or other information.

NOTE:  Document updated for VMware Identity Manager 2.9.x, 3.x, 19.03, and 20.01 as well as references for 2.8.x and older appliances.

Index:

You will find the following info within this document.

Prerequisites:

Prerequisites for an online upgrade

Documentation:

Prerequisites for an offline upgrade​

Documentation:

Post-requisites:​

Documentation:

The following are settings to configure after the upgrade completes successfully. If you have set up a VMware Identity Manager cluster for failover, updating it to three nodes is recommended. This is because of a limitation of Elasticsearch, a search and analytics engine embedded in the VMware Identity Manager appliance. You may continue to use two nodes but you should be aware of a few limitations related to Elasticsearch. See "Configuring Failure and Redundancy" in Installing and Configuring VMware Identity Manager for more information.

  • Enable the new portal user interface.
    • In the administration console, click the arrow on the Catalog tab and select Settings.
    • Select New End User Portal UI in the left pane and click Enable New Portal UI.
  • Transport Layer Security (TLS) protocol 1.0 is disabled by default in VMware Identity Manager 2.8 and higher. TLS 1.1 and 1.2 are supported.
    • External product issues are known to occur when TLS 1.0 is disabled. Updating your other product configurations to use TLS 1.1 or 1.2 is recommended. However, if your version of products such as Horizon, Horizon Air, Citrix, or load balancers have a dependence on TLS 1.0, you can enable TLS 1.0 in VMware Identity Manager by following the instructions in Knowledge Base article 2144805.
  • After upgrading to 2.8.x, 2.9.x, or 3.x, set the thread count in the system config parameter, /SAAS/jersey/manager/api/systemconfigparameter/bulkSyncSharedThreadCount in all nodes and restart the node.

Tips and Best Practices:

Backup of IFCFG-ETH0
Make a copy of the IFCFG-ETH0 file before upgrading.

  1. Login to the appliance console or remote in as SSHUSER and then SU to root.
  2. Run the following command to make a backup of IFCFG-ETH0.
    cp /etc/sysconfig/networking/devices/ifcfg-eth0 /etc/sysconfig/networking/devices/ifcfg-eth0.bak
  3. Exit the console or SSH session.
    exit

Tips-Backup_ifcfg-eth0.png

VM Snapshot Backup

Make a VM snapshot before upgrading (no memory state needed unless you want one).

  1. Login to vSphere or vCenter.
  2. Browse to the VM in question.
  3. Right click on the VM and select options to create a new snapshot.

Tips-VM_Snapshot.png

Database Backups or Snapshots

When upgrading the full VMware Identity Manager on-premises single virtual appliance (SVA), when taking a VM snapshot of appliance(s), it is also good practice to take a backup of the database or VM snapshot of the database server.

Online Updates and Proxy Settings

When doing an online update, you may need to set an outbound proxy within the appliance in order for it to reach the Internet and download the update packages.  Essentially the appliance must be able to reach vapp-updates.vmware.com on TCP port 80 (HTTP).  The latest instructions for setting a proxy for online upgrade can be found in the VMware Identity Manager Documentation online.

The procedure for setting a proxy is as follows:

  1. Log in to the VMware Identity Manager virtual appliance as the root user.
  2. Enter YaST on the command line to run the YaST utility.
  3. Select Network Services in the left pane, then select Proxy.
  4. Enter the proxy server URLs in the HTTP Proxy URL and HTTPS Proxy URL fields.
  5. Select Finish and exit the YaST utility.
  6. Restart the Tomcat server on the VMware Identity Manager virtual appliance to use the new proxy settings by typing the following command:
    service horizon-workspace restart

Post-Update Apply Hot Patches

Apply any necessary hot patches after upgrading.

The Upgrade Process

The short version of the over-the-air procedure is…

  1. Login to the appliance console and make a backup of IFCFG-ETH0.
  2. Make a backup of /etc/sysconfig/networking/devices/ifcfg-eth0.
    1. Login to the appliance console or remote in as SSHUSER and then SU to root.
    2. Run the following command to make a backup of IFCFG-ETH0.
      cp /etc/sysconfig/networking/devices/ifcfg-eth0 /etc/sysconfig/networking/devices/ifcfg-eth0.bak
    3. Exit the console or SSH session.
      exit
  3. Make a VM snapshot (no memory state needed unless you want one).
    1. Login to vSphere or vCenter.
    2. Browse to the VM in question.
    3. Right click on the VM and select options to create a new snapshot.
  4. If you have a cluster of on-premises VMware Identity Manager full appliances (SVA) version 2.8 or older, you will need to prepare the RabbitMQ server on each. NOTE: Skip this step for VMware Identity Manager On-premises Connectors.
    1. Stop RabbitMQ nodes on each VMware Identity Manager appliance in the cluster.
      1. Type rabbitmqctl stop.
      2. Do this for each RabbiMQ node in the cluster before continuing.
    2. Verify that RabbitMQ is detached from the cluster. Type rabbitmqctl cluster_status.
      rabbitmq health.png
      NOTE: In the above image, we see the running_nodes lists only node 1, so this means nodes 2 and 3 are not running RabbitMQ (yet).

    • If using a cluster AND upgrading to 2.9.1 or newer, note that RabbitMQ is disabled in 2.9.1 and higher. See the Upgrading a Cluster procedure.

  5. Perform an Online Upgrade:
    1. Log in to the VMware Identity Manager virtual appliance as the root user.
    2. Run the following updatemgr.hzn command.
      /usr/local/horizon/update/updatemgr.hzn updateinstaller
    3. Run the following command to check that on online upgrade exists.
      /usr/local/horizon/update/updatemgr.hzn check
    4. Run the following command to update the appliance.
      /usr/local/horizon/update/updatemgr.hzn update
      1. Messages that occur during the upgrade are saved to the update.log file at /opt/vmware/var/log/update.log.
    5. Run the updatemgr.hzn check command again to verify that a newer update does not exist.
      /usr/local/horizon/update/updatemgr.hzn check
    6. Check the version of the upgraded appliance.
      vamicli version --appliance
      1. The new version is displayed.
    7. Check that IFCFG-ETH0 is present and properly configured.
      1. If not, copy or move the backup of the file to the original or recreate the original using VI editor with the contents of the backup. When done in VI, save the changes.
        mv /etc/sysconfig/networking/devices/ifcfg-eth0.bak /etc/sysconfig/networking/devices/ifcfg-eth0
    8. Restart the virtual appliance.
      1. Type reboot
  6. Validate the upgrade works properly after the reboot.
  7. If you have a cluster of on-premises VMware Identity Manager 2.8.1 and older nodes, validate the RabbitMQ cluster status is good​.
    1. As each node is upgraded, run the rabbitmgctl cluster_status command on the upgraded node to verify that all the nodes upgraded so far are listed in the running_nodes section of the output. After upgrading node 1, the running_nodes section lists only node1. After upgrading node 2, run the rabbitmqctl cluster_status command on both nodes and the running_nodes section should each list node1 and node2. This indicates that the RabbitMQ nodes are clustered together correctly.
    2. When all nodes are upgraded, RabbitMQ forms a cluster with the 2.8.x (or older) nodes in the correct order.
  8. If you have a cluster of on-premises VMware Identity Manager 2.9.1 and newer nodes, the Upgrading a Cluster procedure to ensure RabbitMQ is disabled.
  9. Shut down each appliance within the cluster and boot each one up one at a time - waiting until the app server is fully started before booting the next.  After this is done, you should see all nodes in the cluster properly sync and go green.
  10. After upgrading 2.8.x or 2.9.x, set the thread count in the system config parameter, /SAAS/jersey/manager/api/systemconfigparameter/bulkSyncSharedThreadCount in all nodes and restart each node.
    1. See the online documentation for how to adjust this parameter correctly.
  11. Apply any post-upgrade patches if you have any to apply in accordance with their instructions.
  12. Delete the VM snapshot after a couple of days or if you are certain the upgrade process was successful.

Upgrade Issues and Troubleshooting Options:

Troubleshooting Upgrade Errors

UpdateInstaller Fails to Run on VMware Identity Manager On-Premises Full Appliance

Symptoms:
Update, Update Check, Update Installer fail to run and/or show any outcome.

Cause:

Not enough available inodes (number of allowed files).

Correction:

  1. Run df. This will give you an idea of how much space you have.
  2. Run df -i.  This will tell you how much inode space you have left.
    1. Example: If /var says 100% and you see inodes has less than 200 then you may have this issue.
      NOTE: An upgrade from VMware Identity Manager 2.6 to 2.7 typically requires around 209 inodes.
  3. The solution is to clear off enough files to free up enough inode space.
    NOTE: This can be done in any folder where there are enough files to make a difference.
    1. The recommendation is to use the /var/log folder as this is commonly the offender for inode consumption.
  4. To clear off bz2 log files more than 90 days old, do the following:
    1. Login to the appliance console directly (via vCenter) or via SSH (and SU to root).
    2. Browse to the /var/log folder
      cd /var/log
      1. Find and delete bz2 files older than 90 days.  The below command deletes anything ending with the extension “bz2” which is older than 90 days from current date. You may need to delete more recent files to clear up enough inodes by changing the time frame to a lower range such as 45 days instead of 90 days.
        find ./*.bz2 -mtime +90 -exec rm {} \;
        1. Retest update installer, update check, and update.
          /usr/local/horizon/update/updatemgr.hzn updateinstaller
          /usr/local/horizon/update/updatemgr.hzn check
          /usr/local/horizon/update/updatemgr.hzn update

        Reference:

        Information courtesy of Wibowo Leksono

        UpdateInstaller Reports No Updates on VMware Identity Manager On-Premises Connector Appliance Version 2016.11.1.0


        Symptoms:

        Update Installer runs on VMware Identity Manager On-Premises Connector Version 2016.11.1.0 but reports no updates are available.

        Cause:

        Known Issue with VMware Identity Manager Connector version 2016.11.1.0.

        Correction:

        1. Apply VMware KB 2149179​

        Reference:

        https://kb.vmware.com/kb/2149179

        No Networking Detected on Reboot After Upgrade

        Link:  Networking Error after Upgrade

        Symptoms:

        The upgraded SUSE appliance shows NO NETWORKING DETECTED errors.  Logging in and looking at /etc/sysconfig/network (ls /etc/sysconfig/network -l -a) shows source file for ifcfg-eth0 is missing in /etc/sysconfig/networking/devices/ for /etc/sysconfig/network/ifcfg-eth0 link file.

        No-Networking-1.png

        No-Networking-2.png

        Cause:

        Known Issue with upgrade process wiping ifcfg-eth0 information

        Correction:

        1. Move the backup of ifcfg-eth0 taken during step 2.2 of The Upgrade Process back into place.
          1. Login to the appliance console or remote in as SSHUSER and then SU to root.
          2. Run the following command to move the backup of IFCFG-ETH0 into place
            mv /etc/sysconfig/networking/devices/ifcfg-eth0.bak /etc/sysconfig/networking/devices/ifcfg-eth0
            1. Exit the console or SSH session.
              exit
            2. Alternately (instead of step 1.), manually create the file.
              1. Login to the appliance console or remote in as SSHUSER and then SU to root.
              2. Open the VI editor and create the file by typing the following command.
                vi /etc/sysconfic/network/ifcfg-eth0
                1. Type in all of the settings as shown in the below reference image. All lines except IPADDR=, NETMASK=, and BROADCAST= will be identical to what is in the image.  To find your IP address, either ping or attempt to resolve the FQDN of the appliance to get the IP address, or look within the vApp advanced options of the VM to get the exact IP address.  From there you should be able to fill in the below values.
                  IPADDR=‘<your_appliance_ip_address>’
                  NETMASK=‘<your_appliance_ip_subnet_address>’
                  BROADCAST=‘<your_appliance_ip_broadcast_address>’
                  1. Exit the INSERT mode by pressing the ESC key.
                  2. Save the file and exit the VI editor by typing `:` (colon) then `x` (lower case "x") and then pressing the ENTER key.
                2. Reboot the appliance by typing reboot and pressing ENTER.

                Reference Image:

                No-Networking-Ref.png

                Version history
                Revision #:
                1 of 1
                Last update:
                ‎04-18-2017 02:27 PM
                Updated by:
                VMware Employee
                 
                View Article History