NSX Edge Load Balancer Configuration Rollback without Full Restore

[cnrz]

Hello,

NSX Manager configuration backup and restore includes Edge Configurations, restore is possible through Backup and Restore of NSX Manager. Since this restore requires full reinstallation of all NSX Manager components, could there be an easier option for restoring the configuration of a specific Edge Load Balancer Configuration to a last known good configuration buy rolling back this edge to some number of configuration changes or time based 24 hours before?

dFW and dVS configurations can be specifically backed up and restored, could not find similar scenario for Edge.

http://darrylcauldwell.com/nsx-backup-restore/

VRO seems to have rollback functionality, is it usable for similar scenario and does it require all configuration change to be done through VRO, i.e. could the configuration be rolled back if done through GUI or Rest API?

https://docs.vmware.com/en/vRealize-Automation/7.3/vrealize_orchestrator_coding_design_guide.pdf

Master workflow design must encompass and deliver on the full lifecycle considerations, including use cases for provisioning, reconfiguring, and retiring resources in the VMware ecosystem. Additionally, the master workflow should be designed for availability with validation performed between each step to enable pause, rollback, or failure functionality.

Thanks,

https://docs.vmware.com/en/VMware-NSX-for-vSphere/6.3/com.vmware.nsx.upgrade.doc/GUID-72EFCAB1-0B10-...

At a minimum, we recommend taking regular backups of NSX Manager and vCenter. Your backup frequency and schedule might vary based on your business needs and operational procedures. We recommend taking NSX backups frequently during times of frequent configuration changes.

NSX Manager backups can be taken on demand or on an hourly, daily, or weekly basis.

We recommend taking backups in the following scenarios:

• Before an NSX or vCenter upgrade.
• After an NSX or vCenter upgrade.
• After Day Zero deployment and initial configuration of NSX components, such as after the creation of NSX Controllers, logical switches, logical routers, edge services gateways, security, and firewall policies.
• After infrastructure or topology changes.
• After any major Day 2 change.

To provide an entire system state at a given time to roll back to, we recommend synchronizing NSX component backups (such as NSX Manager) with your backup schedule for other interacting components, such as vCenter, cloud management systems, operational tools, and so on.

[cnrz]

Could Snapshotting the Edge and/or NSX Manager be an option, taking the snapshot at t1, and at t2 rolling back the snapshot to t1 is supported or recommended?

Backup technologies using Snapshot is not supported as below, but this procedure Snapshot technology for a short duration  would be used, and deleted if rollback is not needed.

https://kb.vmware.com/s/article/2142263

NSX is unavailable from the vSphere Web Client plug-in after taking a backup of NSX Manager with quiesced snapshot.

• The NSX plug-in User Interface (UI) is reachable but the pages do not display content.
• In the vmware.log file of the NSX Manager VM, you see entries similar to:
  
  2016-02-25T15:00:48.586Z| vmx| I120: SnapshotVMXTakeSnapshotComplete: Snapshot 0 failed: Failed to quiesce the virtual machine (31).
  
  Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.

---

Cause

---

This issue occurs due to a scheduled backup run by a third-party that leverages quiesced snapshots. This causes the NSX Manager to become unresponsive through the vSphere Web Client.

---

Resolution

---

To avoid this issue, VMware recommends you to follow these best practices when taking a backup of the NSX Manager:

Note: Backup/restore functionality provided by NSX is the only supported way to backup/restore the NSX Manager. VMware does not currently test or support any third party tool that takes snapshot of the NSX Manager.

[parmarr]

Please ensure to review the Backup and Restore NSX Manager section of the NSX Upgrade Guide - https://docs.vmware.com/en/VMware-NSX-for-vSphere/6.3/com.vmware.nsx.upgrade.doc/GUID-4467950E-1E95-.... Note: Also review the section: Restore NSX Edges as all NSX Edge configurations (logical routers and edge services gateways) are backed up as part of NSX Manager data backup.

[cnrz]

Thanks for the answer,

It is recommended to take periodic backup as well as before major changes on the whole infrastructure, but a full restore to recover 1-2 hours of configuration changes done on one Edge may not be the best option. Since the configuration of NSX components are done through GUI as well as Rest API or Vro workflows, if it is possible to rollback the configuration then it may ensure to turn to the previous state before the start of the configuration of that specific configurations done for that time period.

For example if configuration changes are done through 10 REST API PUT  or POST commands, then similar commands with DELETE option reverses this configuration. VRO or automation tools as Ansible. Qquestion was if there are simpler tools to learn or use rather than full automation solutions just for this purpose. This may be a Postman addon or similar

This is similar to VRNI and ARM ono NSX, ARM has microsegmentation features and comes with NSX, VRNI is more diverse, and may be used if more enhanced Visibility or planning is needed.

https://blogs.vmware.com/management/2017/10/vra-nsx-application-services-design.html

http://blog.ipcraft.net/nsx-configuration-management-ansible/

http://www.punchingclouds.com/2016/11/29/hci-automated-deployment-configuration-vsphere-vsan-nsx-vio...

For this solution, I decided to utilize an Open Source deployment and configuration automation toolkit developed by VMware called Chaperone. Chaperone “aka El Chapo” as I like to call it, is an Ansible based tool that is intended to expedite and standardize “typical” deployments and configuration of VMware solutions. VMware open source software projects can be found on the VMware Open Software page on GitHub. For those that are not familiar with Ansible here is a brief definition for context and to get you started:

[tanurkov]

HI

you can use this scrip

please test it before using

______________________________________________________________________________________________

#!/usr/bin/env python

import requests

from lxml import etree

# Authentication Credential

username = 'admin'

password = 'password'

# Request Body Format

body_format = {'Content-Type': 'application/xml'}

# URI to LB

uri_query_lbx = 'https://nsx-manager1/api/4.0/edges/edgeId-X/loadbalancer/config'

# Query Existing LB

rqst = requests.get (uri_query_lbx, auth = (username, password), verify = False, headers = body_format)

# Print status of request

print "Status is :" , rqst.status_code "for /n" , rqst.text

#Write to the file

file = open(path/to/the/file, "w+")

file.write(rqst.text)

file.close()

________________________________________________________________________________________

There is no reason to back up NSX Manger VM it self. just use a backup option in NSX Manager.

Yes you can create a snapshot but only due to perform some task and in short time. Snapshot are not supported on NSX Manager and Edge.

[cnrz]

Thanks, I think Python and NSX Python could be used, but not sure if possible to rollback to steps in between time t0 and t5.  If at t1 changes done ch1 and ch2, t2 ch3,4,5, t3 ch6, t4 ch7,8,9,10 and t5 ch11 and 12.  Could it be possible to rollback to any of  t0, t1,t2,t3,t4 states if configurations are done on a single Edge, or different Edges, dFW rules or Logical Switches? Also showing the difference of 2 different times for example  t1 and t3.

General terminology seems as CRUD, but being a broad topic, how to use the (mostly developer) tools for a specific task is not very clear. There may be tools for these type of tasks as it would be useful for testing and change management purposes.

This example may be helpful, although some customization could be needed. (NSX-T is used with json format API Programmability, not sure if usable with NSX-v.

https://blogs.vmware.com/networkvirtualization/2018/01/nsx-t-openapi-sdks.html/

A similar example again using  same json format, although not for NSX, but the logic could be similar as providing rollback and comparing difference of different states.

http://acitoolkit.readthedocs.io/en/latest/snapback.html

Snapback is a Configuration Snapshot and Rollback tool for ACI fabrics. Specifically, the tool allows an administrator to perform the following tasks:

• Live snapshots of the running ACI fabric configuration
• One-time and recurring snapshots, both immediate and scheduled
• Versioned storage of the configuration
• Full viewing of any snapshot configuration including the differences between snapshots
• Rollback to any previous configuration snapshot; Full or Partial
• Web based or Command Line administration