The goal of that document is to give a very deep technical understanding on
. How to configure the different network and security services in OpenStack
. How OpenStack/NSX-T works
It is based on OpenStack Queens.
Note: It also highlights the specifics with VIO.
Excellent presentation! I've returned to this presentation repeatedly for reference and understanding during our cloud build. Particularly the edge case (2nd tier-0 router, provider NSGs, etc) explanations have been very helpful. Walking through what happens on the backend gives me a solid mental map from OpenStack to NSX-T.
One suggestion, please dive into how Neutron Availability Zones are implemented with NSX-T. It looks like it's a simple mapping of Neutron AZ to a Tier 0 router uuid/edge cluster uuid pair. Some things I'm trying to better understand about it:
- When is it useful? We'd like to avoid them if possible by deploying Edge Nodes across our hardware fault domains in a single edge cluster. But limitations on Tier 1 A/S scheduling may break those plans.
- Will our OpenStack logical routers be resilient to an AZ failure? How does that work with the tier 1? (ie Active in one az, standby in another?)
- Like upstream, will users select multiple availability zones when creating networks? Can a single application take advantage of multiple Neutron AZ's?
- Can I isolate tier-1 routers to an edge cluster by selecting an edge cluster that does not host the tier 0 router? I believe the answer is yes, by explicitly specifying the default edge cluster in the driver config.
- Impacts to BGP (advertising aggregate vs /32, etc)
I'd really like to avoid the Neutron AZ's as it pushes additional complexity on the user. I'm also concerned they'll create an uneven load on our edge clusters. Here's some ideas I had for how NSX-T might better support this:
1) Tier 1 routers are deployed to Edge Nodes of a Cluster in a deterministic order (node 1, node 2, node 3). I can build all odd nodes in one fault domain, and all even edge nodes in the other. This would guarantee that an A/S pair is not built in a single fault domain.
2) The driver and NSX-T could support multiple standbys for a Tier 1 and deploy a standby on all other nodes in a cluster. I'm sure this is easier said than implemented. Primary election becomes more challenging.
3) NSX-T deploys a new standby when both the active and standby Tier 1 SRs fail. Effectively allowing any Edge Node in the cluster to take over the Tier 1 workload.
Thanks so much for providing this slide deck!
Just did add the "AZ" section.
It's a small section, as it's simply offers specific NSX-T configuration per AZ.
And use cases are:
. different NSX-T Mgr
. different Edge Nodes for default_T0, and/or default_overlay, and/or default_vlan, and/or the metadata-proxy, and/or DHCP.
Can you detail how you expect users to leverage multiple network AZ's to enhance their application availability? I'm not quite understanding how this would work in practice.
Couple assumptions (please check):
The only network design that comes to mind is multi-homing all the VM instances. I'd need to attach each VM instance to each network backed by a different AZ. And in this design, I couldn't use LBaaS to load balance traffic across these, and I couldn't have a single external IP address.
Is there a better or different way?
. I can only attach one router per network.
And with Neutron AZ, you can decide on which Edge Cluster it will be deployed (configuring special "default_tier0_router").
. With NSX-T a router can be associated with only one AZ. (This differs from openvswitch OpenStack Docs: Availability zones)
. I can not attach routers to routers. My application needs to exchange some data between instances.
With Neutron NSX-T plugin, only 1 OpenStack Router can be attached to a specific OpenStack Network.
This OpenStack Router is "translater" to one NSX-T Tier-1 Gateway.
If your application is on different Openstack Networks each connected to different OpenStack Routers, then the communication is still possible with Neutron NSX-T Plugin. It will go from VM-A to T1-A to T0 to T1-B to VM-B.
Now if you have very specific design question, please send me a diagram on my email (firstname.lastname@example.org)
Thanks for sharing.
I try to deploy openstack with NSX-T via devstack, but failed.
It looks like something configured in local.conf was wrong:
++ tools/install_prereqs.sh:source:84 : python3_enabled
++ inc/python:python3_enabled:591 : [[ False == \T\r\u\e ]]
++ inc/python:python3_enabled:594 : return 1
+++ tools/install_prereqs.sh:source:88 : which python
++ tools/install_prereqs.sh:source:88 : export PYTHON=/usr/bin/python
++ tools/install_prereqs.sh:source:88 : PYTHON=/usr/bin/python
++ tools/install_prereqs.sh:source:94 : date +%s
++ tools/install_prereqs.sh:source:95 : date
+ ./stack.sh:main:759 : [[ False != \T\r\u\e ]]
+ ./stack.sh:main:760 : PYPI_ALTERNATIVE_URL=
+ ./stack.sh:main:760 : /opt/stack/devstack/tools/install_pip.sh
/opt/stack/devstack/.localrc.auto: line 102: DEFAULT_OVERLAY_TZ_UUID: command not found
++ ./stack.sh:main:760 : err_trap
++ ./stack.sh:err_trap:556 : local r=127
stack.sh failed: full log in /opt/stack/logs/stack.sh.log.2019-06-08-074441
Error on exit
Cloud you please give me some advice on troubleshooting?
My local.conf is given below:
# DevStack server devstack/local.conf #
# Specific post configuration for LBaaS with native NSX-T + QoS
service_provider = LOADBALANCERV2:VMWareEdge:neutron_lbaas.drivers.vmware.edge_driver_v2.EdgeLoadBalancerDriverV2:default
endpoints = RegionOne|http://172.16.18.65:9696
endpoint_type = publicURL
timeout = 30
admin_username = designate
admin_password = Eccom123
admin_tenant_name = service
auth_url = http://172.16.18.65/identity
insecure = False
auth_strategy = keystone
service_plugins = neutron_lbaas.services.loadbalancer.plugin.LoadBalancerPluginv2
service_plugins = neutron.services.qos.qos_plugin.QoSPlugin
# To allow VM with VLAN Trunk
vlan_transparent = true
# For Designate
dns_domain = dimi.fr.
external_dns_driver = designate
enabled = True
driver = vmware_nsxv3_edge
notification_drivers = vmware_nsxv3_message_queue
url = http://172.16.18.65:9001/v2
username = designate
password = Eccom123
project_name = service
auth_type = password
allow_reverse_dns_lookup = True
project_domain_name = Default
user_domain_name = Default
insecure = true
datastore_regex = NFS_DG
# local config
# Get OpenStack via HTTPS
# Enable Logging
# Use IPv4 only
# VMware nsxlib
# Horizon (Dashboard UI)
# Heat (Orchestration)
enable_plugin heat http://git.trystack.cn/openstack/heat stable/rocky
enable_plugin heat-dashboard http://git.trystack.cn/openstack/heat-dashboard stable/rocky
# Nova - Compute Service
# VNC server
# Glance - Image Service
# Neutron - Networking Service
# Use native DHCP and Metadata support
# Neutron - Firewall as a Service
enable_plugin neutron-fwaas http://git.trystack.cn/openstack/neutron-fwaas stable/rocky
enable_plugin neutron-fwaas-dashboard http://git.trystack.cn/openstack/neutron-fwaas-dashboard stable/rocky
# Enable LBaaS plugin
enable_plugin neutron-lbaas http://git.trystack.cn/openstack/neutron-lbaas stable/rocky
enable_plugin neutron-lbaas-dashboard http://git.trystack.cn/openstack/neutron-lbaas-dashboard stable/rocky
#enable_plugin octavia http://git.trystack.cn/openstack/octavia stable/rocky
#enable_plugin barbican http://git.trystack.cn/openstack/barbican stable/rocky
# Enable QoS
# Enable Designate
enable_plugin designate http://git.trystack.cn/openstack/designate stable/rocky
# L2 Gateway with NSX-T
enable_plugin networking-l2gw https://github.com/openstack/networking-l2gw stable/rocky
# Neutron - VPN as a Service
# Cinder - Block Device Service
# Apache fronted for WSGI
# Install Neutron Plugin #
# Neutron service with NSX-T
enable_plugin vmware-nsx https://github.com/openstack/vmware-nsx stable/rocky
DEFAULT_OVERLAY_TZ_UUID = 6bdb981c-a030-4a11-a235-6ea243c2dbb8
DEFAULT_TIER0_ROUTER_UUID = 2f913944-fa88-4bcd-bbe5-35fc3d91c254
NSX_MANAGER = 172.16.18.210
NSX_PASSWORD = Eccom@123Eccom@123
# DHCP server + MetaData Proxy with NSX-T
DHCP_PROFILE_UUID = 8eafc183-ff65-42bf-98d3-719741940d5d
METADATA_PROXY_UUID = 2e511a2e-8805-4833-9c86-a73187d6e1ef
METADATA_PROXY_SHARED_SECRET = Eccom123
As discussed by email, you have an extra “space” before the “=”.
thanks for sharing!
Hi, thanks for the excellent PPT.
For the external network, is someone able to explain what happened under the wood ? Because when you configure for example 184.108.40.206/24 as external network (with no SNAT), the default subnet used between T0 and T1 is 100.64.224.0/31 and except in Openstack, there is no reference to the 220.127.116.11/24 network in the route table of edges or external router.
When you enable SNAT, we are able to see the external ip in the route table.