OpenStack&NSX-T - Network Topologies Configuration Guide

OpenStack&NSX-T - Network Topologies Configuration Guide

The goal of that document is to give a very deep technical understanding on

     . How to configure the different network and security services in OpenStack

     . How OpenStack/NSX-T works

 

It is based on OpenStack Queens.

Note: It also highlights the specifics with VIO.

 

Dimitri

Attachments
Comments

Excellent presentation! I've returned to this presentation repeatedly for reference and understanding during our cloud build. Particularly the edge case (2nd tier-0 router, provider NSGs, etc) explanations have been very helpful. Walking through what happens on the backend gives me a solid mental map from OpenStack to NSX-T.

One suggestion, please dive into how Neutron Availability Zones are implemented with NSX-T. It looks like it's a simple mapping of Neutron AZ to a Tier 0 router uuid/edge cluster uuid pair. Some things I'm trying to better understand about it:

- When is it useful? We'd like to avoid them if possible by deploying Edge Nodes across our hardware fault domains in a single edge cluster. But limitations on Tier 1 A/S scheduling may break those plans.

- Will our OpenStack logical routers be resilient to an AZ failure? How does that work with the tier 1? (ie Active in one az, standby in another?)

- Like upstream, will users select multiple availability zones when creating networks? Can a single application take advantage of multiple Neutron AZ's?

- Can I isolate tier-1 routers to an edge cluster by selecting an edge cluster that does not host the tier 0 router? I believe the answer is yes, by explicitly specifying the default edge cluster in the driver config.

- Impacts to BGP (advertising aggregate vs /32, etc)

I'd really like to avoid the Neutron AZ's as it pushes additional complexity on the user. I'm also concerned they'll create an uneven load on our edge clusters. Here's some ideas I had for how NSX-T might better support this:

1) Tier 1 routers are deployed to Edge Nodes of a Cluster in a deterministic order (node 1, node 2, node 3). I can build all odd nodes in one fault domain, and all even edge nodes in the other. This would guarantee that an A/S pair is not built in a single fault domain.

2) The driver and NSX-T could support multiple standbys for a Tier 1 and deploy a standby on all other nodes in a cluster. I'm sure this is easier said than implemented. Primary election becomes more challenging.

3) NSX-T deploys a new standby when both the active and standby Tier 1 SRs fail. Effectively allowing any Edge Node in the cluster to take over the Tier 1 workload.

Thanks so much for providing this slide deck!

Just did add the "AZ" section.

It's a small section, as it's simply offers specific NSX-T configuration per AZ.

And use cases are:

. different NSX-T Mgr

. different Edge Nodes for default_T0, and/or default_overlay, and/or default_vlan, and/or the metadata-proxy, and/or DHCP.

Can you detail how you expect users to leverage multiple network AZ's to enhance their application availability?  I'm not quite understanding how this would work in practice.

Couple assumptions (please check):

  • I can only attach one router per network.
  • With NSX-T a router can be associated with only one AZ. (This differs from openvswitch OpenStack Docs: Availability zones)
  • I can not attach routers to routers.
  • My application needs to exchange some data between instances.

The only network design that comes to mind is multi-homing all the VM instances. I'd need to attach each VM instance to each network backed by a  different AZ. And in this design, I couldn't use LBaaS to load balance traffic across these, and I couldn't have a single external IP address.


Is there a better or different way?

. I can only attach one router per network.

Correct.

And with Neutron AZ, you can decide on which Edge Cluster it will be deployed (configuring special "default_tier0_router").

. With NSX-T a router can be associated with only one AZ. (This differs from openvswitch OpenStack Docs: Availability zones)

Correct.

. I can not attach routers to routers. My application needs to exchange some data between instances.

With Neutron NSX-T plugin, only 1 OpenStack Router can be attached to a specific OpenStack Network.

This OpenStack Router is "translater" to one NSX-T Tier-1 Gateway.

If your application is on different Openstack Networks each connected to different OpenStack Routers, then the communication is still possible with Neutron NSX-T Plugin. It will go from VM-A to T1-A to T0 to T1-B to VM-B.

Now if you have very specific design question, please send me a diagram on my email (ddesmidt@vmware.com)

Thanks for sharing.

I try to deploy openstack with NSX-T via devstack, but failed.

It looks like something configured in local.conf was wrong:

++ tools/install_prereqs.sh:source:84       :   python3_enabled

++ inc/python:python3_enabled:591           :   [[ False == \T\r\u\e ]]

++ inc/python:python3_enabled:594           :   return 1

+++ tools/install_prereqs.sh:source:88       :   which python

++ tools/install_prereqs.sh:source:88       :   export PYTHON=/usr/bin/python

++ tools/install_prereqs.sh:source:88       :   PYTHON=/usr/bin/python

++ tools/install_prereqs.sh:source:94       :   date +%s

++ tools/install_prereqs.sh:source:95       :   date

+ ./stack.sh:main:759                      :   [[ False != \T\r\u\e ]]

+ ./stack.sh:main:760                      :   PYPI_ALTERNATIVE_URL=

+ ./stack.sh:main:760                      :   /opt/stack/devstack/tools/install_pip.sh

/opt/stack/devstack/.localrc.auto: line 102: DEFAULT_OVERLAY_TZ_UUID: command not found

++ ./stack.sh:main:760                      :   err_trap

++ ./stack.sh:err_trap:556                  :   local r=127

stack.sh failed: full log in /opt/stack/logs/stack.sh.log.2019-06-08-074441

Error on exit

Cloud you please give me some advice on troubleshooting?

My local.conf is given below:

#######################################

# DevStack server devstack/local.conf #

#######################################

# Specific post configuration for LBaaS with native NSX-T + QoS

[[post-config|$NEUTRON_LBAAS_CONF]]

[service_providers]

service_provider = LOADBALANCERV2:VMWareEdge:neutron_lbaas.drivers.vmware.edge_driver_v2.EdgeLoadBalancerDriverV2:default

[[post-config|$DESIGNATE_CONF]]

[network_api:neutron]

endpoints = RegionOne|http://172.16.18.65:9696

endpoint_type = publicURL

timeout = 30

admin_username = designate

admin_password = Eccom123

admin_tenant_name = service

auth_url = http://172.16.18.65/identity

insecure = False

auth_strategy = keystone

[[post-config|$NEUTRON_CONF]]

[DEFAULT]

service_plugins = neutron_lbaas.services.loadbalancer.plugin.LoadBalancerPluginv2

service_plugins = neutron.services.qos.qos_plugin.QoSPlugin

# To allow VM with VLAN Trunk

vlan_transparent = true

# For Designate

dns_domain = dimi.fr.

external_dns_driver = designate

[fwaas]

enabled = True

driver = vmware_nsxv3_edge

[qos]

notification_drivers = vmware_nsxv3_message_queue

[designate]

url = http://172.16.18.65:9001/v2

auth_url = http://172.16.18.65/identity

username = designate

password = Eccom123

project_name = service

auth_type = password

allow_reverse_dns_lookup = True

project_domain_name = Default

user_domain_name = Default

[[post-config|$NOVA_CONF]]

[vmware]

insecure = true

use_linked_clone=true

datastore_regex = NFS_DG

# local config

[[local|localrc]]

# Get OpenStack via HTTPS

GIT_BASE=http://git.trystack.cn/

NOVNC_REPO=http://git.trystack.cn/kanaka/noVNC.git

SPICE_REPO=http://git.trystack.cn/git/spice/spice-html5.git

HOST_IP=172.16.18.65

MULTI_HOST=1

SERVICE_HOST=172.16.18.65

DATABASE_PASSWORD=Eccom123

ADMIN_PASSWORD=Eccom123

SERVICE_PASSWORD=Eccom123

SERVICE_TOKEN=Eccom123

RABBIT_PASSWORD=Eccom123

# Enable Logging

USE_SCREEN=True

LOGFILE=/opt/stack/logs/stack.sh.log

VERBOSE=True

LOG_COLOR=False

SCREEN_LOGDIR=/opt/stack/logs

RECLONE=True

# Use IPv4 only

IP_VERSION=4

PIP_UPGRADE=True

# VMware nsxlib

LIBS_FROM_GIT=vmware-nsxlib

NSXLIB_BRANCH=stable/rocky

# Pre-requisite

ENABLED_SERVICES=rabbit,mysql,key

# Horizon (Dashboard UI)

ENABLED_SERVICES+=,horizon

# Heat (Orchestration)

ENABLED_SERVICES+=,h-eng,h-api,h-api-cfn,h-api-cw

enable_plugin heat http://git.trystack.cn/openstack/heat stable/rocky

enable_plugin heat-dashboard http://git.trystack.cn/openstack/heat-dashboard stable/rocky

# Nova - Compute Service

ENABLED_SERVICES+=,n-api,n-api-meta,n-obj,n-cond,n-sch,placement-api

DOWNLOAD_DEFAULT_IMAGES=False

# VNC server

ENABLED_SERVICES+=,n-novnc,n-xvnc,n-cauth

NOVNC_BRANCH=v0.6.0

# Glance - Image Service

ENABLED_SERVICES+=,g-api,g-reg

# Neutron - Networking Service

ENABLED_SERVICES+=,q-svc,neutron

# Use native DHCP and Metadata support

# ENABLED_SERVICES+=,q-dhcp,q-meta

# Neutron - Firewall as a Service

ENABLED_SERVICES+=,q-fwaas-v1

enable_plugin neutron-fwaas http://git.trystack.cn/openstack/neutron-fwaas stable/rocky

enable_plugin neutron-fwaas-dashboard http://git.trystack.cn/openstack/neutron-fwaas-dashboard stable/rocky

# Enable LBaaS plugin

enable_plugin neutron-lbaas http://git.trystack.cn/openstack/neutron-lbaas stable/rocky

enable_plugin neutron-lbaas-dashboard http://git.trystack.cn/openstack/neutron-lbaas-dashboard stable/rocky

#enable_plugin octavia http://git.trystack.cn/openstack/octavia stable/rocky

#enable_plugin barbican http://git.trystack.cn/openstack/barbican stable/rocky

#ENABLED_SERVICES+=q-lbaasv2,octavia,o-api,o-cw,o-hk,o-hm

ENABLED_SERVICES+=,q-lbaasv2

# Enable QoS

ENABLED_SERVICES+=,q-qos

# Enable Designate

enable_plugin designate http://git.trystack.cn/openstack/designate stable/rocky

ENABLED_SERVICES+=,designate,designate-central,designate-api,designate-worker,designate-producer,designate-mdns

# L2 Gateway with NSX-T

enable_plugin networking-l2gw https://github.com/openstack/networking-l2gw stable/rocky

NETWORKING_L2GW_SERVICE_DRIVER=L2GW:vmware-nsx-l2gw:vmware_nsx.services.l2gateway.nsx_v3.driver.NsxV3Driver:default

# Neutron - VPN as a Service

ENABLED_SERVICES+=,q-vpn

# Cinder - Block Device Service

#ENABLED_SERVICES+=,cinder,c-api,c-vol,c-sch,c-bak

# Apache fronted for WSGI

APACHE_ENABLED_SERVICES+=keystone,swift

##########################

# Install Neutron Plugin #

##########################

# Neutron service with NSX-T

enable_plugin vmware-nsx https://github.com/openstack/vmware-nsx stable/rocky

Q_PLUGIN=vmware_nsx_v3

DEFAULT_OVERLAY_TZ_UUID = 6bdb981c-a030-4a11-a235-6ea243c2dbb8

DEFAULT_TIER0_ROUTER_UUID = 2f913944-fa88-4bcd-bbe5-35fc3d91c254

#DEFAULT_BRIDGE_CLUSTER_UUID=100a94c2-26f1-45cf-89fc-eb57ec971f0b

NSX_MANAGER = 172.16.18.210

NSX_USER=admin

NSX_PASSWORD = Eccom@123Eccom@123

# DHCP server + MetaData Proxy with NSX-T

DHCP_PROFILE_UUID = 8eafc183-ff65-42bf-98d3-719741940d5d

METADATA_PROXY_UUID = 2e511a2e-8805-4833-9c86-a73187d6e1ef

METADATA_PROXY_SHARED_SECRET = Eccom123

NATIVE_DHCP_METADATA=True

As discussed by email, you have an extra “space” before the “=”.

DEFAULT_OVERLAY_TZ_UUID=6bdb981c-a030-4a11-a235-6ea243c2dbb8

thanks for sharing!

Hi, thanks for the excellent PPT.

For the external network, is someone able to explain what happened under the wood ? Because when you configure for example 30.30.30.0/24 as external network (with no SNAT), the default subnet used between T0 and T1 is 100.64.224.0/31 and except in Openstack, there is no reference to the 30.30.30.0/24 network in the route table of edges or external router.

When you enable SNAT, we are able to see the external ip in the route table.

Thanks


Regards

 

Alban

Version history
Revision #:
2 of 2
Last update:
‎09-01-2021 01:27 PM
Updated by:
 
Contributors