The goal of that document is to give a very deep technical understanding on
. How to configure the different network and security services in OpenStack
. How OpenStack/NSX-T works
It is based on OpenStack Queens.
Note: It also highlights the specifics with VIO.
Dimitri
Excellent presentation! I've returned to this presentation repeatedly for reference and understanding during our cloud build. Particularly the edge case (2nd tier-0 router, provider NSGs, etc) explanations have been very helpful. Walking through what happens on the backend gives me a solid mental map from OpenStack to NSX-T.
One suggestion, please dive into how Neutron Availability Zones are implemented with NSX-T. It looks like it's a simple mapping of Neutron AZ to a Tier 0 router uuid/edge cluster uuid pair. Some things I'm trying to better understand about it:
- When is it useful? We'd like to avoid them if possible by deploying Edge Nodes across our hardware fault domains in a single edge cluster. But limitations on Tier 1 A/S scheduling may break those plans.
- Will our OpenStack logical routers be resilient to an AZ failure? How does that work with the tier 1? (ie Active in one az, standby in another?)
- Like upstream, will users select multiple availability zones when creating networks? Can a single application take advantage of multiple Neutron AZ's?
- Can I isolate tier-1 routers to an edge cluster by selecting an edge cluster that does not host the tier 0 router? I believe the answer is yes, by explicitly specifying the default edge cluster in the driver config.
- Impacts to BGP (advertising aggregate vs /32, etc)
I'd really like to avoid the Neutron AZ's as it pushes additional complexity on the user. I'm also concerned they'll create an uneven load on our edge clusters. Here's some ideas I had for how NSX-T might better support this:
1) Tier 1 routers are deployed to Edge Nodes of a Cluster in a deterministic order (node 1, node 2, node 3). I can build all odd nodes in one fault domain, and all even edge nodes in the other. This would guarantee that an A/S pair is not built in a single fault domain.
2) The driver and NSX-T could support multiple standbys for a Tier 1 and deploy a standby on all other nodes in a cluster. I'm sure this is easier said than implemented. Primary election becomes more challenging.
3) NSX-T deploys a new standby when both the active and standby Tier 1 SRs fail. Effectively allowing any Edge Node in the cluster to take over the Tier 1 workload.
Thanks so much for providing this slide deck!
Just did add the "AZ" section.
It's a small section, as it's simply offers specific NSX-T configuration per AZ.
And use cases are:
. different NSX-T Mgr
. different Edge Nodes for default_T0, and/or default_overlay, and/or default_vlan, and/or the metadata-proxy, and/or DHCP.
Can you detail how you expect users to leverage multiple network AZ's to enhance their application availability? I'm not quite understanding how this would work in practice.
Couple assumptions (please check):
The only network design that comes to mind is multi-homing all the VM instances. I'd need to attach each VM instance to each network backed by a different AZ. And in this design, I couldn't use LBaaS to load balance traffic across these, and I couldn't have a single external IP address.
Is there a better or different way?
. I can only attach one router per network.
Correct.
And with Neutron AZ, you can decide on which Edge Cluster it will be deployed (configuring special "default_tier0_router").
. With NSX-T a router can be associated with only one AZ. (This differs from openvswitch OpenStack Docs: Availability zones)
Correct.
. I can not attach routers to routers. My application needs to exchange some data between instances.
With Neutron NSX-T plugin, only 1 OpenStack Router can be attached to a specific OpenStack Network.
This OpenStack Router is "translater" to one NSX-T Tier-1 Gateway.
If your application is on different Openstack Networks each connected to different OpenStack Routers, then the communication is still possible with Neutron NSX-T Plugin. It will go from VM-A to T1-A to T0 to T1-B to VM-B.
Now if you have very specific design question, please send me a diagram on my email (ddesmidt@vmware.com)
Thanks for sharing.
I try to deploy openstack with NSX-T via devstack, but failed.
It looks like something configured in local.conf was wrong:
++ tools/install_prereqs.sh:source:84 : python3_enabled
++ inc/python:python3_enabled:591 : [[ False == \T\r\u\e ]]
++ inc/python:python3_enabled:594 : return 1
+++ tools/install_prereqs.sh:source:88 : which python
++ tools/install_prereqs.sh:source:88 : export PYTHON=/usr/bin/python
++ tools/install_prereqs.sh:source:88 : PYTHON=/usr/bin/python
++ tools/install_prereqs.sh:source:94 : date +%s
++ tools/install_prereqs.sh:source:95 : date
+ ./stack.sh:main:759 : [[ False != \T\r\u\e ]]
+ ./stack.sh:main:760 : PYPI_ALTERNATIVE_URL=
+ ./stack.sh:main:760 : /opt/stack/devstack/tools/install_pip.sh
/opt/stack/devstack/.localrc.auto: line 102: DEFAULT_OVERLAY_TZ_UUID: command not found
++ ./stack.sh:main:760 : err_trap
++ ./stack.sh:err_trap:556 : local r=127
stack.sh failed: full log in /opt/stack/logs/stack.sh.log.2019-06-08-074441
Error on exit
Cloud you please give me some advice on troubleshooting?
My local.conf is given below:
#######################################
# DevStack server devstack/local.conf #
#######################################
# Specific post configuration for LBaaS with native NSX-T + QoS
[[post-config|$NEUTRON_LBAAS_CONF]]
[service_providers]
service_provider = LOADBALANCERV2:VMWareEdge:neutron_lbaas.drivers.vmware.edge_driver_v2.EdgeLoadBalancerDriverV2:default
[[post-config|$DESIGNATE_CONF]]
[network_api:neutron]
endpoints = RegionOne|http://172.16.18.65:9696
endpoint_type = publicURL
timeout = 30
admin_username = designate
admin_password = Eccom123
admin_tenant_name = service
auth_url = http://172.16.18.65/identity
insecure = False
auth_strategy = keystone
[[post-config|$NEUTRON_CONF]]
[DEFAULT]
service_plugins = neutron_lbaas.services.loadbalancer.plugin.LoadBalancerPluginv2
service_plugins = neutron.services.qos.qos_plugin.QoSPlugin
# To allow VM with VLAN Trunk
vlan_transparent = true
# For Designate
dns_domain = dimi.fr.
external_dns_driver = designate
[fwaas]
enabled = True
driver = vmware_nsxv3_edge
[qos]
notification_drivers = vmware_nsxv3_message_queue
[designate]
url = http://172.16.18.65:9001/v2
auth_url = http://172.16.18.65/identity
username = designate
password = Eccom123
project_name = service
auth_type = password
allow_reverse_dns_lookup = True
project_domain_name = Default
user_domain_name = Default
[[post-config|$NOVA_CONF]]
[vmware]
insecure = true
use_linked_clone=true
datastore_regex = NFS_DG
# local config
[[local|localrc]]
# Get OpenStack via HTTPS
GIT_BASE=http://git.trystack.cn/
NOVNC_REPO=http://git.trystack.cn/kanaka/noVNC.git
SPICE_REPO=http://git.trystack.cn/git/spice/spice-html5.git
HOST_IP=172.16.18.65
MULTI_HOST=1
SERVICE_HOST=172.16.18.65
DATABASE_PASSWORD=Eccom123
ADMIN_PASSWORD=Eccom123
SERVICE_PASSWORD=Eccom123
SERVICE_TOKEN=Eccom123
RABBIT_PASSWORD=Eccom123
# Enable Logging
USE_SCREEN=True
LOGFILE=/opt/stack/logs/stack.sh.log
VERBOSE=True
LOG_COLOR=False
SCREEN_LOGDIR=/opt/stack/logs
RECLONE=True
# Use IPv4 only
IP_VERSION=4
PIP_UPGRADE=True
# VMware nsxlib
LIBS_FROM_GIT=vmware-nsxlib
NSXLIB_BRANCH=stable/rocky
# Pre-requisite
ENABLED_SERVICES=rabbit,mysql,key
# Horizon (Dashboard UI)
ENABLED_SERVICES+=,horizon
# Heat (Orchestration)
ENABLED_SERVICES+=,h-eng,h-api,h-api-cfn,h-api-cw
enable_plugin heat http://git.trystack.cn/openstack/heat stable/rocky
enable_plugin heat-dashboard http://git.trystack.cn/openstack/heat-dashboard stable/rocky
# Nova - Compute Service
ENABLED_SERVICES+=,n-api,n-api-meta,n-obj,n-cond,n-sch,placement-api
DOWNLOAD_DEFAULT_IMAGES=False
# VNC server
ENABLED_SERVICES+=,n-novnc,n-xvnc,n-cauth
NOVNC_BRANCH=v0.6.0
# Glance - Image Service
ENABLED_SERVICES+=,g-api,g-reg
# Neutron - Networking Service
ENABLED_SERVICES+=,q-svc,neutron
# Use native DHCP and Metadata support
# ENABLED_SERVICES+=,q-dhcp,q-meta
# Neutron - Firewall as a Service
ENABLED_SERVICES+=,q-fwaas-v1
enable_plugin neutron-fwaas http://git.trystack.cn/openstack/neutron-fwaas stable/rocky
enable_plugin neutron-fwaas-dashboard http://git.trystack.cn/openstack/neutron-fwaas-dashboard stable/rocky
# Enable LBaaS plugin
enable_plugin neutron-lbaas http://git.trystack.cn/openstack/neutron-lbaas stable/rocky
enable_plugin neutron-lbaas-dashboard http://git.trystack.cn/openstack/neutron-lbaas-dashboard stable/rocky
#enable_plugin octavia http://git.trystack.cn/openstack/octavia stable/rocky
#enable_plugin barbican http://git.trystack.cn/openstack/barbican stable/rocky
#ENABLED_SERVICES+=q-lbaasv2,octavia,o-api,o-cw,o-hk,o-hm
ENABLED_SERVICES+=,q-lbaasv2
# Enable QoS
ENABLED_SERVICES+=,q-qos
# Enable Designate
enable_plugin designate http://git.trystack.cn/openstack/designate stable/rocky
ENABLED_SERVICES+=,designate,designate-central,designate-api,designate-worker,designate-producer,designate-mdns
# L2 Gateway with NSX-T
enable_plugin networking-l2gw https://github.com/openstack/networking-l2gw stable/rocky
NETWORKING_L2GW_SERVICE_DRIVER=L2GW:vmware-nsx-l2gw:vmware_nsx.services.l2gateway.nsx_v3.driver.NsxV3Driver:default
# Neutron - VPN as a Service
ENABLED_SERVICES+=,q-vpn
# Cinder - Block Device Service
#ENABLED_SERVICES+=,cinder,c-api,c-vol,c-sch,c-bak
# Apache fronted for WSGI
APACHE_ENABLED_SERVICES+=keystone,swift
##########################
# Install Neutron Plugin #
##########################
# Neutron service with NSX-T
enable_plugin vmware-nsx https://github.com/openstack/vmware-nsx stable/rocky
Q_PLUGIN=vmware_nsx_v3
DEFAULT_OVERLAY_TZ_UUID = 6bdb981c-a030-4a11-a235-6ea243c2dbb8
DEFAULT_TIER0_ROUTER_UUID = 2f913944-fa88-4bcd-bbe5-35fc3d91c254
#DEFAULT_BRIDGE_CLUSTER_UUID=100a94c2-26f1-45cf-89fc-eb57ec971f0b
NSX_MANAGER = 172.16.18.210
NSX_USER=admin
NSX_PASSWORD = Eccom@123Eccom@123
# DHCP server + MetaData Proxy with NSX-T
DHCP_PROFILE_UUID = 8eafc183-ff65-42bf-98d3-719741940d5d
METADATA_PROXY_UUID = 2e511a2e-8805-4833-9c86-a73187d6e1ef
METADATA_PROXY_SHARED_SECRET = Eccom123
NATIVE_DHCP_METADATA=True
As discussed by email, you have an extra “space” before the “=”.
DEFAULT_OVERLAY_TZ_UUID=6bdb981c-a030-4a11-a235-6ea243c2dbb8
thanks for sharing!
Hi, thanks for the excellent PPT.
For the external network, is someone able to explain what happened under the wood ? Because when you configure for example 30.30.30.0/24 as external network (with no SNAT), the default subnet used between T0 and T1 is 100.64.224.0/31 and except in Openstack, there is no reference to the 30.30.30.0/24 network in the route table of edges or external router.
When you enable SNAT, we are able to see the external ip in the route table.
Thanks
Regards
Alban