Skip navigation

Dear readers

this is the second blog of a series related to NSX-T. This second coverage provide you relevant information required to better understand the implication of a centralized services in NSX-T. In the first blog where I have provided you an introduction of the lab setup, this second blog will now discuss the impact when you add a Tier-1 Edge Firewall for the tenant BLUE. The diagram below shows the logical representation of the lab setup with the Edge Firewall attached to the Tier-1 uplink interface of the Logical Router for tenant BLUE.



For this blog I have selected to add an Edge Firewall for a Tier-1 Logical Router, but I could have also selected a Load Balancer, VPN service or NAT service. The implication to the "internal" NSX-T networking are similar. However, please keep in mind, not all NSX-T centralized services are supported at the Tier-1 level (as example VPN) or at Tier-0 (as example Load Balancer) with NSX-T 2.3 and not all services (as example DHCP or Metadata Proxy) will instantiate a Service Router.


Before I move forward and try to explain what is happen under the hood when you enable an Edge Firewall, I would like to update you with some additional information to the diagram below.


I am sure you are already familiar with this diagram above, as we have talked about the same in my first blog. Each of the four Transport Nodes (TN) has two tenants Tier-1 Logical Routers instantiated. Inside of each Transport Node there are two Logical Switches with VNI 17295 and 17296 between the Tier-1 tenant DR and Tier-0 DR used. These two automatically instantiated (sometimes referred as auto-plumbing) transit overlay Logical Switches have got subnets and automatic assigned. Internal filtering avoids duplicate IP address challenges;  in the same way as NSX-T is doing already for gateways IP (.254) the Logical Switches 17289 and 17294 where the VMs are attached. Each of this Tier-1 to Tier-0 transit Logical Switch (17295 and 17296) could be showed as linked together in the diagram, but as internal filtering takes place, this is for the moment irrelevant.

The intra Tier-0 Logical Switch with the VNI 17292 is used to forward traffic between the Tier-0 DRs and towards northbound via the Service Router (SR). This Logical Switch 17292 has again an automatic assigned IP subnet ( Each Tier-0 DR has assigned the same IP address (.1), but the two Service Routers use different IPs (.2 and .3), otherwise the Tier-0 DR would not be able to forward based on equal cost with two different next hops.


Before the network administrator is able to configure an Edge Firewall for tenant BLUE at the Tier-1 level, he has to assign and edge-cluster to the Tier-1 Logical Router along the edge-cluster members. This is shown in the diagram below.


Please be aware, as soon as you assign an edge-cluster to a Tier-1 Logical Router a Service Router is automatically instantiated, independent of the Edge Firewall.


These two new Service Routers are running on the edge-nodes and they are in active/standby mode. Please see in the next diagram below.



The configuration of the tenant BLUE Edge Firewall itself is shown in the next diagram. Here we use for this lab the default firewall policy.


This simple configuration step with adding the two edge-nodes to the Tier-1 Logical Router for tenant BLUE will cause that NSX-T "re-organize" the internal auto-plumbing network. To understand what is happening under the hood, I have divided these internal network changes into four steps instead showing only the final result.


In step 1, NSX-T will internally disconnect the Tier-0 DR to Tier-1 DR for the BLUE tenant, as the northbound traffic needs to be redirected to the two edge-nodes, where the Tier-1 Service Routers are running. The internal Logical Switch with VNI 17295 is now explicit linked together between the four Transport Nodes (TN).



In step 2, NSX-T automatically instantiate on each edge-node a new Service Router at Tier-1 level for the tenant BLUE with an Edge Firewall. The Service Router are active/standby mode. In this example, the Service Router running on the Transport Node EN1-TN is active, where the Service Router running on EN2-TN is standby. The Tier-1 Service Router uplink interface with the IP address is either UP or DOWN.



In step 3, NSX-T connects the Tier-1 Service Router and the Distributed Router for the BLUE tenant together. For this connection is a new Logical Switch with VNI 17288 added. Again, the Service Router running on EN1-TN has the active interface with the IP address up and the Service Router on EN2-TN is down. This ensure, that only the active Service Router can forward traffic.



In the final step 4, NSX-T extends the Logical Switch with VNI 17288 to the two compute Transport Nodes ESX70A and ESX71A. This extension is required to route traffic from vm1 as example on the local host before the traffic is forwarded to the Edge Transport Nodes. NSX-T adds finally the required static routing between the different Distributed and Service Routers. NSX-T does all these steps under the hood automatically.



The next diagram below shows a traffic flow between vm1 and vm3. The traffic sourced from vm1 will first hit the local DR in the BLUE tenant on ESX70A-TN. The traffic now needs to be forwarded to the active Tier-1 Service Router (SR) with the Edge Firewall running on Edge Transport Node EN1-TN. The traffic reach then the Tier-0 DR on EN1-TN and then is the traffic forwarded to the RED Tier-1 DR and finally arrives at vm3. The return traffic will hit first the local DR in the RED tenant on ESX71A-TN before the traffic reach the Tier-0 DR on the same host. The next hop is the BLUE Tier-1 Service Router (SR). The Edge Firewall inspects the return traffic and forwards the traffic locally the BLUE Tier-1 DR before finally the traffic arrives back at vm1. The majority of the traffic is handled locally on the EN1-TN. The used bandwidth between the physically hosts and therefore the GENEVE encapsulated traffic is the same as without the Edge Firewall. But as everybody could imagine an edge-node which might hosts multiple Edge Firewalls for multiple tenants or any other centralized services should be designed accordingly.



Hope you had a little bit fun reading these two blogs. Feel free to share this blog!


Lab Software Details:


vSphere: 6.5.0 Update 1 Build 5969303

vCenter:  6.5 Update 1d Build 2143838


Version 1.0 - 10.12.2018

Dear readers

this is the first blog of a series related to NSX-T. This first coverage provide you a simple introduction to the most relevant information required to better understand the implication of a centralized services in NSX-T. A centralized service could be as example a Load Balancer or an Edge Firewall.


NSX-T has the ability to do distributed routing and supports distributed firewall. Distributed routing in the context that each host, which is prepared for NSX-T, can do local routing. From the logical view is this part called Distributed Router (DR). The DR is part of a Logical Router (LR) and this LR can be configured at Tier-0 or at Tier-1 level. Distributed routing is perfect for scale and could reduce the bandwidth utilization of each physical NIC on the host, as the routing decision is done on the local host. As example, when the source and the destination VM is located on the same host, but connected to different IP subnets and therefore attached to different overlay Logical Switches, then the traffic never leaves the host. All traffic forwarding is processed on the host itself instead at the physical network as example on the ToR switch.

Each host which is prepared with NSX-T and attached to a NSX-T Transport Zone is called a Transport Node (TN). Transport Nodes have implicit a N-VDS configured, which provides as example GENEVE Tunnel Endpoint or is responsible for the distributed Firewall processing. However, there are services like load balancing or edge firewalling which is not a distributed service. VMware call these services "centralized services". Centralized services instantiate a Service Router (SR) and this SR runs on the NSX-T edge-node (EN). An edge-node could be a VM or a bare metal server. Each edge-node is also a Transport Node (TN).


Lets have now a look to a simple two tier NSX-T topology with a tenant BLUE and a tenant RED. Both have for new no centralized services at Tier-1 level enabled. For the North-South connectivity to the physical world, there is already a centralized services at the Tier-0 instantiated. However, we don't want focus on this North-South routing part, but as we later would like the understand, what it means to have a centralized services configured on a Tier-1 logical router, it is important to understand this part as well, because North-South routing is also a centralized service. The diagram below shows the logical representation of a simple lab setup. This lab setup will later be used to instantiate the a centralized service at a Tier-1 Logical Router.


For those which like to get a better understanding of the topology, I have included a diagram of the physical view below. In this lab, we actually use 4 ESXi hosts. For simplification we focus in this blog on the Hypervisor ESXi, instead KVM, even we could build a similar lab with KVM too. On each of these two Transport Nodes ESX70A-TN and ESX71A-TN is a VM installed. The two other hosts ESX50A and ESX51A are NOT* prepared for NSX-T, but they host on each a single edge-node (EN1 and EN2) VM. These two edge-nodes don't have to run on two different ESXi hosts, but it is recommended for redundancy reason.


As shown in the next diagram, we combine now the physical and logical view. The two Transport Nodes ESX70A-TN and ESX71A-TN have only DRs at Tier-1 and Tier-0 level instantiated, but no Service Router. That means the Logical Router consists of only a DR. These DRs at Tier-1 level provide the gateway (.254) for the attached Logical Switch. The tenant BLUE uses VNI 17289 and the tenant RED uses VNI 17294. NSX-T assign these VNIs out of a VNI pool (default pool: 5000 - 65535). The edge-nodes VMs, now showed as Edge Transport Node (EN1-TN and EN2-TN) have the same Tier-1 and Tier-0 DRs instantiated, but only the Tier-0 includes a Service Router (SR).


The two Tier-1 Logical Routers respective DRs can only talk to each other via the green Tier-0 DR. But before you are able to attach the two Tier-1 DRs to a Tier-0 DR a Tier-0 Logical Router is required. And a Tier-0 Logical Router mandates the assignment of an edge-cluster during the configuration of the Tier-0 Logical Router. Lets assume at this point, we have already configured two edge-node VMs and these edge-node VMs are assigned to an edge-cluster. A Tier-0 Logical Router consists always of a Distributed Router (DR) and depending on the node type as well with a Service Router. A Service Router is always required for the Tier-0 Logical Router, as the Service Router is responsible for the routing connectivity to the physical world. But the Service Router is only instantiated on the edge-nodes. In this lab both Service Router are configured on two edge-nodes respective as Edge Transport Node in active/active mode to provide ECMP to the physical world.

All the internal transit links, as shown in the diagram below, are automatically configured through NSX-T. The only task for the network administrator is to connected the Tier-0 DR to the Tier-1 DRs.

The northbound connection to the physical world requires further a configuration of an additional (or better two Transport Zones for routing redundancy) VLAN based Transport Zone plus the routing peering (typically eBGP). Below is the resulting logical network topology.

One probably ask, why NSX-T instantiate on each edge-node the two Tier-1 DRs too? Well, this is required for an optimized forwarding. As already mentioned, routing decisions are always done on that hosts where the traffic is sourced. Assume vm1 in tenant BLUE would like to talk to a server in the physical world. Traffic sourced at vm1 is forwarded to its local gateway on the Tier-1 DR and then towards the traffic to the Tier-0 DR on the same host. From the Tier-0 DR is then the traffic forwarded to the left Tier-0 SR on EN1-TN (lets assume, traffic is hashed accordingly) and then the flow reach the external destination. The return traffic reach first Tier-0 SR on EN2-TN (lets assume again based on the hash), then the traffic is forwarded locally to Tier-0 DR on the same Edge Transport Node and then to the Tier-1 DR in tenant BLUE. The traffic never leaves EN2-TN until the traffic reach locally the Logical Switch where the vm1 is attached. This is what is called optimized forwarding which is possible due the distributed NSX-T architecture. The traffic needs to be forwarded only once over the physical data center infrastructure and therefore encapsulated into GENEVE per direction!


For now we close this first blog. For the second blog we will dive into the instantiation of a centralized service at Tier-1. Hope you had a little big fun reading this first write-up.





*Today, NSX-T supports to run edge-node VMs on NSX-T prepared hosts too. This capability is important to combine compute and edge-node services on the same host.

Version 1.0 - 19.11.2018

Version 1.1 - 27.11.2018 (minor changes)

Version 1.2 - 04.12.2018 (cosmetic changes)

Version 1.3 - 10.12.2018 (link for second blog added)