NSX-T Active-Active Tier 0 SR with ECMP disabled

[pavel_stefanov]

Good day everyone,

I am trying to understand what the ECMP option under the various routing protocols accomplishes in NSX-T e.g. under BGP when deploying an active-active Tier 0 SR and documentation is not particularly clear on this. Configuration-wise active-active HA for a T0 gateway and ECMP under BGP are completely different configuration options, yet the difference is not particularly clear to me and I feel like these two concepts get used interchangeably (incorrectly) in the documentation.

Does the ECMP option allow a given active edge instance of a T0 SR to load-balance across multiple (local) upstream paths (e.g. one of the active edges has 4 upstream peers and load-balances traffic across all 4 peers) or does this ECMP option somehow influence the traffic pattern from the Tier 0/Tier 1 DR towards the Tier 0 SR edges?

I've managed to disable ECMP and inter-SR routing on an active-active T0 SR edge but I do not see any change in traffic patterns (note that both active edges have a single upstream peer i.e. same upstream physical router peering from the two active edges in this sample topology). What should be happening in such a scenario?

Any clarifications on this would be much appreciated.

[shank89 · Accepted solution]

Active-Active means both Edge nodes are active for data forwarding and routing operations.  Each additional 'Active' Edge node also adds an extra next hop for the  T1/T0's to balance across, for example.

The above addresses 169.254.0.2/3  exist on each Edge node (T0SR) on an Active/Active Tier-0 gateway.  The forwarding table in the image is that of the Tier-0 DR on a host transport node.  If the Tier-0 were in Active-Standby, there would only be one next hop / default route.

ECMP (Equal cost multipathing) means multiple paths to a destination prefix can be installed in the routing table and forwarding table.  The load / packets can be balanced using a 5 tuple hashing algorithm with the available paths.

For example, the below is the routing and forwarding table with ECMP enabled and a default route advertised from both upstream peers.

ECMP disabled

The other thing to consider is, if you have a Tier-0 gateway configured with Active-Active and ECMP disabled, you could inadvertently discard traffic if uRPF is enabled, as traffic may be received on an interface the T0SR did not expect it on.

The behaviour of ECMP with the various protocols is the same, so I won't demonstrate between each protocol.

Let me know if this clears it up for you.

PS. I cover this topic in my book 🙂

Thanks

[pavel_stefanov]

Many thanks, Shashank, for the detailed explanation. This certainly clarifies the open points I had. In summary:

- active-active HA mode affects the static routes provisioned on the transport nodes' DRs towards the T0 SRs i.e. multiple upstream next-hops.

- ECMP option affects the routing protocol and its route selection and the installation of multiple routing table entries for the same prefix in the routing table on the same edge. This would be the equivalent of BGP multipath on other vendors' equipment configured with the maximum-paths command.