NSX Edge Cluster VTEP Load Balancing and Failover Order

[Carlos_E]

Hello,

So I have a vSphere 6.5 + vCloud Director 8.20 + NSX 6.3.2 deployment with 2 Clusters, one for Edge / Management and one for Compute.

The vDS dedicated to the Compute Cluster uses 4 uplinks with a LACP configuration.

All the Distributed Port Groups use "Route based on IP hash" as the Load Balancing algorithm and the LAG with the 4 uplinks as the active uplink.

All the "logical switches / virtual wires / nsx port groups" (I´m not sure what´s the best way to call them) on this vDS use "Route based on originating virtual port" as the Load Balancing algorithm and the LAG with the 4 uplinks as the active uplink.

All is nice and well with this configuration of NSX on the Compute Cluster.

The vDS dedicated to the Edge / Management Cluster used 2 uplinks , all the Distributed Port Groups on this vDS used "Route based on originating virtual port" as the Load Balancing algorithm and uplink 1 as the Active and uplink 2 as the Standby adapter.

All is nice and well with this configuration of NSX on the Edge / Management Cluster.

I want to improve on my Edge / Management Cluster and be able to use a configuration which let´s me use more than 1 uplink as the active one, this has been a confusing issue for me and I need some help to understand if what I have done is a supported / recommended configuration.

Here are a couple of questions to which I have not found a clear answer :

- It´s my understanding that it is not recommended to use LACP on the Edge / Management Cluster, this is correct, right ?

- Can I have on my vDS dedicated to the Edge / Management Cluster one configuration for my Distributed Port Groups and another one for my "logical switches / virtual wires / nsx port groups" ?

From a ticket that I had with VMware I ended up understanding that one is not supposed to change the "logical siwtches / virtual wires / nsx port groups" configuration by hand once they are created, much less to change the port group created by NSX "vxw-vmknicPg" and that information in addition to this chart which I have found in several places and documentation :

Have left me wondering if what I have done is supported / recommended or not, remember my objective is to improve on my Edge / Managament Cluster and use more than 1 uplink as active.

What I did was :

- I unconfigured the Edge / Management Cluster completely.

- I configured on my vDS dedicated to Edge / Management Cluster all of the Distributed Port Groups with "Routed based on physical NIC load" as the Load Balancing algorithm and both uplink 1 and uplink 2 as the active adapters, and when I configured the Cluster for NSX I choose "Load Balance - SRCID" as the VMKNic Teaming Policy, this ended up configuring all the "logical switches / virtual wires / nsx port groups" with "Route based on originating virtual port" as the Load Balancing algorithm and in the Failover order both uplinks ended up being in the active section.

Is this supported ?

Being the scenario as I described, is this the best way to improve it to use more than 1 uplink as active ?

Well, thanks in advanced to anyone for reading and if you have some feedback it will be appreciated!!

Regards,

Carlos.

[Sreec]

Hello Carlos,

                      You are correct, LACP for Edge is not the best choice considering the connectivity in physical layer like vPC/MLAG  . In some of the environments i have seen single VPC going all the way till Core. The whole purpose of Second Uplink is for better bandwidth/availability and SRC-ID based Teaming is better for Edge cluster,but remember if you are going with Multi Vtep - each vtep will be pinned to each uplink and that way with teaming policy it will utilize both the uplinks .Changing Logical Switch names don't have any direct impact on traffic flow. But there is a strong reason why the naming pattern is like that -easy identification and troubleshooting. Down the line when we keep doing this(Rename) - i don't find any difference between creation of vSphere port-group(Manual process keeping vcd aside) ,renaming each and every NSX logical switches. So that defeats the purpose on-fly port-group creation ,especially in VCD environment. So don't do that and it is not supported as well(Any VCD managed object should not be changed at VC layer)

[Carlos_E]

Sreec,

Reading your answer "You are correct, LACP for Edge is not the best choice considering the connectivity in physical layer like vPC/MLAG  . In some of the environments i have seen single VPC going all the way till Core. The whole purpose of Second Uplink is for better bandwidth/availability and SRC-ID based Teaming is better for Edge cluster,but remember if you are going with Multi Vtep - each vtep will be pinned to each uplink and that way with teaming policy it will utilize both the uplinks ."

Seems like I´m on the right track on improving my Edge / Management Cluster, so just to see if I understood your answer correctly :

- Using "Load Balance - SRCID" as the VMKNic Teaming Policy is a valid way to achieve better bandwidth/availability for all "NSX Logical Switches / Port Groups" on my Edge / Management Cluster, either with 2 uplinks or more, correct ? (I´m thinking if this is valid to go up to 4 uplinks)

What I didn´t fully grasp from your answer was what would be the answer regarding having one Load Balancing algorithm´s in my "Edge / Management" vDS on my regular/legacy Distributed Port Groups ("Route based on physical NIC load") and have a different one "Load Balance - SRCID" for the NSX Logical Switches / Port Groups on that same vDS.

Am I good with this configuration or is there some conflict in mixing one algorithm for the Distributed Port Groups and another one for NSX that I might not be aware ? (the GUI let´s me do it...and it seems reasonable but if I have learn something about NSX in this past year and a half is that sometimes certain things just need to be done in a NSX friendly way and the GUI does not always reminds you of what is friendly and what is not friendly)

Thanks for your feedback, some of this topics are complicated to find a definitive answer and after reading almost everything related to this topic on google my options are reduced to the good will of someone like you on this forum ; )

Regards,

Carlos.

[bayupw]

Hi Carlos,

In my opinion, the recommendation would be depend on your environment but the general is to use Multi-VTEP and avoid LACP if it is not really needed.

Just want to confirm, is your Compute VDS uses LACP + IP Hash on the VDS level configuration and on the VXLAN uses Port ID?

Taken from NSX Install doc Configure VXLAN Transport Parameters

Do not mix different teaming policies for different portgroups on a vSphere Distributed Switch where some use Etherchannel or LACPv1 or LACPv2 and others use a different teaming policy. If uplinks are shared in these different teaming policies, traffic will be interrupted. If logical routers are present, there will be routing problems. Such a configuration is not supported and should be avoided.

The best practice for IP hash-based teaming (EtherChannel, LACPv1 or LACPv2) is to use all uplinks on the vSphere Distributed Switch in the team, and do not have portgroups on that vSphere Distributed Switch with different teaming policies. For more information and further guidance, see the VMware® NSX for vSphere Network Virtualization Design Guide at https://communities.vmware.com/docs/DOC-27683.

In multi-VTEP environment, the number of VTEP will match the number of VDS dvUplinks.

Do you use blade servers or rack servers?
Do you have MC-LAG - Wikipedia such as vPC or MLAG on your physical switches?

Here's a traffic flow of single-VTEP (without LACP)

and this one multi-VTEP (two VTEP in this case)

This single & multi-VTEP traffic flow is also explained in this whitepaper: NSX+Cisco Nexus 7000/UCS Design Guide

To utilise multiple uplinks the option would be

1. Multi-VTEP Source ID or Source MAC (Source ID has performance advantage over MAC as it does not require MAC address lookup when forwarding packet)

2. Single VTEP LACP

Regarding your question on this: Can I have on my vDS dedicated to the Edge / Management Cluster one configuration for my Distributed Port Groups and another one for my "logical switches / virtual wires / nsx port groups" ?

For distributed port groups, do you mean VLAN-backed portgroups/non-VXLAN portgroups?

You can technically use different teaming depending on requirements on that specific portgroup and your environment.

What are you trying to achieve?

Please take note of mixing different teaming as per doc would interrupt traffic e.g. one VDS some portgroup with LACP some with with other policy.

[Sreec]

Dear Carlos,

                      Mixing NIC teaming is not the right approach with/without NSX , Bayu has already updated that in the below thread with precise details. You should understand your application traffic pattern ,to be more precise "Who is generating more/less , Are all generating equal amount etc ",  For me that is the baseline for placing them in the correct logical switch with a teaming policy - Both SCID & MAC based policy might end up in same state of uplink selection/usage -if traffic pattern is not identical . But less CPU intensive is SCID based policy and that is preferred when LCAP is not used . 

[Carlos_E]

Bayu,

I read both of your answers several times and took some time to think about them so as to not waste your time and make the most of the following questions, let´s see if I understood correctly your suggestions, I´ll try to be concise about the questions and divide this into 2 sections, one regarding the Compute Cluster and the other one for the Edge / Management Cluster.

COMPUTE CLUSTER SECTION :

Damn! I was looking on improving my "Edge / Management Cluster", but it seems in the process you help me found a problem on the configuration of my "Compute Cluster", let´s see if you can help me clear that out first.

This is my present vDS and NSX configuration on the Compute Cluster :

All the VLAN-Backed Distributed Port Groups were configured by hand with the following selections :

- Load Balancing = "Route based on IP hash"

- Failover order = As active a LACP named LAGCompute01 with 4 Uplinks

The LACP Load Balancing mode of LAGCompute01 is "Source and destination IP address, TCP/UDP port and VLAN"

NSX :

- VMKNic Teaming Policy selected when configuring the Cluster = Enhanced LACP

- Load Balancing that ended up being configured by NSX on the VXLAN Port Groups based on the VMKNic selection = "Route based on originating virtual port"

- Failover order that ended up being configured by NSX on the VXLAN Port Groups based on the VMKNic selection = As active a LACP named LAGCompute01 with 4 Uplinks

The LACP Load Balancing mode of LAGCompute01 is "Source and destination IP address, TCP/UDP port and VLAN"

At this point if I understood you correctly even though things work it´s not a best practice to have a different Load Balancing algorithm on the Distributed Port Groups, I should change "Route based on IP hash" to "Route based on originating virtual port" to match what was configured by NSX after selecting "Enhanced LACP", is this correct ?, am I currently in an unsupported configuration ? (so far things have worked on the Compute Cluster)

I seem to recall making the "Route based on IP hash" on the Distributed Port Groups decision based on something regarding the LACP, but maybe the LACP works also great with "Route based on originating virtual port".

EDGE / MANAGEMENT CLUSTER SECTION :

I quote your answer and answer some parts inline (blue)"Regarding your question on this: Can I have on my vDS dedicated to the Edge / Management Cluster one configuration for my Distributed Port Groups and another one for my "logical switches / virtual wires / nsx port groups" ?

For distributed port groups, do you mean VLAN-backed portgroups/non-VXLAN portgroups?

Yes, I mean VLAN-Backed Port Groups created by hand and not NSX VXLAN Port Groups.

You can technically use different teaming depending on requirements on that specific portgroup and your environment.

So I guess the problem with mixing Load Balancing Algorithm´s between VLAN-Backed Distributed Port Groups and NSX/VXLAN Port Groups only applies when there´s a LACP in the mix ?

What are you trying to achieve?

I want to have more bandwidth/availability on my Edge/Management Cluster.

Please take note of mixing different teaming as per doc would interrupt traffic e.g. one VDS some portgroup with LACP some with with other policy."

This is my present vDS and NSX configuration on the Edge / Management Cluster :

All the VLAN-Backed Distributed Port Groups were configured by hand with the following selections :

- Load Balancing = "Route based on originating virtual port"

- Failover order = Active : Uplink 1, Standby : Uplink2

NSX :

- VMKNic Teaming Policy selected when configuring the Cluster = Fail Over

- Load Balancing that ended up being configured by NSX on the VXLAN Port Groups based on the VMKNic selection = "Use explicit failover order"

- Failover order that ended up being configured by NSX on the VXLAN Port Groups based on the VMKNic selection = Active : Uplink 2, Standby : Active 1 (the behavior where NSX randomizes the order of the uplinks).

The change that I wanted to do (and actually did it on a Testing Cluster) in order to achieve better bandwidth/availability was this :

All the VLAN-Backed Distributed Port Groups were configured by hand with the following selections :

- Load Balancing = "Route based on physical NIC load"

- Failover order = Acitve : Uplink 1 & Uplink 2

NSX :

- VMKNic Teaming Policy selected when configuring the Cluster = Load Balance - SRCID

- Load Balancing that ended up being configured by NSX on the NSX Port Groups based on the VMKNic selection = "Route based on originating virtual port"

- Failover order that ended up being configured by NSX on the NSX Port Groups based on the VMKNic selection = Active-Uplink 2 & Uplink 1

Is this change supported on an Edge / Management Cluster without ? (this Edge / Management cluster does not have LACP in the mix, just to be clear)

Wow...this was long, and complex!!!, my head hurts ; )

I hope I at least stated the issue clearly.

Thanks so much for your time in advanced.

Regards,

Carlos.

[bayupw · Accepted solution]

Hi Carlos,

You have a very good understanding and explanations.

For the compute cluster side, you actually have used the correct teaming which is LACP as the other VLAN-backed portgroups on Compute VDS are on LACP and I assume your vmnic uplinks to physical switches are also in LACP.

I can't find an explanation about this in vSphere/NSX doc but as you are using LAG, my understanding is that the load balancing policy in the portgroup level is not relevant anymore as depicted in the following screenshot

Just make sure when you use LACP as the load balancing policy for VXLAN/VTEP on your compute, it is using the correct lag ports.

So I don't think you need to touch this for your compute as the LAG overrides the load balancing policy anyway.

On your Edge/Management, again in my opinion you are in the right.

No LACP on this cluster so you can use LBT (Load Based Teaming/Route Based on Physical NIC Load) for VLAN-backed portgroups.

For NSX, as it does not support LBT you would use Multi-VTEP and Load Balance Source ID is the perfect and recommended teaming to use multiple uplinks.

Make sure your VDS number of uplinks matches the number of vmnics attached to the VDS as the number of VTEP will match this dvUplinks

Configure VXLAN Transport Parameters

The number of VTEPs is not editable in the UI. The VTEP number is set to match the number of dvUplin...

[Carlos_E]

Bayu,

Now that you mention it I remember that message saying "the load balancing mode of the LAG overrides the load balancing of the port group´s..." but it had completely slip my mind, sorry!

Yes, all the ESXi´s of the Compute Cluster are using the correct LAG Ports so we can forget about this part.

Now regarding the Edge / Management Cluster : Excellent!!

Yes, the number of physical vmnics matches the number of uplinks defined on the vDS, so no problem there.

Some of the documentation and blogs about this issue had me quite confused.

You are an instructor, is this a common issue people have difficulty dealing with or is it just me ? feel free to say you never read a question so dumb about NSX ; )

Well, it´s good to read your feedback on the issue, I´m gonna go ahead and during a change window on the environment do this change to the Edge / Management Cluster.

If not for yours and Sreec feedback I would have probably had spend a lot more time digging for an answer on this and probably not finding a clear one ; )

So a big thank you to both.

Regards,

Carlos.

[bayupw]

Hi Carlos,

Awesome, all of my customers always ask about this so in my opinion this is common questions.

This topic is covered in the NSX reference design guide but not everyone have read or maybe know there is a design guide, here is the link in case you are unaware

VMware® NSX for vSphere Network Virtualization Design Guide ver 3.0

There are also two VMworld presentations that talks about NSX and vSphere design, if you are interested below are the links:

NET5770 - Reference Design for SDDC with NSX & vSphere - Part 1

NET5792 - Reference Design for SDDC with NSX & vSphere - Part 2

On the LACP side, all of my works in NSX does not involve LACP, design guide recommends to use Source ID and discourage LACP use due to vendor specific requirements.

Some people also deploy NSX on blade servers and not all blade switches support LACP.

But that does not mean you cannot or should not use LACP, it just depends on requirements and environment.

[Carlos_E]

Yes, I´ve read the design guide but obviously based on this thread I didn´t digest it well ; )

Same thing with the vCAT-SP, now that some time has passed since that first read through maybe things start making sense, but when I was just starting to learn neither guide fully got me from 0 to 10.

I think I saw those presentations (I´ve consumed pretty much all material available on NSX on the web hehe)  but I´ll give them another pass, maybe now with more hands on experience things make  more sense.

Thanks once again.

Regards,

Carlos.