Need clarification of LBT options for dvSwitch

[TheKenApp]

Hi,

I am tasked with creating a new 6.5 cluster, and it will utilize a dvSwitch.

I am trying, with not much success, to determine which load based teaming  option will best meet our needs:

• Route based on originating virtual port
• Route based on source MAC hash, or
• Rout based on physical NIC load.

We have four Dell R730 ESXi 6.5 hosts.

Each host has two 10GB 2-port copper cards, and one 10GB 2-port fiber card.

We will use one port on each of the two copper cards for all back-end traffic (vMotion, Fault Tolerance, vSAN and NFS storage). Each host will have 2 physical connections to a 2 member 10GB physical switch stack (Enterasys 7100) for this traffic.

We will use both fiber ports for all front end traffic (five virtual machine networks, backup network for Veeam backups of VMs, and ESX management).

Each fiber port will connect to one of our two core switches, which are stacked as well (Extreme 670's that have a  MLAG connection).

Since the traffic will be physically separated, I am not sure if I should create one or two distributed switches. This is a new architecture for us, but I am assuming that one switch would be fine, and I define which vmnics each port group would use, correct?

My current thought is that route based on source MAC hash may not be the best option, basically due to the higher resource consumption.

Are there any real advantages over the other two load based teaming options? I have been pouring over documentation, this forum and other resources in an effort to better understand this technology, but I am still kind of fuzzy. Our goal would be to have redundancy between the two fiber ports, and the two copper ports, as well as a certain degree of load balancing for each pair. It is the load balancing that is confusing me.

From what I am reading, route based on physical NIC load will load balance as needed so that neither link becomes saturated, is this correct?

From what I understand, when using route based on originating virtual port, load balancing occurs as VMs power on, selecting the best path to use. However, they will use the same port until either they are powered off, are vMotioned to another host, or a physical port fails. There is no active load balancing based on the traffic that is being sent by the running VMs, correct?

If my understanding is correct, our best bet would be go utilize route based on physical NIC load. Would you concur? If not, I am very open to any suggestions or insight you could provide.

Thanks,

Ken

[bayupw]

Hi Ken,

Let say this is your ports and vmnic numbering

Port Type and Port#	vmnic#	Physical Switch
Copper Port#1	vmnic0	Enterasys 7100
Copper Port#2	vmnic1	Enterasys 7100
Fibre Port#1	vmnic2	Extreme 670
Fibre Port#2	vmnic3	Extreme 670

I would create 2 dvswitches one for back-end traffic vmnic0 & vmnic1 and another one for vmnic2 & vmnic3 as they are connected to different switches

If you use one dvswitch for all 4 vmnics, you would need to overrides each of the portgroup configuration.

For example, if you have a vMotion portgroup on a dvswitch that connects to all vmnics, you would need to override the portgroup to not use vmnic2 and vmnic3.

Separating them into two dvswitches will make it more easier to manage.

You mentioned about stacking and MLAG, if you are going to configure the uplinks as active-active using port-channel or LACP, the options for teaming are IP Hash and LACP Enhanced LACP Support on a vSphere 5.5 Distributed Switch (2051826)

Your understanding on route based on physical NIC load (LBT) is correct and this is normally used for VM traffic.

If you want to use LBT or route based on originating virtual port (port ID) then you should not configure port-channel or LACP down to the ESXi hosts

[TheKenApp]

Thanks for the reply Bayu.

We want to avoid using LACP on the switch stacks, which is why I am evaluating the load based teaming options.

Yes, it does seem to make sense to create 2 different distributed vSwitches, since different traffic is routing to different physical switches. All the port groups on each switch can then be separate, which seems like it would be easier to set up, and for potential trouble shooting in the future.

For specificity's sake, on all hosts:

vmnic0 & vmnic1 are the fiber connections to the Extreme 670's

vmnic2 & vmnic4 are the copper connections to the Enterasys 7100's

vmnic3 & vmnic5 will not be physically connected.

[bayupw]

Yep, I agree with all of your statements.

I can see that you will have vSAN, NFS, etc

Some blog posts/links that may be useful for you:

Example Architectural Decision – Virtual Switch Load Balancing Policy | CloudXC

Designing vSAN Networks - Using Multiple Interfaces? - Virtual Blocks

Storage and Availability Technical Documents

[TheKenApp]

vSAN will not be implemented at this point, but we want to have the network defined and available for when we get to that point. Right now, all our VMs are on NFS.

[TheKenApp]

Bayu,

These will be the network loads for each set of vmnics:

Port Type and Port #	vmnic#	Physical switch	Networks
Copper Port#1	vmnic2	Enterasys 7100	vMotion, Fault Tolerance, vSAN and NFS storage
Copper Port#2	vmnic4	Enterasys 7100	vMotion, Fault Tolerance, vSAN and NFS storage
Fibre port#1	vmnic0	Extreme 670	5 different VM networks, ESX management, backup storage
Fibre port#2	vmnic1	Extreme 670	5 different VM networks, ESX management, backup storage

Separate port groups will be made for each network, with vmkernel ports created for vMotion, FT, ESX management, vSAN, and NFS storage (not tied to a specific service).

All ports are 10GB. I am not planning any link aggregation on the physical ports, only VLAN trunking.

Right now, my concern is the NFS traffic. I read through the information you pointed me towards, as well as several other sources. I found a series of whitepapers on NFS on vSphere Part 2 - Technical Deep Dive on Same Subnet Storage Traffic - Wahl Network

If I am to stick strictly with LBT (route based on physical NIC load) and no link aggregation, I realize that NFS traffic will not be shared among the 2 physical host ports to the 7100 switch stack. Considering we have two 10GB copper ports on each of the 4 new hosts, and a 4x10GB lag from the 7100 to the storage, I am not sure that this would be an issue with 20GB bandwidth on each host to the 7100 stack. I am assuming that the distributed vSwitch will move workloads other than NFS to the other vmnic, if one is saturated. Is this a correct understanding?

Is there any reason I should consider route based on originating virtual port, rather than on physical NIC load, in our opinion?

I have yet to dig into vSAN, but have some time since this will be a future use case. Though I hope that this architecture would be sufficient.

What are your thoughts?

Thanks,

[bayupw]

Love that blog post series by chriswahl​

For VMkernel PortGroups I normally use active/standby for things like ESXi Management VMkernel & vMotion VMkernel as these type of VMkernels based on my understanding do not use other vmnics if the first active vmnic is not failed.

I think someone also have same opinion/theory as written in this blog post: vMotion NIC load balancing fails even though there is an active link « rakhesh.com

"my theory is that for a VMkernel NIC (such as vMotion) backed by multiple physical NICs and using the default load balancing algorithm of virtual port ID – all traffic by default goes into one of the physical NICs and the other physical NIC is never used unless the chosen one fails"

For VMkernels to use multiple vmnic, they typically require additional IP address such as in NFS, iSCSI, Multiple-NIC vMotion in vSphere (2007467) or VTEP VMkernel for NSX.

I normally use LBT/physical NIC load only for non VMkernel PortGroups such as VM networks or backup networks.

Depending on the requirements, I also had a project that requires the backup to be active on a specific vmnic and standby on the other vmnic.

Regarding bandwidth, you can use vSphere Network I/O Control to allocate bandwidth for each of the traffic type in vDS

[TheKenApp]

The more I read about LBT, Route Based on Physical NIC Load, from what I understand, if the physical NIC load exceeds 75% utilization, it is the VM load that is taken into account for balancing. I am having a hard time determining if other loads, like you mention vMotion, or more importantly in my case the NFS VM storage, is also monitored and balanced if a physical NIC becomes more than 75% saturated.

In are case, I am considering both links being active, which is how our current VM infrastructure is designed (using Rout based on originating VM port).

You said that you use LBT/Physical NIC load only for non VMkernel PortGroups such as VM networks. If you use NFS (or ISCSI for that matter), how do you handle load balancing/NIC teaming?

The blog by Chris Wahl uses LBT/Physical NIC load with NFS. He states "any portgroup will proactively monitor the vmnic utilization in their team and shift workloads around." This would seem to indicate that it is not only VM workloads that are monitored and balanced.

Our NFS arrays are on the same subnet. NFS traffice is not routed, and the hosts, and storage are connected to the same physical switch. There is a 4x 10GB lag on the physical switch to the NFS array, so the bottle neck will be on the host. If I am understanding LBT correctly, the NFS portgroup/vmk in the DvSwitch will use one of the phyiscal NICs in the host (Without LACP on the switch for these ports, not sure how that traffic would be shared between the ports), but other loads, such as ESX management, and vMotion may be moved if the NFS link is over 75% utilization.

Am I understanding how LBT will work with these non-VM loads correctly?

Thanks again.

[bayupw · Accepted solution]

Yep I have same understanding as yours.

As you may aware, Chris Wahl has a blog post on NFS with LBT NFS on vSphere Part 4 – Technical Deep Dive on Load Based Teaming - Wahl Network

As he covered in his blog post series, if you want to load balance the NFS (NFS use multiple vmnic uplinks) you would need multiple IP addresses for the VMKs.

If the VMKs are in one subnet, NFS uses only one vmnic uplink and will use the other uplink for failover (high availability) or when you use LBT and the vmnic is over 75% utilisation.

The NFS will only able to get up to 10G (1 vmnic) maximum.

If you would like NFS to use both vmnic uplinks you would need VMKs on multiple subnets.

To apply network control, bandwidth allocation, limit for your vmk vmnics (e.g. make sure vMotion and FT do not eat all the available bandwidth) you would want to use NIOC as mentioned earlier.

Also covered in this blog post Using both Storage I/O Control & Network I/O Control for NFS - VMware vSphere Blog

[TheKenApp]

Thanks for taking the time to respond to all my questions bayupw, I appreciate it.