2 single 10 GB uplinks - LACP trouble with management network

[btniko]

Hi -

What is VMware recommended setup for a vDS with 2 single 10 GB uplinks?

My portgroups are only VM-traffic, vMotion and Management.

I notice when enabling LACP I get trouble with the management network (host disconnects).

Please advice.

KInd r

Bjørn.

[chriswahl · Accepted solution]

Typically it's using the default load balancing policy (route based on originating virtual port ID) or, for those with a distributed switch, load based teaming (route based on physical NIC load aka LBT).

LBT is my default go-to teaming policy, except for things like iSCSI vmk binding or some converged infrastructure scenarios as explained in this blog post.

I'm not a fan of using a link aggregation group (LAG) between a physical switch and a vSphere host. It adds complexity while offering little real world value, at the same time it eliminates the ability to use features such as iSCSI vmk binding and multi-NIC vMotion.

[vfk]

The Guru on VMware Networking has spoken, while he will not plug his own book, I will do it for him : Networking for VMware Administrators

It is a good read.

[btniko]

Thank you for great answer. Really helpfull.

I have another scenario with 4 uplinks (4 single 10 GB NICs).

- 2 uplinks are used for NFS/Mgmt/vMotion traffic, and is currently on vDS1. (No LACP enabled for these portgroups).

- 2 uplinks are used for VM-traffic, and is currently on vDS1. (LACP enabled on these portgroups)

So same vDS, but LACP enabled only on VM-traffic.

What would be your recomandation here?

Kind regards

Bjørn.

[chriswahl]

2 uplinks are used for VM-traffic, and is currently on vDS1. (LACP enabled on these portgroups)

So same vDS, but LACP enabled only on VM-traffic.

What would be your recommendation here?

Virtual machines are often best served by the load based teaming (LBT) policy, because LACP uses a hash to determine the path taken for traffic and does not look at active load; I consider LACP to be load distribution, not balancing.

While LACP does a rather nice job at randomizing the links that VMs will use (because the VMs tend to talk to a wide variety of different destination addresses), LACP has no way of intelligently balancing load. LBT, on the other hand, will help fix lopsided traffic flows by moving VMs to a lighter utilized vmnic when a link's load exceeds 75% for 30+ seconds.

Be sure that you have the physical ports connected to your ESXi host configured as network edge ports (spanning-tree immediately moves into a forwarding state) to avoid TCNs and other nastiness, and be aware that each time LBT migrates a workload there will be an update to the MAC address table on your physical switch.

I hope that helps. I go a bit deeper into the proper configuration of LACP, and some reasons you would want to use it, in this blog post.

[btniko]

Great, thank you very much for answer.