ESX NIC TEAMING

[RSEngineer]

I need to understand NIC Teaming so I can understand traffic flows and patterns and failover.

So bare with me if I ask very basic questions....

The scenario I have in mind is a blade server, but I guess the question applies to any server. So for simplicity's sake, lets use rack mount servers...

Assume a rack mount with 2 NICs that are teamed in any of the 3 teaming mechanisms used in ESX. With NIC teaming, there will never be a physical NIC that is inactive, correct? In oither words, both physical NICs in the team will have traffic going through them - ie load balancing/sharing. No?

[a_p_]

Although by default both uplinks are active, a single VM will only be assigned to one uplink at a time (except for IP hash load balancing). I'd recommend you take a look at http://www.vmware.com/files/pdf/virtual_networking_concepts.pdf which describes the different settings. In addition to this document there's an interesting blog post (http://blogs.vmware.com/networking/2011/05/lbt-load-based-teaming-explained-part-13.html) about load based teaming which was introduced with vSphere 4.1.

André

[chriswahl]

At it's heart, the teaming in vSphere is more for availability than it is for load balancing. Typically the VMs and vmkernels will use only one uplink out of the team, and have the ability to use the others if that one fails (link detection goes down) or if the upstream switch dies (beacon probing).

With IP or MAC hash, a VM or vmkernel will use more than one uplink for individual sessions, but a single session won't use both uplinks at the same time.

[rickardnobel]

RSEngineer wrote:

With NIC teaming, there will never be a physical NIC that is inactive, correct?

As noted above all NICs are typically active, but it is possible to create an active/standby setup (by some advanced settings) if this is needed.

[RSEngineer]

Hi, folks:

Thank you very much for your help...much appreciated.

I did some reading today: Scott Lowe's Mastering vSphere5.0. Chapter 5 covers vNetworking, including NIC Teaming. I have amuch better understanding. I am not really trying to achieve anything, just trying to understand how things work. IP-based hashed NIC teaming will most likely allow a single VM with a single vNIC to utilize both physical NICs (depends on the flows). And although port-based and MAC-based do not allow a VM to use both physical NICs, it will most likely allow an ESXi host with multiple VMs to use both physical NICs. Therefore, no matter what kind of NIC teaming is used, it is very likely that both physical NICs in the team will be forwardign traffic - perhaps at different levels, but nonetheless both will be active.

Is the above a fair statement?

BY THE WAY, I would mark each of your posts as the correct answer, but I dont want to discourage others from joining the discussion, as they may think the conversation is over. I also want to take this to the next level once I get my understanding validated.

[vGuy · Accepted solution]

IP-based hashed NIC teaming will most likely allow a single VM with a single vNIC to utilize both physical NICs (depends on the flows). And although port-based and MAC-based do not allow a VM to use both physical NICs, it will most likely allow an ESXi host with multiple VMs to use both physical NICs. Therefore, no matter what kind of NIC teaming is used, it is very likely that both physical NICs in the team will be forwardign traffic - perhaps at different levels, but nonetheless both will be active.

Yes, both the pNICs in the team can be active depending on the teaming policy and traffic flow. Although a single VM cannot use multiple Uplinks unless it's configured with atleast 2 Network adapters in Port-Based and MAC-Based Policy. With IP-Hash policy, it's possible for a single adapter VM to use multiple uplinks if the destination addresses are different. Addition to these is LBT (as Andre mentioned), it will monitor the uplinks, and if one of the uplinks is saturated (above 75% by default), it will move the VM traffic to another uplink.

Apart from the docs shared by others, I will suggest you to look at pt.3 of "Great vSwitch Debate"...i have found it quite helpful for myself.

http://kensvirtualreality.wordpress.com/2009/04/05/the-great-vswitch-debate%E2%80%93part-3/

[RSEngineer]

Thanks, vGuy. I am asking about NIC teaming and traffic flows because I am trying to determine best practices for esuring timely fault detection and failover. We have a situation involving virtualized blade servers that have a CNA on them that is dual-homed to 2 blade switches. Sometimes one of the blade switches suffers a software failure that prevents traffic from being forwarded, HOWEVER, the internal port remains UP! Therefore, the CNA never has knowledge of the failed uplinks on the physical switch and traffic gets blackholed.

This makes sense, since there won't be any failover if the fault has not first been detected, of course. So, the application(s) fail for about 5 minutes and then come up again. That is an area of confusion. Some people think that the MAC-aging time on the physical blade switch is causing this 5 minute outage. The MAC-aging time is 300 seconds...so , they are outting "2 and 2 together." But that does not make sense, since aging out the MAC entries does not force a reroute of traffic to an active path (the 2 blades are NOT connected to each other) and will definitely not force the internal switch por tto go down (and therefore the CNA port). So, there must be another reason for the roughly 5-minute outage. Perhaps failover is occurring on the application level....

[vGuy]

Is it possible that the traffic is being redirected by some mechanism at the blade switches itself? Are you using HP blades with Virtual Connec/Flex-10t?

Also, have you looked at enabling Beacon Probing on the vSwitches, it can be used to detect downstream failures:

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=100557...

[Josh26]

Be aware, any teaming policy that requires switch configuration (MAC hash and IP hash) won't work when you have two separate switches*.

Therefore for your situation, you would be limited to "route by virtual port ID" and "use explicit failover order".

* Caveat - Cisco Nexus with virtual port group. Since you are in a blade environment I don't believe this is available to you.

[rickardnobel]

Josh26 wrote:

Be aware, any teaming policy that requires switch configuration (MAC hash and IP hash) won't work when you have two separate switches*.

Therefore for your situation, you would be limited to "route by virtual port ID" and "use explicit failover order".

I really agree about the IP Hash - here you can only have one physical switch (unless some vendor specific method of cross switch etherchannel), but the MAC hash should work on any two physical switches.

[RSEngineer]

I agree. The MAC-hash has nothing to do with the physical switches. It basically uses a pinning model between the vNIC and an uplink, just like port-based Hash.

[RSEngineer]

vGuy:

I will look into probing. For it to work, though, you need at least 3  NICs in the team, otherwise, a failure will cause both NICs not to  receive each other's b'cast and the failover will be non-deterministic.

[a_p_]

Yes, Beacon Probing will only work with 3 or more uplinks. With only two uplinks you would never know whether it's the sender or the receiver which doesn't work. Depending on your switch vendor you may want to take a look at "Link State Tracking". This will set all downlinks (to the blades/hosts) in the Link State group to down if all uplinks go offline.

André

[ealaqqad]

Check out the following article which explain how traffic flow with the different NIC teaming policies & how LBT got advantages over the other methods available in there. The article can be found at: ESXi NIC TEAMING Load Balancing Policies

Hope this help,

Eiad Al-Aqqad

B: http://www.VirtualizationTeam.com

B: http://www.TSMGuru.com

T: @VirtualizationT