Trying to understand how a network loop occurred..

[rayvd]

I'm sure I won't be providing nearly enough information, but hopefully can fill in the blanks where needed.  Had an issue in our ESXi 5.1 environment a couple days ago where our Cisco Nexus switches observed a network loop and went into some sort of state where traffic wasn't being properly forwarded (presumably STP kicked in and stopped forwarding on certain key uplinks).  Solution ended up being the reboot of one of the Nexus switches.  We aren't the network team but are trying to wrap our heads around what may or may not have happened.

We have a Dell Blade Center with dual Force10 switches in it.  Each "blade" has one connection to each of these switches.  The switches each have two uplinks to separate Nexus 5K switches.  We run numerous VLANs over these uplinks (both storage -- NFS and production traffic).  Ideally we'd have physical separation for storage and front end traffic, but they are 10GbE lines and things were architected to just send everything over the two links with VLAN segmentation only.  We're not close to saturating the lines.

Within ESXi, there are two dvSwitches using teaming... traffic is balanced based on physical NIC load.  In thinking about this it's not clear that this is the right setting for accessing NFS-based datastores on our NetApp filers, but is how things were set up originally and has been working OK for a long time.

Our network admins tell us that at some point, the MAC address associated with the Storage vmkernel NIC showed up in two different places.  The Nexus switches saw this as a loop and apparently this triggered all of our issues.

I'm trying to wrap my head around how that MAC could have showed up two places at once?  Could our use of Physical NIC Load balancing be the cause?  One would think it would expose the MAC only one direction and wouldn't be flipping back between the two ports unless there was sufficient load (highly unlikely that we'd see 75% load on a 10GbE link in our environment).

Too many gaps in the above tale?  Just looking for some ideas on where to start troubleshooting first.  We're working with VMware Support, Dell and Cisco, but none are seeing anything obviously wrong in "their" layer of the stack.

[HeathReynolds]

Load based teaming is designed to work without specific requirements on the upstream switches. LBT shouldn't be the cause of this issue. Normally when I see a loop caused by a server it is a sun box with misconfigured teaming. I have never seen ESXi cause a loop.

Have them review the version of NXOS they are running in the bug toolkit, over time we have hit some nasty bugs although more on the 7k than 5k.

The ports on the Dell facing down to ESXi need to be configured to the Dell version of "spanning tree portfast trunk" on cisco. At least with cisco switches without the trunk command the port can go through the long STP learning phase when things fail over, causing a short loss of network connectivity.

[Punitsolanki]

If a MAC address learned on one switch port has moved to a different port, the MAC address and time stamp are recorded for the most recent arrival port. Then, the previous entry is deleted. If a MAC address is found already present in the table for the correct arrival port, only its time stamp is updated.

Exactly what happens when a host’s MAC address is learned on one switch port, and then the host moves so that it appears on a different switch port?Ordinarily, the host’s original CAM table entry would have to age out after 300 seconds, while its address was learned on the new port. To avoid having duplicate CAM table entries, a switch purges any existing entries for a MAC address that has just been learned on a different switch port. This is a safe assumption because MAC addresses are unique, and a single host should never be seen on more than one switch port unless problems exist in the network. If a switch notices that a MAC address is being learned on alternating switch ports, it generates an error message that flags the MAC address as “flapping” between interfaces.

Check the network switches for misconfigurations that might cause a data-forwarding loop.

If you aren't running spanning-tree, turn it on.

To track down a loop, you start with the #show mac-address-table address [flapping mac] command

We see that the MAC is coming in on port gi0/5 and gi0/16. One port will lead us to where that MAC is plugged in and the other will lead us to the loop. Pick a port and start working through.

Or Some load balancing techniques can send traffic to both ports, and that would cause the switch to go crazy, since it is receiving traffic from the same MAC on two or more different ports.

Fix this type of LB make it active/standby or make sure the server uses 2 different mac addresses, one per NIC

[GMCON]

When you say you are using NIC teaming I assume you mean you are using multiple NIC's or are you running LACP?  The VMware recommended practice is to use portfast on Cisco switches or what is called "Port Edge" for Nexus switches.  This is to be configured on all ports that are hooked up to ESXi hosts, this is only to be set up on end points like a server.  The reason you set this up is if you do have a hiccup or failure and the switch sees MAC addresses pop up on different ports then it does not shutdown ports and send it into a error mode thinking there is a loop.  As a vSwitch cannot cause a loop there is no risk in turning on portfast or the equivalent on Nexus, port edge.