VMware Cloud Community
HieuPhan
Contributor
Contributor

vSphere 5 : Active-Active NIC and RSTP

Hi everyone,

I have a situation that I would like to get opinion and recommendation from you guys.

we have 2 Force10 S4810 with RSTP turned on, LAGGED together using their 40Gbps ports. VLAN configured.

We have 1 CNA cards with 2 10Gbps port on each of our ESXi 5 hosts. Each physical port on the card give us 4 virtual NICs (VMware physical adapters), therefore I have total of 8 to use for both iSCSI connection and Network. I use NIC Teaming with Active Active enabled.

Connection to switches using 2 10GB fiber cables.

1.) If I have 2 VM,VM01 and VM02 for example,  on the same port-group (VM_10G),same VLAN (VLAN ID 10) same host (ESX1), multiple physical adapters in portgroup(vmnic1,vmnic2) , they can talk to each other ok

2.) If I only setup 1 physical adapter (VMnic1) per portgroup (VM_10G) on both hosts, ESX1 and ESX2, and migrate the Vm02 to ESX2, both VMs are still able to ping each other.

3.) This is where I'm stuck: if I add an addition physical adapter (VMNIC2) to the port group, while the VM lies on 2 different ESX hosts, the communication drops.

   The communication only comes back if I migrate both VM into the same hosts.

any other physical computers/servers can communicate to both VMs.

Any physical and virtual servers/computers from different VLANS are able to communicate to both VMs.

Do you think this is because of Spanning-tree? Since spanning-tree, even with RSTP, is still an active-standby situation. And on Vmware NIC-teaming, I set everything is Active-Active.

My next step is to switch Nic Teaming back to Active-Passive  to see if it resolves the problem. However, I want to hear your opinion first. and also, I don't like the idea of Active-Passive Nic teaming, kinda defeat the purpose of Nic-teaming and LoadBalancing.

Please help me!

Tags (2)
Reply
0 Kudos
10 Replies
rickardnobel
Champion
Champion

HieuPhan wrote:

Do you think this is because of Spanning-tree? Since spanning-tree, even with RSTP, is still an active-standby situation. And on Vmware NIC-teaming, I set everything is Active-Active.

No, it should not be a RSTP issue. This is for several reasons, e.g. the VMware vSwitches does not participate in any RSTP and will just drop any BPDU frames being sent into them.

Also, spanning-tree is kind of "active-passive", but more exactly a loop-free topology. If there are links between switches that would create a loop this should indeed put the most suitable link into passive mode. However, this will not be the case for the vSwitch.

A vSwitch with Port ID Nic Teaming and for example with 2 vmnics will to the outside world actually look like two "physical" switches and will for that reason never trigger RSTP handling. This is often not explained in VMware documentation, perhaps as it would confuse things ever more. Smiley Happy

My VMware blog: www.rickardnobel.se
HieuPhan
Contributor
Contributor

Thank you for answering my question Rickard. If RSTP is not a problem, what should I do in this situation.

The oddest thing is communication between different VLANs are ok, doesn't matter how nic-teaming or switches are setup. Only communication among the same VLAN using the 10G channel is giving me trouble.

Would this be something with QLOGIC CNA?

This is what I also tried earlier and failed to resolve the issue.

I create new vSwitch for one particular portgroup on 2 hosts, make port-group become Trunk/Tagged (means All VLAN accessible). Since I use VMXNET3 driver on the VM, i modify the NIC setting on each VM (windows OS) to tag the VLAN ID into it. The VM still didn't communicate with each other.

I'm pulling my hair out with this.

Reply
0 Kudos
rickardnobel
Champion
Champion

When you have two vmnics connected to the vSwitch, are these two from the two different fibre cables leaving the host?

Do you have one or two physical switches connected to the hosts?

My VMware blog: www.rickardnobel.se
Reply
0 Kudos
HieuPhan
Contributor
Contributor

Hi Rickard,

I purposely chose 2 vmnics that went out to 2 different fiber cable leaving the host. And these 2 fiber cables are connected to 2 different physical Force10 switches.

On the switches, I have RSTP enabled. The switches are interconnected using LACP.

All ports connected to VM Hosts and the SAN are independent from each other.

Reply
0 Kudos
rickardnobel
Champion
Champion

HieuPhan wrote:

Hi Rickard,

I purposely chose 2 vmnics that went out to 2 different fiber cable leaving the host. And these 2 fiber cables are connected to 2 different physical Force10 switches.

On the switches, I have RSTP enabled. The switches are interconnected using LACP.

All ports connected to VM Hosts and the SAN are independent from each other.

All this seems good. Could you however check the connection between these two switches and make sure it is not blocked by RSTP. This should be available through the switch CLI or web GUI.

Do the connection between these switches also have the specific VLANs?

My VMware blog: www.rickardnobel.se
Reply
0 Kudos
HieuPhan
Contributor
Contributor

Hi Rickard,

The LACP between the switches are up. All interfaces are up.

The connection between the switches are trunk, also are the connections between the switches and VM hosts.

Reply
0 Kudos
rickardnobel
Champion
Champion

The link might be up, but logically blocked by RSTP. Could you verify this?

My VMware blog: www.rickardnobel.se
Reply
0 Kudos
HieuPhan
Contributor
Contributor

Hi Rickard,

the link is not blocked by RSTP. Smiley Happy

Reply
0 Kudos
rickardnobel
Champion
Champion

HieuPhan wrote:

Hi Rickard,

the link is not blocked by RSTP. Smiley Happy

You have checked on both switches? If the link should be blocked by RSTP only one side will be put into the blocked mode, the other will look like it is up.

My VMware blog: www.rickardnobel.se
Reply
0 Kudos
milton123
Hot Shot
Hot Shot

In order to configure RSTP your all switch to switch link is trunk mode.

Cheers, Udin

Reply
0 Kudos