Hi everyone,
I have a situation that I would like to get opinion and recommendation from you guys.
we have 2 Force10 S4810 with RSTP turned on, LAGGED together using their 40Gbps ports. VLAN configured.
We have 1 CNA cards with 2 10Gbps port on each of our ESXi 5 hosts. Each physical port on the card give us 4 virtual NICs (VMware physical adapters), therefore I have total of 8 to use for both iSCSI connection and Network. I use NIC Teaming with Active Active enabled.
Connection to switches using 2 10GB fiber cables.
1.) If I have 2 VM,VM01 and VM02 for example, on the same port-group (VM_10G),same VLAN (VLAN ID 10) same host (ESX1), multiple physical adapters in portgroup(vmnic1,vmnic2) , they can talk to each other ok
2.) If I only setup 1 physical adapter (VMnic1) per portgroup (VM_10G) on both hosts, ESX1 and ESX2, and migrate the Vm02 to ESX2, both VMs are still able to ping each other.
3.) This is where I'm stuck: if I add an addition physical adapter (VMNIC2) to the port group, while the VM lies on 2 different ESX hosts, the communication drops.
The communication only comes back if I migrate both VM into the same hosts.
any other physical computers/servers can communicate to both VMs.
Any physical and virtual servers/computers from different VLANS are able to communicate to both VMs.
Do you think this is because of Spanning-tree? Since spanning-tree, even with RSTP, is still an active-standby situation. And on Vmware NIC-teaming, I set everything is Active-Active.
My next step is to switch Nic Teaming back to Active-Passive to see if it resolves the problem. However, I want to hear your opinion first. and also, I don't like the idea of Active-Passive Nic teaming, kinda defeat the purpose of Nic-teaming and LoadBalancing.
Please help me!
HieuPhan wrote:
Do you think this is because of Spanning-tree? Since spanning-tree, even with RSTP, is still an active-standby situation. And on Vmware NIC-teaming, I set everything is Active-Active.
No, it should not be a RSTP issue. This is for several reasons, e.g. the VMware vSwitches does not participate in any RSTP and will just drop any BPDU frames being sent into them.
Also, spanning-tree is kind of "active-passive", but more exactly a loop-free topology. If there are links between switches that would create a loop this should indeed put the most suitable link into passive mode. However, this will not be the case for the vSwitch.
A vSwitch with Port ID Nic Teaming and for example with 2 vmnics will to the outside world actually look like two "physical" switches and will for that reason never trigger RSTP handling. This is often not explained in VMware documentation, perhaps as it would confuse things ever more.
Thank you for answering my question Rickard. If RSTP is not a problem, what should I do in this situation.
The oddest thing is communication between different VLANs are ok, doesn't matter how nic-teaming or switches are setup. Only communication among the same VLAN using the 10G channel is giving me trouble.
Would this be something with QLOGIC CNA?
This is what I also tried earlier and failed to resolve the issue.
I create new vSwitch for one particular portgroup on 2 hosts, make port-group become Trunk/Tagged (means All VLAN accessible). Since I use VMXNET3 driver on the VM, i modify the NIC setting on each VM (windows OS) to tag the VLAN ID into it. The VM still didn't communicate with each other.
I'm pulling my hair out with this.
When you have two vmnics connected to the vSwitch, are these two from the two different fibre cables leaving the host?
Do you have one or two physical switches connected to the hosts?
Hi Rickard,
I purposely chose 2 vmnics that went out to 2 different fiber cable leaving the host. And these 2 fiber cables are connected to 2 different physical Force10 switches.
On the switches, I have RSTP enabled. The switches are interconnected using LACP.
All ports connected to VM Hosts and the SAN are independent from each other.
HieuPhan wrote:
Hi Rickard,
I purposely chose 2 vmnics that went out to 2 different fiber cable leaving the host. And these 2 fiber cables are connected to 2 different physical Force10 switches.
On the switches, I have RSTP enabled. The switches are interconnected using LACP.
All ports connected to VM Hosts and the SAN are independent from each other.
All this seems good. Could you however check the connection between these two switches and make sure it is not blocked by RSTP. This should be available through the switch CLI or web GUI.
Do the connection between these switches also have the specific VLANs?
Hi Rickard,
The LACP between the switches are up. All interfaces are up.
The connection between the switches are trunk, also are the connections between the switches and VM hosts.
The link might be up, but logically blocked by RSTP. Could you verify this?
Hi Rickard,
the link is not blocked by RSTP.
HieuPhan wrote:
Hi Rickard,
the link is not blocked by RSTP.
You have checked on both switches? If the link should be blocked by RSTP only one side will be put into the blocked mode, the other will look like it is up.
In order to configure RSTP your all switch to switch link is trunk mode.
Cheers, Udin