NIC Teaming for HS21 Blade with 2 Cisco IGESMs

[squeeknet]

I need some help determing how to configure NIC Teaming in ESX 3.0.1 for a network upgrade here shortly.

Right now, I have two vSwitches: vSwitch0 and vSwitch1, each have one NIC, vmnic0 and vmnic1, respectively. Each NIC is an different Interface on my two seperate Cisco IGESMs in the IBM BladeCenter.

I want to be able to have one vSwitch, with two NICs that connect to 2 different Cisco IGESMs for redundancy/fault-tolerance.

My questions are:

1. Is this possible?

2. What is best practice for this?

3. How do I do it without creating any loops in my network?

I am not the ESX Server administrator, merely a network guy trying to configure ESX without any training on the product.

Any and all help is appreciated.

Thanks!!!

[nayudu]

Hi,

You can do this.

Go to vSwitch1 click on edit settings and unlink the physical adapter associated with it (vmnic1)

Now go to vSwitch 0 and editsettings and go to adapters and add the second adapter to it.(vmnic1)

The moment you add a more than one or more nic's to a vSwitch they automatically fall in to a 802.3 ad Team.

I dont think you will have any loop issues by doing this.

Regards

Kishore

[squeeknet]

Thanks for the reply. Got the first part but....

Isn't it impossible to have the NICs fall into an 802.3ad link to two seperate physical switches?

[NASFORCE]

No. That would require you to be into one switch. etherchanneling or lacp is not supported over two switches.

[Monoman]

You typically need to plug into the same switch. You can get around this limitiation if you use switches that stack together and are configured as one large switch ... like the Cisco 3750. I don't believe the Cisco IGESMs stack.

[squeeknet]

So basically I have two options:

1. Have NIC1 as a failover and continue to operate with all my VM's going through NIC0.

or

2. Load balance VM's between NIC0 and NIC1, so that lets say, 3 VMs through NIC0 and 3 VMs through NIC1. and if one link fails, set the failover so all VMs remain operational.

I'm leaning towards option #2, unless it can't be done....

Also, is there going to be any problem with vMotion if I have one ESX server with 1 NIC and another ESX server with 2 NICs?

[Monoman · Accepted solution]

We have a couple of blades with the two NIC limitation. We have them both configured as trunk ports in the switch. Both NICs are assigned to vSwitch0 and are both active. vSwitch0 contains the console port, vmkernel (vmotion) port, and all of the port groups (VLANs) for VMs. The load balancing is set to "Route based on the originating virtual port ID". This configuration is far from ideal but it was the recommended setup.

As far as Vmotion is concerned. It should work if the CPUs match and the networks match. I don't think it will care how you have the NICs assigned as long as you have the same networks created with the same names. You defintely want to test first.

Personally, I would want the server setups to match to keep it simple.

[RyanWI]

1. Yes it is possible. Not suggested, but possible. Have you looked into getting a MSIM and adding additonal networking to the 21? Also, if you're stuck on blades. Look at the LS41, its' a much more robust esx host.

2. Best practice is to use one vSwitch. Bond both NICs into the vSwitch. You MUST use PortID load balancing because CISCO does not support LACP accross different physical switches (excluding the 6000 series stack switches). Also, just as an FYI, VMware does not support Dynamic LACP. If you were going to do an etherchannel, it needs to be a static setup. Anyhow, I'm assuming you are using vlan tagging. There are three things you should do if you are limited with that network setup:

a. Use traffic shaping to limit the port group for VMotion traffic. Throttle that bandwidth so that it doesn't affect your VMs or service console.

b. Use different networks for VMotion and Service console. Considering spanning tree, you should be able to find a design that puts the SVC console using one NIC and VMotion using the other by default and in a outage, they will migrate together.

c. Think about setting your SVC pg to rolling to stop HA issues on rollback. Guess that depends if you use HA or not. But it's important for the SVC to be available all the time of course. There is also a Too Fast RARP issue with ESX 3.x that can impact the service console on failback. Look for esx 3.0.3 for a fix. (HOPEFULLY!).

3. Loops? What loops? It's tough to say exactly since I'm not sure of your network config or STP config. Basically tell your ESX admin, to not add two nics to a VM using two port groups and manually bond them together. What you wouldn't do on a physical anyhow, i'd imagine.