Host with 10 NICS -- Best practices?

[dborgill]

We have new IBM x3950's 8 way, 32 gigs of RAM and 2 onboard NICS with 2 - 4 port PCI NICs.

We are using one onboard for the console, one for VMotion and the other 8 for the VMs. Is it best just to join those 8 together for the host to share for VMs? Our network guys are asking if they have to do anything on their end on the Cisco switches as far as teaming or to help performance.

Anyone have any suggestions or comments on how to best configure this?

[oreeh]

as always - it depends

As long as all your VMs are in the same network you can plug all NICs in the same vSwitch.

Your network guys don't have to configure any trunks or stuff like that on the switches.

With so many NICs available I'd rather use two NICs for vMotion and COS, since your guests probably don't need that much bandwidth.

[masaki]

teaming is handled by ESX you must only create virtual switches.

You could try with three vswitches.

One for console, one for vmotion and the last one for the vm.

I'd put one nic on the first, one or two in the second and 7-8 on the last.

[dborgill]

Thanks for the responses. So just to be clear, it isn't "true" NIC teaming where all NICs are combined into one pipe (Where configuration would have to be done on the Cisco switches also) is that correct?

Also, is it good to have two Vmotion ports for bandwidth or just redundancy?

On a side note, the reason we have so many NICs is because we can put a TON of servers on those boxes but they are network intensive. On all our existing ESX hosts, our CPU is hardly untouched but memory and network usage is pushed to the limit.

[esiebert7625 · Accepted solution]

It's basically NIC teaming, the below guides cover it pretty well. It does not have true load balancing between NICs. Once a VM uses a NIC it continues to use that one until there is a failure.

Networking Scenarios & Troubleshooting - http://download3.vmware.com/vmworld/2006/tac9689-b.pdf

Networking Virtual Machines - http://download3.vmware.com/vmworld/2006/TAC9689-A.pdf

NIC teaming and VLAN trunking - http://blog.scottlowe.org/2006/12/04/esx-server-nic-teaming-and-vlan-trunking/

http://www.vmware.com/pdf/esx3_vlan_wp.pdf

[masaki]

It's a true hardware fault tolerance and a powerful increase of bandwidth.

You shouldn't do anything on the Cisco Switches if you don't use vlan tagging in your network. It's all managed by ESX.