Best practice deployment with 4 10gb nic per host

[christg]

Hi, we play to deploy new VMware esxi 5 server. i want to know what are the best practice and recommendation for the network configuration.

We are using 2x dual port 10gb fcoe adapter. our backbone is 2 nexus 5000 converged switch. the nexus switch is connect to our netapp san using fc redundant link. Each nic have a path to each nexus 5000 switch.

Nic 1 to nexus switch a, nic 2 to Nexus switch b, nic 3 to nexus  switch a, nic 4 to nexus switch b.

Adapter1 port 1 = nic1, Adapter1 port 2 = nic2, Adapter2 port 1 = nic3, Adapter2 port 2 = nic4

i was planning

VMKERNEL DVS binding to nic 1 and 3

management Vmkernel port active nic 1 and 3

vmotion vmkernel port active nic 1 passive nic 3

vmotion vmkernel port active nic 3 passive nic 1

(Dual nic vmotion configuration)

VMNetworkDVS binding to nic 2  and 4

Vmnetwork port  active nic 2 and 4

Fcoe LUN acces  type: round robin  mode: (alua)  using nic 2 and 4

i want my configuration to be reliable flexible and powerful.

Should we have advantage using port channel  ?

Is there a better configuration  ?

Thank Christian Gélinas

Vmware administrator from Ville de Quebec

2 cluster  500 server vm

[logiboy123]

See the following two articles by the VMware Networking blog;

http://blogs.vmware.com/networking/2011/12/vds-best-practices-rack-server-deployment-with-two-10-gig...

http://blogs.vmware.com/networking/2011/12/vds-best-practices-blade-server-deployments.html

Also you should have a look at the NIOC and SIOC documentation;

http://www.vmware.com/files/pdf/techpaper/VMW_Netioc_BestPractices.pdf

http://www.vmware.com/files/pdf/techpaper/VMW-vSphere41-SIOC.pdf

Cheers,

Paul

[VirtuallyMikeB]

Good day,

It sounds like you have a lot of nice equipment.  Those Nexus switches are expensive, too.  I suggest using features unique to them, like virtual port channels (vPC).  You can get started with them here:

http://www.cisco.com/en/US/prod/collateral/switches/ps9441/ps9670/configuration_guide_c07-543563.htm...

Cheers,

Mike

http://VirtuallyMikeBrown.com

https://twitter.com/#!/VirtuallyMikeB

http://LinkedIn.com/in/michaelbbrown

[christg]

We finally ended up with a simple configuration

We are not planing to use cisco port channel since using port channel configuration demand cpu to the esx host to process each outgoing packet.

DVS01 binding to nic 1,2,4 and 3

Portgroup 1, vlan 15, route based on originating virtual port/link status only

vmkernel port for mannagement binfing on portgroup 1

Portgroup 2, vlan 1001, route based on originating virtual port/link status only

vmkernel port vmotion binding on portgroup 2

Fcoe LUN acces  type: round robin  mode: (alua)  using nic 1,2,3 and 4

[esxguide]

Can you post your Nexus configuration? I think that is the place where most of these scenarios have problems. I'm curious if you can max out an HP blade server with 6 10GbE ports (2 dual port mezzanine cards), and use that fabric extender interconnect module, I think it's a B22HP, but basically a cisco nexus 2000 in an interconnect form factor.

Could you go with a DVS with 6 uplinks? Then vMotion, Fault Tolarance, and VM data could all spread out over the 6 ports dynamically.

So if you don't use port channels, do you use load balancing based on Physical Nic Load?

[christg]

i check with the nexus administrator for the nexus configuration.  i will post it soon.

You can go with a DVS with 6 uplinks for vMotion, Fault Tolerance, and VM data. but by default vmware will not dynamically balance the load. basically VMware esxi host will use random available uplink to communicate. for vm each is bind on a upling when boot, for vmotion it will use one availible uplink , same for management.

i do not use port channel because the only supported balancing configuration is route based on ip hash, so se each communication based on source/destination will be balances through each uplink. The host have to handle each session this have bad effect on overall cpu use. also since  vmotion will communicate with only one destination address, the vmotion traffic will use 1 uplink  for each destination host.

I prefer using destination based on originating virtual port to offload the host cpu, and then use traffic shaping to reduce VMotion burst or management traffic quality if needed .

[esxguide]

I agree, I DO NOT want to use port channels. So does that mean you don't have a vPC between the Nexus switches?

Load Balancing

Are you saying that if you have 4 links you can load balance, but if you have 6 it is only random selection?

In your scenario, could you change to Load Balanced based on Physical Nic Load? According to VMware, it checks Every 30 sec for load. But I can see where even that would cause some CPU resources to be used.

And on top of that you have FCOE working? Nice job!!! Congratulations!

[christg]

i am saying no matter how many link you are using over 2. that VMware will balance depending of the policy you chose. i say i use the policy:route based on originating virtual port policy. Each policy have some benefit and some disadvantage.

Route base on physical nic load is a very good and simple policy to dynamicaly balance

Benefit: light load on host cpu, easy to configure ( no switch configuration), will use most of all nic

Disadvantage: will not prevent a very ungry vm to saturate a link up to 30 sec ( mostly impossible with 10g nic), if you have application server that are packet sensitive,  client can report problem since the vm can move dynamicaly over multiple nic.

i use route based on originating virtual port because it is the most reliable. i monitor the vnic usage and corect ungry vm when occur.

[esxguide]

Hey Christian,

I have two more questiona. What CNA are you using? Emulex, Broadcomm, or QLogic? And are you using HP blades?

Thanks!