Utilization estimates network and FCoE

[vmproteau]

We have a greenfield vSphere 5.1 U1 environment comprising of Hosts with 4 socket (8-core) processors and 384GB Memory. Network is composed of 4-10GbE CNA that will pass VM network and FCoE storage traffic to Nexus 5K switches.

There is discussion of the Cisco 1000v because they want to implement LACP and VMware's 5.1 vDS won't allow for more than 1-LACP group per Host. While turning this over I backed up a few steps and posed some questions that I needed assitiance answering. I do understand the other benefits to the Cisco 1000v with respect aligning roles and responsibilities with the Network Team, QoS, and other advanced features but, wanted to focus these questions specifically to LACP and LB, in this particular environment.

1. The most basic question is about LACP benefits in such an environment. I understand workload is a key factor but, considering the size of these Hosts, does anyone has insight or estimates on expected utilization ranges in average or high workload environments? My uninformed guess would be that we wouldn't come close to utilizing the 4-10GbE links even with FCoE.
2. Related to (1), if we wouldn't come close to utilizing these links, perhaps MAC pinning or some other native vDS LB would be sufficient?
3. Someone had also mentioned to me that Cisco was moving away LACP as a best practice for 1000v in recommending MAC pinning or IP Pinning. I have not read that anywhere so, wondered if anyone else had heard this?

[MKguy]

VMware's 5.1 vDS won't allow for more than 1-LACP group per Host.

To be precise, the limit is one LACP group per vDS. So you could just use multiple vDS each with their own LACP groups on the same hosts:

http://www.vmware.com/pdf/vsphere5/r51/vsphere-51-configuration-maximums.pdf

1. The most basic question is about LACP benefits in such an environment. I understand workload is a key factor but, considering the size of these Hosts, does anyone has insight or estimates on expected utilization ranges in average or high workload environments? My uninformed guess would be that we wouldn't come close to utilizing the 4-10GbE links even with FCoE.

As you said yourself, workload is key and you can't discern anything about the possible network utilization just by looking at the hardware specs. Even if you max out your hosts with CPU and memory intensive applications, this does not necessarily correlate to a lot of required network bandwidth (or IO) at all. As usual, it depends.

2. Related to (1), if we wouldn't come close to utilizing these links, perhaps MAC pinning or some other native vDS LB would be sufficient?

Considering the main benefit of LACP in this case, a single VM being able to utilize more bandwidth than a single 10Gb uplink can provide, do you really think that there will be such a case and that this would be critical? If not, then I think you don't need to insist on shiny but going-to-be-heavily-underutilized-LACP. I personally would go with the Load-based teaming approach of the default vDS, providing dynamic load balancing of vNICs to uplinks.

3. Someone had also mentioned to me that Cisco was moving away LACP as a best practice for 1000v in recommending MAC pinning or IP Pinning. I have not read that anywhere so, wondered if anyone else had heard this?

That's news to me too.

[vmproteau]

Thank you for your reply. I knew about the 1 LACP group per vDS limitation but, I've also read in many places including KB2034277 about a 1 per Host limitation. It is pretty important to our decisions going foward so, can you verify (see first bullet point)?

LACP limitations on a vSphere Distributed Switch

• vSphere supports only one LACP group (Uplink Port Group with LACP enabled) per distributed switch and only one LACP group per host 
• LACP does not support Port mirroring 
• LACP settings do not exist in host profiles 
• LACP only works with IP Hash load balancing and Link Status Network failover detection 
• LACP between two nested ESXi hosts is not possible

[MKguy]

I'm afraid I can't really anything, but looking at that KB article, it sure looks like only one group per host is supported unfortunately (whether it works could be a different story though).

I only checked the maxium configs guide which only mentioned the limit per vDS.

[vmproteau]

Understood. I will validate what is "supported" we won't implement it if it's not officially supported. Thanks.