henry857
Contributor
Contributor

Load Based Teaming utilizing FCoE

Hi everyone,

We are getting ready to install two Qlogic CNA's into our hosts and I really want to move away from IP Hash and move towards LBT because I like what it brings to the table. My one concern is that LBT isn't FCoE aware. So when LBT is looking to see which uplink is saturated, it is not getting the full picture. What can I do in my setup to address these concerns? Does LBT work well even while being oblivious to FCoE?

0 Kudos
10 Replies
henry857
Contributor
Contributor

LBT looks at pNIC stats, but only pNIC stats generated by vDS, correct?

0 Kudos
MKguy
Virtuoso
Virtuoso

If you configure the CNA as a native hardware FCoE initiator, then it should show up as a regular storage controller device (vmhba) on your host straight away and not as a vmnic NIC device, so naturally any kind of vSwitch NIC teaming policy is out because it's simply irrelevant and you would use multipathing. If you carve up the CNA into

Are you planning to run ESXi software-based FCoE with the CNA presented in the network NIC function? In that case teaming policies are also irrelevant:

https://blogs.vmware.com/vsphere/2011/12/vmwares-software-fcoe-adapter.html

This is an important point to note – the FCoE traffic does not go through the vSwitch. However we need the vSwitch binding so that the frames will be forwarded to the dcbd in userworld through the CDP VMkernel module for DCBx negotiation.  The vSwitch is NOT for FCoE data traffic.

Other useful links on the topic:

http://blog.scottlowe.org/2011/11/29/setting-up-software-fcoe-on-vsphere-5/

http://pubs.vmware.com/vsphere-50/index.jsp?topic=%2Fcom.vmware.vsphere.storage.doc_50%2FGUID-A9503E...

-- http://alpacapowered.wordpress.com
0 Kudos
henry857
Contributor
Contributor

MKguy,

I'm not sure I understand.  I will be using a Hardware FCoE Adapter (CNA), so I should see both vmnic and vmhba adapters. Everything I read about LBT states that it checks in 30 second intervals to see if the pNIC is 75% saturated or more, and then it moves it to another uplink. If SAN traffic is using more than 25% utilization on a pNIC then LBT switch over may never be triggered since LBT doesn't see SAN FCoE traffic. So essentially you're left with route based on virtual port ID without the benefits of LBT. Which makes me want to reconsider sticking with IP Hash.

0 Kudos
henry857
Contributor
Contributor

There has to be something I'm missing though. It seems that, from reading sources on the internet, 2 x 10GB setup running vMotion, VM Network, and SAN FCoE traffic is not that uncommon with LBT. I just don't see how LBT can be effective if the utilization stats that it's reading on a 10Gb connection isn't accurate since FCoE shares the same pipe.

0 Kudos
MKguy
Virtuoso
Virtuoso

If you carve the CNA  into multiple functions like one for hardware FCoE and one for normal Ethernet, then sure, you should see a storage controller vmba device and a network vmnic device. But these are completely independent from each other, being limited to link bandwidth you configured in the CNA.

Your FCoE traffic does simply not travel through the vmnic device which is purely for normal networking, so there is no teaming policy/vswitch or whatever to configure from the FCoE point of view. (Again, software-based FCoE is a different story). A vSwitch distributes traffic with LBT or whatever policy according to virtual VM vNICs or ESXi vmkernel ports. A hardware FCoE HBA does not depend on or can be linked to a vmkernel port.

You can use LBT for your network function NIC devices but this has no effect or implications on the hardware FCoE HBA device which operates on it's own.

-- http://alpacapowered.wordpress.com
0 Kudos
henry857
Contributor
Contributor

Ok. That makes sense. So once carved, let's say 6Gb ethernet/4Gb FCoe, vSphere should recognize the vmnic as a 6Gb uplink, rather than a 10Gb uplink? WIthout carving, the CNA card will simply show up as a vmhba OR a vmnic? If vSphere still sees it as a 10Gb uplink then LBT is still broken.

0 Kudos
henry857
Contributor
Contributor

I just checked a host we are currently testing this on. It clearly shows 2 vmhba's and 2 vmnic's, with the vmnics being 10Gbe capable. So if carving was done, it is not reflected in the speed that it supports.

0 Kudos
MKguy
Virtuoso
Virtuoso

So once carved, let's say 6Gb ethernet/4Gb FCoe, vSphere should recognize the vmnic as a 6Gb uplink, rather than a 10Gb uplink?

It might depend on the CNA/driver, but yeah it should show up with the custom bandwidth you configured. From experience I can say that at least Emulex-based HP Flex-Fabric CNAs work that way.

WIthout carving, the CNA card will simply show up as a vmhba OR a vmnic?

Yeah, "without carving" would mean you just pick a single role and assign all bandwidth. Carving up the CNA is all about assigning roles and portions of the available bandwidth. It might default to a single Ethernet-only 10Gbe link vmnic device if you don't change the default configuration.

Also see this thread: https://communities.vmware.com/thread/461952

-- http://alpacapowered.wordpress.com
0 Kudos
MKguy
Virtuoso
Virtuoso

It's possible that the CNA always presents it's builtin HBA device to the OS, independent of whether bandwidth was actually assigned for FCoE or not. Check the CNA configuration.

You might be able to display the HBA link speed with esxcli commands such as esxcli storage san fcoe list, esxcli fcoe nic list, esxcli fcoe adapter list from the ESXi shell, but then again I'm not sure if link speed values displayed there are abstracted accurately.

-- http://alpacapowered.wordpress.com
0 Kudos
henry857
Contributor
Contributor

Thanks.

0 Kudos