vSphere 4.1 w/ CNA & 10 GbE NIC's

[RKrulla20111014]

Working on a migration project that will use 2x Brocade 1020 CNA (Dual Port) to connect to a new Brocade 8000 switch for the storage network to use both NFS for the VM's and FCoE to Boot from SAN for the ESXi hosts.  We also have the legacy IP network that must be connected to through the Cisco 6509's which we plan on using 2x Dual Port10GbE NIC's for VM traffic.

Initially we plan on using a single port on each card for connectivity and the second port for future expansion.  The servers will be DELL R910's.

My question deals with the configuration maximums with vSphere 4.1.

http://www.vmware.com/pdf/vsphere4/r41/vsp_41_config_max.pdf

Broadcom 10Gb iSCSI HBA initiator ports per server - 4

bnx2x 10GB Ethernet ports (Broadcom) - 4

So if I have 4 CNA ports and then 4 NIC ports per host how are the 4 CNA ports used for IP traffic seen by vSphere?  Are they seen as more 10Gb Ethernet ports?  If so will I not be able to go with the dual port cards since the maximum for vsphere 4.1 is only 4 10Gb Ethernet ports for Broadcom?  Are these hard maximums or just what was tested?

Thank You!

[ats0401]

This design is not going to work and will give you all sorts of problems. I know this from personal experience as I have seen a customer attempt this before (not believing the maximums guide for some reason).

I would put two Brocade 1010 single port CNA cards in the servers and call it a day. 20GB should be plenty for storage and legacy IP traffic for the majority of the setups out there. What kind of bandwidth are you pushing?

With Network IOC, you can chop up the 20GB to support Fibre Channel, NFS, Vmotion, and VM IP traffic (setting limits and shares on each).

A single port brocade 1010 CNA will appear to the ESX server as a Fibre Channel storage adapter AND a 10GBE network adapter.

From the brocade 8000, you can uplink one of the CEE ports to the 6509 for the legacy IP network.

I assume you will be using 8GB FC to uplink to the SAN array for the ESXi hosts on the FC ports on the 8000.

If you insist on having an extra two 10GBE embedded nics on the servers, I would dedicate these to management and vMotion traffic.

NFS, FCOE, and legacy IP should all be on the CNA adapters.

However I would insist on only having the two single port CNA's if it were me. The point of a CNA (converged network adapter) is to

converge storage and IP onto the same card, reducing cabling and complexity. Ideally there will only be two cables from each R910

handling all storage and network needs.

The only time I think you should be using a dual port CNA is if you have a 1u server and only 1 available PCI slot.

Just one last thing to hammer across again - do NOT exceed 4 10GBe ports, it WILL cause you problems.

[FlorianLenz]

Please contact Brocade Support for the latest drivers! maybe these are not officially published yet. One of our customers had quite a lot of problems with loosing network connectivity with the CNAs (Storage connectivity was stable). At least brocade engineering considered, that there was a MAC adress mangling in their driver.

Another customer of me had similar problems with a 10GB network adapter from HP (NC522).

Good Luck,

Florian