Dell R610 w/Broadcom 5709 and EqualLogic PS4000

[iPhrankie]

Hello Everyone,

I've done some digging to try to solve this one on my own. I have found some discussions regarding the 5709, but I need some additional clarity.

In our R610 we have the embedded Broadcom 5709 switch and a dual port Broadcom 5709 in the PCI slot. When configuring the Storage Adapters I was forced to add each NIC to VMkernel as Active and Unused. This was the only way it would allow me to enable iSCSI for the dual port 5709. I was then able to successfully connect to the SAN on each of the dual ports.

This has lead me to conclude:

1. There is no failover between the two iSCSI adapters assigned to each port. Each iSCSI adapter is connected to the SAN independent of one another.

2. Volumes are mounted on one or the other iSCSI adapter. It's up to me to distribute the volumes between each one to ensure either of the ports don't get overloaded.

3. Even though I have connected the R610 via multiple switches and multiple ports on the SAN, since volumes are connected on one or the other iSCSI adapters, if one of those two ports failed, those volumes would be unavailable depending on which port they were connected on.

I have attached our configuration that hopefully helps. Many thanks for any insight.

[dwilliam62]

Re: Failover / binding.   Not quite. The ESX initiator will create a session from each bound VMK port to the Group IP address.  It will then construct an MPIO SCSI device from those two disks.   By default,  VMware will use the FIXED Path Selection Policy (PSP).   In which only one port will be active.

There are other PSP algorithms available.  If you do not have an Enterprise or Enterprise+ license, then you will want to select the  VMware Round Robin option, under Configuration->Storage->Datastore->Properties->Manage Paths.  However, the default IOs per path is set to 1000 IOs.  So all paths won't be effectively used.  This needs to be changed to 3.

If you DO have Enterprise or Enterprise+ licensing, then download the Dell VMware MEM v1.1.1 module from the Equallogic Support site.  This will automatically configure optimal MPIO pathing on EQL volumes. 

This script will change all EQL volume to Round Robin and set the IOPs to 3.  You need to run this on ALL nodes and re-run it, when you add new volumes.

#esxcli storage nmp satp set --default-psp=VMW_PSP_RR --satp=VMW_SATP_EQL ; for i in `esxcli storage nmp device list | grep EQLOGIC|awk '{print $7}'|sed 's/(//g'|sed 's/)//g'` ; do esxcli storage nmp device set -d $i --psp=VMW_PSP_RR ; esxcli storage nmp psp roundrobin deviceconfig set -d $i -I 3 -t iops ; done

After you run the script you should verify that the changes took effect.
#esxcli storage nmp device list

This blog which has info agreed upon by Vmware, Dell/EQL, etc  has more info on this.

http://virtualgeek.typepad.com/virtual_geek/2009/09/a-multivendor-post-on-using-iscsi-with-vmware-vs...

You should also disable Delayed ACK and Large Recieve Offload (LRO)

How to Disable Delayed ACK

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=100259...

Solution Title	HOWTO: Disable Large Receive Offload (LRO) in ESX v4/v5
Solution Details	Within VMware, the following command will query the current LRO value. # esxcfg-advcfg -g /Net/TcpipDefLROEnabled To set the LRO value to zero (disabled): # esxcfg-advcfg -s 0 /Net/TcpipDefLROEnabled NOTE: a server reboot is required. Info on changing LRO in the Guest network. http://docwiki.cisco.com/wiki/Disable_LRO