Dell Compellent Firmware 6.7 and vSphere

[#1]

I'm currently running vSphere 5.5. Our storage runs on Dell Compellent SC8000 systems. We have 2 Storage Centers per geographical site.

Initially when setting up the ESXi hosts, I looked through the Compellent's VMWare Best Practices, and set the VMW_SATP_DEFAULT_AA rule to VMW_PSP_RR as per the document.

Then recently, as part of a project to consolidate all volumes on one Storage Center per site, we upgraded both SCs firmware in one site, and then both in the other site. We went from 6.5 to 6.7 on all.

The next step in the project was to create all new volumes on the SC that we were keeping, present them to VMWare and svMotion all our VMs. No problem!

After all that was done, in both our sites, I then moved on to update our hosts to 5.5 U3, and in the process of upgrading the first host and rebooting, I noticed that on that one host, all the datastores were now using VMW_SATP_ALUA, which uses the VMW_PSP_MRU instead of Round Robin as per the best practices. Uh? How can that be?

So I went and checked the other hosts that I hadn't updated or rebooted and here's what I found:

1. All volumes created prior to the Compellent's firmware upgrade, running on hosts that hadn't been rebooted were still using VMW_SATP_DEFAULT_AA and VMW_PSP_RR

2. All volumes created prior to the Compellent's firmware upgrade, but running on a host that had been rebooted, were using VMW_SATP_ALUA and VMW_PSP_MRU

3. All volumes created AFTER the Compellent's firmware upgrade were using VMW_SATP_ALUA and VMW_PSP_MRU

So I called Compellent Co-pilot, which wasn't too much help, but after digging, this is what we found out.

Compellent Storage Centers running firmware 6.6 and later, now include support for ALUA, which makes it so you have to change your VMW_SATP_ALUA rule to use VMW_PSP_RR by default. This is stated on Compellent's DellStorageCenterBPsWithVMware-vSphere5x-(680-041-020) document, page 38. It says:

Note: Because of the ALUA protocol addition to the Storage Center firmware, the vSphere SATP module settings, such as configuring round robin as the default PSP, will need to be changed on each host when upgrading to Storage Center firmware versions 6.6 and later.

I have a few issues with this:

1. This change was not mentioned anywhere on the new firmware release notes for either versions 6.6 or 6.7 - which is where people usually look for changes like this. It was listed on a best practices guide, that I don't really check once a system is setup. Maybe it's just me, and everybody else does. I don't know.

2. The Best Practices guide mentioned above is a bit confusing, at least to me, on page 38 it also says:

The ALUA protocol was designed for arrays that VMware classifies as Asymmetrical Storage Systems. Historically, since the Storage Center is considered an Active-Active storage system where all the paths are active at all times (unless a path failed), ALUA was not necessary.

However, with the introduction of the SAS protocol to the SCv2000 series arrays, ALUA became necessary for implementation into the front end target ports for controller failover purposes. For example, the path to the controller where the volume is active, ALUA will set the path state as active-optimized, and for the path to the secondary controller, ALUA will set the path state to standby. During a controller failover, the ALUA path state will change such that the secondary controller will become the active-optimized path.

To me, this means this applies to SCv2000 controllers and not to the one I have. But that's not what I'm seeing in practice.

3. Also, I find this part even more confusing:

It is important to note that for the other protocols that Storage Center supports, such as Fibre Channel, iSCSI, and FCoE, that the pathing architecture remains mostly unchanged. Consistent with the former architecture, all of a volume’s paths to the controller the volume is active on will be labeled active optimized. Since the Storage Center does not proxy volume traffic between controllers, the paths to the second controller will be set to the standby state, and only change during a controller failover.

So which is it? Then if that's the case, why the note right below it telling us to change the SATP settings to make sure ALUA uses RR?

4. The other issue I have with this is that if we had an actual Active/Passive array in our environment that we needed to use that ALUA with MRU rule for, that would be even more work with creating custom rules, etc, etc.

So keep this in mind if you're thinking of upgrading your Compellent to firmware 6.7 and you're running vSphere....

[gooseandbull]

We experienced the same issues at our site. Copilot stated we need to upgrade so they sent technicians onsite to prepare and execute the SAN OS firmware upgrade to 6.7.4. No one from Dell onsite bothered to check release notes nor any best practice updates about the significant changes from 6.6 or later now mentioned throughout their best practices for vsphere 5.x and 6.x. we spent almost 2 weeks trying to diagnose why a rebooted host would no longer communicate with datastores and a very lengthy boot time trying to discover LUNs witje the wrong prescribed multipathing commands. So frustrating. Phone support took almost 2 weeks prior to finally suggesting the ALUA and RR changes.

Glad to know we were not the only victims of such casual tomfoolery by support and faulty documentation.