Equallogic MPIO MEM Released for ESX/ESXi 4.1 - Now Available

[s1xth]

I just wanted to make a quick post about this as I know a lot of people on the forums here have been waiting a long time for this.

There are a lot of items in the user guide that jump out at me that I am sure will be discussed heavily here as we all start deploying the MEM. For one, the user guide only recommends ONE vmk port PER pnic, NOT the previous configuration TR docs recommendng a 3-to-1 or more vmks per pnic. According to this doc that is NO longer required and if you are using the MEM it will actually ignore the extra vmk ports.

Well..I am excited to hear if anyone will be installing this, I stil havent upgraded my cluster to 4.1 yet and that isnt sheduled for at least another month. On top of that, lets not all forget the EQL FW 5.0.1 just went general release today also....:-)

Blog: www.virtualizationbuster.com

Twitter: s1xth

[DonDallas71]

Well, I upgraded my VMWare environment to 4.1 last week. I was excited to try the MEM and was in the process of installing it. I removed my extra VMK ports so I only have 1 VMK per nic like the doc suggests. I was installing it using the Update Manager process. Usually that process includes staging the update and then remediating, however once I staged the update it is no longer in the list of needed updates so I can not remediate it. I guess I am going to go through the manual install process now. If that works I may try just going direct to remediation and skipping the staging process on the next host.

[DonDallas71]

I did the manual install VIA CLI 4.1 and rebooted. Immediately after I was able to see 2 connections per slice. So with 2 arrays in my pool, 2 nics and 2 IPs I saw 4 connections (2 per IP). This is a big improvement over 1 connection per IP in the past.

I tried changing my settings with the setup.pl --setparam command but it just errors out with a "500 Server closed connection without sending any data back" message. I cannot even run a setup.pl --listparam command. However setup.pl --enable and setup.pl --query work, so I know my connection and settings are good.

I ended up just manually editing my /etc/cim/dell/ehcmd.conf file manually. and changing the volumesessions to 12 and membersessions to 4. After another reboot I now have 4 connections per IP (8 total), all active and all sharing the load. This translates to one connection per array interface which is what I am doing from all of my Windows servers as well.

I noticed that in the path management in vCenter the balaning method changed from "Round Robin (VMware)" to "DELL_PSP_EQL_ROUTED". I am assuming this is the new intelligent I/O, least queue depth method they talked about at the EQL user conference? Not really sure, but I hope so.

[zmclean]

We are using QLogic HBA's and the user guide indicates no performance increase by using this. So what is the advantage to doing this if you have Hardware HBA's?

Z-Bone

[DonDallas71]

I am thinking that the MEM only helps if you use the VMware iSCSI initiator to make your connections. If all of your connections are being made from your iSCSI HBAs then I don't see how it could change anything. Now if you are also using software iSCSI connections from within ESX in addition to iSCSI HBA connections then it would help.

It helps by allowing multiple connections from a single IP instead of the old way of needing additional IPs for every connection. The more connections you have to each volume, the better it will load balance throughput on the array interfaces. That keeps you from trying to pump too much data through individual array interfaces and creating a bottleneck.

Also, from what the Dell engineers at the EQL user conference told me... The new MEM uses intelligent I/O, meaning it will attempt to transfer data from a connection to the array where the data is actually stored. In the old round robin method it would just transfer the next chunk from the next connection so that all connections were used evenly. That causes a situation where a chunk of data might be requested from a connection to array-A when the actual data is on array-B (for volumes spanning multiple members). That means additional latency and also additional network utilization since array-A will have to go get the data off array-B itself and then send it to the server.

They also told me the new MEM uses least queue depth like is available in the Windows HIT. That means it will attempt to use the connections to interfaces that are the least used and prone to latency. That means faster responses.

You can usually tell if you are using round-robin as opposed to one of the better methods by looking at the connections to each volume on the array. If each connection shows almost the identical amount of data transfer then you are using round-robin. If the connections all have varying amounts of data transfered then you are likely using a more intelligent connection method.

Of course, this is all based on info Dell engineers told me at the conference. I have not seen any documentation yet that says this is actually happening (haven't looked either). However, since installing MEM on my ESX servers I now see varying amounts of data being transfered across multiple connections to my volumes instead of them all being almost identical. This tells me that the balancing method is more intelligent like what the Windows HIT uses and not just basic round-robin.

[ukdaz]

So can anyone confirm that what I'm seeing is correct with the new plug-in and 5.0.1 Firmware.

Setup

I have a standard ps6000 setup, with 2 dell 5424 switches. PE710 Hosts with 4 Nics for iScsi. 2 nics to each switch which are connected with 4 lags then each controller has 2 connections into each switch. Setup is as per the recommended setup for the switches.

Before the EQL Mpio plugin is installed I have 4 paths to each Lun as shown in the attached image using RR. Then when EQL plugin is installed using setup.pl after a restart I only get 2. What's interesting is that even if i try to change it back to RR(Vmware) I then only get 2 paths...not the 4 i had previously.

Thanks in advance.

[ukdaz]

I made the change detailed above to the ehcmd.conf and now have all my paths back....which is nice.

[DonDallas71]

I finished upgrading my entire environment to ESX 4.1 and all 3 of my arrays to 5.0.1 and installed the MPIO MEM. The day after I upgraded my last array I got an email from Dell saying not to upgrade my firmware, which I had already done. A few days later I took a 3 hour hard outage on my storage around 3-6am. The active controller on my primary member wigged out and my other member thought it had gone down so took all the volumes offline. It was not until 6am when the active controller finally failed and the secondary took over and everything came back up. I still have a ticket open with Dell and they are not sure yet, but think it has something to do with the new firmware. This is the first time we have ever taken down-time on any of the 10 arrays my organization owns.

With the new firmware, the hardware acceleration (VAAI) within VMware 4.1 gets enabled. This hardware acceleration deals with VMware passing over some data moving functions to the array. Dell suggested I disable these new features until they come out with 5.0.2, which is supposed to fix some of the big issues with 5.0.1. They think it will be available mid to late August. Once the acceleration features are disabled, the ESX servers have to be rebooted for it to take effect. If you go to your ESX configuration / Storage Adapters and view devices, if the Hardware Acceleration column says "Supported" then the feature is enabled. This doc will tell how to disable it:

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=102197...

According to Dell, you cannot downgrade the firmware without evacuating the array.

The good part is that the new MPIO MEM features are still working even with the hardware acceleration disabled. The two are not directly related.

Also, since upgrading my firmware I noticed my replication performance averages within SanHQ has changed drastically.

Average Before After

I/O Size 42.1 KB 256 KB

IOPS 358.8 59.8

Latency 3.9 ms 174.6 ms

I/O Rate 14.8 BM/sec 15.0 MB/sec

Before the upgrade, the average I/O size was extremely random. After the upgrade the average I/O size is now at a constant 256 kb. It looks like maybe the increased latency and decreased IOPS is due to the increased I/O size. The overall I/O rate is about the same. So this may not be an issue, but more of a change in how replication is done in the new version. I usually consider high latency an issue, but with constant 256kb I/O sizes it may not be avoidable. My link to our replication site is an OC3 with network acceleration. This latency only seems to be with replication. With replication disabled, my normal array functions have extremely low latency.

[glynnd1]

ukdaz,

The defaults with the EqualLogic MPIO plugin (MEM) do create two sessions per volume slice. As you have one member in your group it will therefore only create two sessions. When you add a second member to the group, the volume will spread across both members and the MEM will automatically create an additional two sessions to the slice of the volume that lives on the new member. Given that have have tweaked this setting from 2 to 4, when you add a new member to the pool it will now create 4 new session, not two).

I would be interested in understanding why you are using four NICs for iSCSI - are you consuming that much bandwidth?

David

[admin]

Anyone else have any additional feedback to post? We are wanting to get this deployed in our environment as well but waiting on view to support 4.1 before we can make the transition. I am curious if anyone has noticed a performance improvement either slight or drastic once mem has been setup.

[Yps]

5.0.2 is out!

[DonDallas71]

I would say that the performance improvement for me was slight. I already had 4 iSCSI connections per volume (2 per slice) prior to the MEM, which I doubled to 8 (4 per slice). I have a single pool with two PS6000XVs that is not too terribly loaded. I would say the main improvements are with configuration within ESX and the improved load balanacing since it is split accross more interfaces on the arrays. My ESX is now down to one IP per iSCSI interface within ESX instead of two IPs per interface. This is the same setup I have had within my windows VMs.

I do BMC collection on all my VMs and I have one in particular that uses a lot of virtual disk. Prior to the update I was seeing around 5ms "Disk Service Time - Lognest". Immediately after the MEM it dropped to less than half that. I also collect BMC data on ESX, but that already was showing almost no disk latency so there was no real change there. Most of my other heavy disk usage is on volumes that are connected within the windows VMs, so the MEM does nothing to improved those connections.

I am gussing that the improvement comes from the least queue depth method the MEM uses as well as spreading the load accross more array interfaces.

[DonDallas71]

I upgraded my 5.0.1 firmware to 5.0.2 on my disater recovery array on Tuesday and my production arrays on Wednesday. My only issue was with replicas. The release notes say they as long as you don't change any group or pool memberships your replicas will be fine. This was not the case with my arrays. I have always had one pool with the same two members since before 5.0.1 and first starting replicating. However, the upgrade process said my replicas were not good and requred me to redo full replicas on 4.5TB of data to my disaster recovery site. Not a very good upgrade experience even though everything else went fine.