LUN thrashing with new 3.5 host

[ajskalski]

We recently added a third host to our cluster. The two existing hosts (vmc1, vmc2) run 3.0.2. The new addition is running 3.5. Our upgrade of the two existing hosts is on hold while we resolve the following.

Each host has a pair of HBAs, each of which connects to a Brocade SW5000 (two switches, separate fabrics). A STK Flex240 SAN presents LUN0 to a host group comprised of the three VMWare hosts (luns are mapped consistenyl on all three hosts). The LUN is owned by SPA and is set to MRU on all three hosts. The hosts are set to the host type "Linux" (which has AVT enabled). Anyone using a different host type for VMWare?

When we boot the 3.5 host, the other two hosts see "FAStT SAN is path thrashing with another system. Check AVT setting." in vmkwarning. On the 3.5 host we see:

root@vmc3# esxcfg-mpath -l

Disk vmhba1:1:0 /dev/sdc (697509MB) has 4 paths and policy of Most Recently Used

FC 10:0.0 2100001b320d42ea<->200700a0b81612e2 vmhba1:1:0 On active

FC 10:0.0 2100001b320d42ea<->200600a0b81612e2 vmhba1:0:0 On preferred

FC 10:0.1 2101001b322d42ea<->200600a0b81612e3 vmhba2:0:0 On

FC 10:0.1 2101001b322d42ea<->200700a0b81612e3 vmhba2:1:0 On

WWN 200600a0b81612e2 is SPA

WWN 200700a0b81612e2 is SPB

On the two other hosts, things are different:

root@vmc1# esxcfg-mpath -l

Disk vmhba1:0:0 /dev/sdb (697509MB) has 4 paths and policy of Most Recently Used

FC 14:0.0 210000e08b9d3d31<->200600a0b81612e2 vmhba1:0:0 On active preferred

FC 14:0.0 210000e08b9d3d31<->200700a0b81612e2 vmhba1:1:0 On

FC 14:0.1 210100e08bbd3d31<->200600a0b81612e3 vmhba2:0:0 On

FC 14:0.1 210100e08bbd3d31<->200700a0b81612e3 vmhba2:1:0 On

As best we can tell, the problem is that vmc3 sets the active path through SPB. We can workaround this by setting the policy to Fixed, picking our desired preferred path, and switching back to MRU. However, this change does not persist across reboots.

Any thoughts as to why vmc3 picks the wrong path? I thought VMWare always picked the lowest adapter/controller combination to be the canonnical path to the LUN. Is there a way to correct this behavior?

Our setup also seems to incorrectly show all paths as "On" though technically the Flex240 is an active/passive SAN. I understand this to be a side-effect of AVT being enabled. We have a call into STK/Sun support to determine if there is a better host type to pick (there is no LNXCL host ype like on the similar IBM hardware).

Thanks!

[mike_laspina]

Hello,

You should only have 1 path to each SP.

Your esxcfg-mpath -l output indicates 4 paths. Do you have 4 SP's?

Your fabric may not be zoned correctly.

[wally]

There's nothing wrong with the 4 paths, when using 2 HBA's and 2 SP's (and probably 2 switches) it's rather normal that you have 4 paths to each lun. This way you can have a HBA, Switch or SP failure without problems.

[ajskalski]

There are only two SPs. The zoning follows best practices:each host has two hbas; each hba connects to one switch; each SP connects both switches. This has always resulted in four paths in the past (i.e. the two existing 3.0.2 hosts).

Besides, even if I disconnect hba2 (and get to two paths, one via each SP) the 3.5 host still picks the active path through the standby SP.

ajs

[mike_laspina]

I don't see how this is a best practice?

Reguardless of how many logical paths you create you still have only two physical SP's.

You are only increasing the number of problems that can occur.

I zone my switches so that the logical path is 2 and the physical fabric can have 2 or 4 for all I care, it will take care of that part.

My hosts only need to see 2 and I have never had pathing problems.

[christianZ]

Well in "San config guide" (page 64) I can see "disable AVT" not enable - maybe that your issue.

And remember the connection rule (SP1 always connected to a lower switch port no than SP2/page 62).

[depping]

Mike.

If you set it up with a redundant fc-switch, of which each has a link to each SP you will have 4 paths. this is the most common setup used for new implementations.

Duncan

My virtualisation blog:

[ajskalski]

I know we need to disable AVT, I am waiting to hear from STK/Sun or others here on the best way to do this (I can use another host type with AVT disabled, but I am unsure if this change would have other impacts, in case the host type encapsulates settings in addition AVT)

I've read the line about the first SP using a lower numbered port a few times and we are not following this guideline: SPB is plugged into lower-numbered ports on both switches. In our defense, it's been this way since before VMWare, and we'll need to schedule downtime to change it (due to some single-pathed hosts).

So, this is likely the reason why our new 3.5 host sees the storage via SPB first? Why wasn't this an issue with 3.0.2?

[icontrolyourpow]

ESX is very sensitive to which port on each SP its HBA's are zoned. If not discovered properly, you will indeed experience thrashing in an active/passive environment that uses an AVT type process. I believe this is covered in the SAN Configuration Guide. On FastT storage, AVT is disabled for ESX by selecting LINUXCL as the host type. I believe this is also covered in the SAN Configuration Guide. The SAN System Design and Deployment Guide addresses these issues as well.

In addition to the guides mentioned above, there was a presentation at TSX last year which covered these topics in some detail. It can be found here:

I don't see how this is a best practice?

Reguardless of how many logical paths you create you still have only two physical SP's.

You are only increasing the number of problems that can occur.

I zone my switches so that the logical path is 2 and the physical fabric can have 2 or 4 for all I care, it will take care of that part.

My hosts only need to see 2 and I have never had pathing problems.

If multiple ESX hosts, each with multiple HBA's, are zoned to an active/passive SAN with each HBA only seeing one storage processor, a path failure (for any reason) on a single host will force a trespass of the LUN to the SP to which the other HBA is zoned, which will cause a failover on all hosts accessing that LUN. So instead of a quick HBA failover causing a brief delay in I/O for VM's running on one host, you get a longer delay for all VM's on that LUN on every host running them.

[TomHowarth]

for a start the StorageTek Flex240 does not seem to be on the HCL, that could be one of the reasons your are reveiving issues. read for further infomation,

Tom Howarth

VMware Communities User Moderator

[mike_laspina]

Thanks,

I can now wrap my head around why it would reduce the possible instances of path/SP interruption for the majority of the hosts.

This method does have a down side when a host HBA failure degrades both of the SP's at the same time, but it's not a high probability.

I have seen many unstable configurations at 4 paths, mostly do to human error. This is why I suggest a simpler approach. There is less to go wrong with it.

As well I have hit the maximum LUN assignment limit before with 4 paths.

It consumes twice as many available LUN assignments so when the host farm has more than 16 instances on an array you can not add more.(SAN Model specific of course)