KB 1008130: VMware ESX and ESXi 3.5 U3 I/O failure on SAN LUN(s) and LUN queue is blocked indefinitely

[vmproteau]

Does anyone have additional information on the current support alert "All those who have upgraded to, or are planning to upgrade to, ESX 3.5 Update 3 or ESXi 3.5 Update 3, please read KB Article 1008130" ?

The KB describes the possible symptoms and identifies the affected version ESX 3.5 U3 but the resolution is a bit vague. I understand they are working on a fix but,

1. Is it related to a specific U3 build?

2. Are there precautions we can take to minimize the risk?

3. How widespread is this? My assumption is not very since 3.5 U3 has been out a while and as far as I know this support alert is recent.

4. Anyone experence this first hand...to what level....were you told anything more than is stated in the KB?

I haven't experienced any symptoms on the Hosts I have at 3.5 U3 but, I have scheduled upgrades to several Hosts this weekend and want to know if I need to go throught the painful process of cancelling and rescheduling.

[BenConrad]

The KB says:

Trigger: This occurs when VMFS3 metadata updates are being done at the same time failover to an alternate path occurs for the LUN on which the VMFS3 volume resides

KB 1005009 says:

The following are examples of VMFS operations that require metadata updates:

• Creating or deleting a VMFS datastore

• Expanding a VMFS datastore onto additional extents

• Powering on or off a virtual machine

• Acquiring or releasing a lock on a file

• Creating or deleting a file

• Creating a template

• Deploying a virtual machine from a template

• Creating a new virtual machine

• Migrating a virtual machine with VMotion

• Growing a file, for example, a Snapshot file or a thin provisioned Virtual Disk

With all of those possible operations it certainly possible a path failover might happen while performing one of these operations. But the KB article you reference says this "can" happen, so that doesn't mean it will happen.... YMMV

Ben

[vmproteau]

Thanks. It's my understanding that there is no HBA load balancing that occurs in ESX 3.5 U3 so, an HBA failover event would have to occur for this to even be a possibility. Is this correct? If so, I'm a bit more at ease about our existing environment as an HBA failover event is uncommon.

The only problem I have now is I will also be replacing the redundant SAN switches these Hosts connect to. When this is done, there will absolutely be a failover of all LUNs which makes me more susceptible to the problems outlined in KB 1008130. It seems like the all the listed operations that VMFS metadata operations occur I can control with the possible exception of "Acquiring or releasing a lock on a file".

Do you know if this affects only the Host where the failover event happens? Can it also affect other Hosts in the Cluster that are sharing this LUN although haven't experienced a failover event themselves?

[BenConrad · Accepted solution]

3.5 has experimental round-robin load balancing but that would not count as a path failover (I don't think).

You could force the LUNs to go through a certain path, it might take a few attempts on busy LUNs because the load balancing can't be changed when an IO is queued (this is how HW iCSSI works). When you fail the switches that still counts as a path failover so I'm not sure how that comes into play re: this issue when no LUNs are using that path.

To be safe maybe you put a hold on:

• DRS

• SSH logins

• Backups

• Restores

• Clones

• power on/off

• etc, etc, etc

while the path failovers are happening.

If a path fails on one host I'd be suprised if it affects other hosts. Hosts will reserve the file on the VMFS file system when they need to update metadata or write data, I think that should continue to work. However, if you are taking down a switch you may have path failures on all of your hosts.....

Ben

[vmproteau]

The reason I ask about this was because I planned to minimize SAN connectivity disruption by doing the following:

Consider a 3 Host Cluster (ESXHOST-A, ESXHOST-B, and ESXHOST-C). Each has 2-HBAs each connected to a seperate physical switch. Switch-1 will be replaced on Saturday, Switch-2 replaced on Sunday.

1. VMotion all VMs off of ESXHOST-A

2. Force all LUNS over the HBA connected to Switch-2.

3. VMotion all VMs off of ESXHOST-B

4. Force all LUNS over the HBA connected to Switch-2.

5. VMotion all VMs off of ESXHOST-C

6. Force all LUNS over the HBA connected to Switch-2.

7. All SAN traffic for the Cluster should now be travelling over Switch-2.

8. Replace Switch-1

This way I'll ensure no ESX Host is communicating over the Switch as I'm replacing it. Worst case, as I manually force path failover and have one of the outlined issues, it only happens to a Host with no active VMs. That was why I wanted to ensure that it only affects the Host that is actually having the LUN failover.

I have a support request in but, thanks for commenting on this. I often find more applicable responses withing the communities.

[jbrauer]

If you have a three node cluster, you should easily be able to do one host at a time...

put one host in maintenance mode and swing the cables, rescan/reboot verify connectivity, exit maintenance mode, rinse repeat..

[Petter_Lindgren]

I'm gonna install two new servers with ESX 3.5 in the next days.

What do you think about this issue? Should I go with U2 or U3?

/Petter

[vmproteau]

I assume you're attaching these to shared storage. That's the only reason to be concerned about this KB. If you are and are specifically concerened about this problem, going with U2 is the only way to ensure it doesn't happen.

There are improvements and bug fixes in U3 so, you'll need to weigh the risks of going with U2.

Either way you'll likely have to patch these early February or so when the next round of patches are released. Among them will be a fix for the issues described in KB1008130.

[jbrauer]

Petter,

Unless you are experiencing problems on an existing server/cluster now, I do not see a reason not to go with U3.

I am pretty sure I have my problem isolated to a VCB issue. Well not really VCB itself, but one that VCB is helping to exploit and cause the ESX servers to think there is a problem with the SAN. I would expect VMware to have a patch out for this problem very soon...

[Petter_Lindgren]

Thanks for the replies.

I'll go with U3 and make myself ready to patch them in the nextcoming weeks...

[jpvlsmv]

We were bit by this two days after the KB article was posted (we had a long-scheduled SAN upgrade that weekend).

The workaround in the KB was really not a good one, since when the ESX host is disconnected in VC, you can't do vmotion to get the perfectly healthy VMs off so you can reboot the ESX host.

However, I was able to get the host connected again to virtualcenter by stopping the mgmt-vmware and vmware-vpxa services on the ESX host and restarting them. Then it showed up as connected, and I was able to use VMotion.

This prevented me having to send out a "yeah, you know how I said the downtime was over, well I lied" email.

--Joe