OK, I know you didn't ask me this question, but I thought my experiences might be interesting to you. We have ESX servers running against several different storage arrays, a CX400 via FC, a CX700 via FC, a AX150i via iSCSI, and an Equalogic PS300E via iSCSI (and also a few systems running on local storage like IBM ServeRAID and Dell PERC controllers).

We've found the problem to be fairly easy to reproduce on the AX150i, as well as the CX700. For whatever reason it's actually more difficult to trigger it on the CX400, we theorize it has to do with the smaller write cache and thus lower latency during heavy writes but it may also be related to the fact that the CX400 simply has less contention because it services fewer hosts.

The Equalogic PS300E is by far the most difficult for us to reproduce the issue. We did mange to create the issue even when running against this array but it took a pretty crazy level of fake I/O load running in multiple VM's to do it.

On the other hand, if you pull the plug on a network cable, it's likely that you'll see the issue on every Linux guest by the time the iSCSI connection fails over to another port. My opinion is that this fix is an important proactive fix if you want your guest to continue running during any failure scenario, especially since that's usually the reason you invest in all of the redundancy.

Later,
Tom

The patch in question appears to be as follows:

--- linux-2.6.9/fs/ext3/dir.c~ 2007-01-08 16:19:36.000000000 -0500
+++ linux-2.6.9/fs/ext3/dir.c 2007-01-08 16:22:10.000000000 -0500
@@ -155,7 +155,7 @@ static int ext3_readdir(struct file * fi
brelse (tmp);
}
if (num) {
- ll_rw_block (READA, num, bha);
+ ll_rw_block (READ, num, bha);
for (i = 0; i < num; i++)
brelse (bha);
}

I'm not sure that this is enough to correct the issue in VMware because the Bugzilla seems to imply the issue was only seen in cases that were using PowerPath or dm-multipath. When using READA request under heavy load the bio request can sometimes return EAGAIN which is basically saying "I had a temporary error, please try again" but the ext3 code did not actually expect to handle this condition.

Redhat initially looked at backporting the code in more recent 2.6 kernels but this appeared to have more complex interactions and they opted for the simple, low-risk fix, which appears to basically forgoe readahead on directory lookups. It's possible that this might have a minor impact on performance, but probably not noticable in most workloads.

I'll be interested in seeing how your tests turn out so let us know.

Later,
Tom

So, as I suspected, the patch is question is simply not enough. Of interest however is that Redhat has commited a patch to the U5 stream which does include a supposed fix (and one that looks "sufficient" to me). Test kernels that include this fix are available at the following location:

http://people.redhat.com/~jbaron/rhel4/RPMS.kernel/

It would be interesting to see what you tests turn up with those kernels.

Later,
Tom

Hi,

VMware has updated the article:
http://kb.vmware.com/vmtnkb/search.do?cmd=displayKC&docType=kc&externalId=51306&sliceId=SAL_Public

Now a hotfix for SLES 9 SP3 is available.
Does anyone if this does work for SLES 10, too?

@doomdevice : Thanks for your work!
But ask that because we are looking for an offiicially supported solution.

Thanks in advance.
/egr

You mean reproduce the error? It's a little difficult to provide clear steps since it can be more difficult to trigger on some hardware, however, the basic formula in my case is, run a continuous loop of ' iozone -r 1m -s 1024m -t 8' in 2 VM's against a single VMFS LUN. Given enough time this has failed on every one of my SAN's without doing much of anything else, usually within just a few hours.

Damin seems to have a more formal test that he feels comfortable with on his setup.

Later,
Tom

Our company had a thorough discussion and meeting sessions setup with redhat on this 'readonly issue' not sure if its identical but we actually had a hot-fix and new kernel created from our testing. If you using RHEL your going to need to lay down 2.6.9-42.0.8 kernel. 42.0.3 is not sufficient enough. Bugzilla 213921 is a good reference to the trouble we had.

The bug fixed in 2.6.9-42.0.8, and documented in Bugzilla 213921, is an issue with ext3 and is not really related to VMware, although I suspect that VMware may make the problem slightly more likely to occur.

The "VMware" bug is an issue with the mptscsih driver, it's interaction with VMware server and how it reports timeouts to the SCSI midlayer in the guest.

While the failure mode of both bugs does cause a nearly indetical end result (ext3 filesystem going read-only) but key differences can be noted in the log files in most cases.

Later,
Tom