VMware Communities: Message List - ESX 3.0.1 - Linux Guests go ReadOnly

Re: ESX 3.0.1 - Linux Guests go ReadOnly

Damin — Sun, 09 Mar 2008 06:19:34 GMT

Well.. now I'm posting again to the thread that just will not die.

I recently switched from running our production SAN on IET (Iscsi Enterprise Target) to the commercial Wasabi Systems iSCSI target. My production VMs are a handful of Centos 4.6 boxes, running the latest 2.6.9-67.0.4.EL kernels. It was my understanding that this contained patches that would prevent the ext3_fs aborts and resulting read-only issues.

I reviewed the code in the centos repository and saw that the Fusion Drivers had been updated to version "3.02.99.00rh". This appears to include the changes to the SCSI mid-layer to support conditional requeue of the timeouts. This patch appears to have gone mainline here: http://www.mail-archive.com/linux-scsi@vger.kernel.org/msg05338.html

After making the change to the new SAN, and updating the kernels on my Centos box, all looked good a while. Then we started getting 5-6 instances of boxes going ReadOnly a day. So I storage vmotioned everything off of the SAN and spent another week testing things to be absolutely sure that the SAN was "clean". I've not had any issues w/ Windows box, and I've checked every inch of the Network (switch ports, duplex settings, error levels, collisions etc..). I can get great performance from the SAN w/ very minimal latency. One caveat, is that I am using Vmware's Round Robin Load balancing w/ a pair of NICs on the servers, so I'm not sure if this plays into the issue.

Basically, I'm stumped on this one. Any suggestions? It looks like a SCSI timeout, but I can't see anything in the Vmware logs or the kernel logs to indicate it. The network is CLEAN and SOLID.. I can achieve 150MB / second sustained reads/writes from the SAN using the 2 Gig-E nics. When I was running on IET, I had NO such issues. The difference here seems to the be Wasabi target. Is there something that I'm overlooking?

P.S. And yes, this is ESX 3.5

Re: ESX 3.0.1 - Linux Guests go ReadOnly

Damin — Mon, 03 Mar 2008 14:26:58 GMT

I've taken to just running Tom's patched driver on all Centos 4 boxes. That seems to be the only thing that works for me. Still having a few issues w/ Centos 5 as well.

Re: ESX 3.0.1 - Linux Guests go ReadOnly

Damin — Tue, 29 Jan 2008 15:49:25 GMT

I guess this is the thread that just won't die!

I'm running a pre-production, test environment in my lab consisting of HCL SuperMicro boxes connected to a Wasabi Systems iSCSI SAN box. The purpose of this testing is to really put the Vmware multipath round-robin load balancing under the microscope and see how it performs.

My test bed consists of 2 SuperMicro boxes w/ 2 x Quad Core Intel boxes each connected via Gig-E iSCSI to a dedicated target NIC on the Wasabi box. The Wasabi box has 16 1 TB Sata drives configured in RAID-10 (it's a screamer!).

I am running 32 bit Centos 4 (latest release) VMs inside of the Vmware infrastructure. Kernel is the latest available from Centos (2.6.9-67.0.1.EL). Each VM is running w/ a single processor and 256 megs of memory, w/ a 4 Gig filesystem. I run 32 total VMs accross the two SuperMicro boxes.

I was under the assumption that the released kernel for the later Centos versions had the fixes to prevent this from happening in the event of a SAN failure, but I am still able to cause the guests to go readonly w/ the stock kernel. So I downloaded and applied Tom's patch, and verified that the driver is being loaded (rhel-vmfix in version string). However, I'm STILL getting timeouts even w/ the patched driver.

What I find particularly confusing, however, is that nowhere in the Vmware logs is there ANY indication of SAN timeouts. Everything looks clean as a whistle. Am I missing something really simple? Inquiring minds want to know..

Re: ESX 3.0.1 - Linux Guests go ReadOnly

tsightler — Fri, 07 Dec 2007 15:10:11 GMT

socius wrote:

Yes, they are indeed installed with a single partition and then cloned from the same installation. What surprised me is that we got this problem with file system corruption on all of the ones that got mounted as read only.

That is pretty unusual. You mentioned RHEL4 U3 and there was a known bug that caused additional corruption on ext3 with some kernels from that release. I don't remember the specifics, but it would cause a similar "ext3 journal abort" even on normal systems on the SAN. These issues are fixed in more recent releases (U5 and above).

Also, even though I've noticed many distro's have stopped suggesting using multiple filesystems for root, /var, and /usr I still think this is a good practice. We continue to use separate filesystems for volumes that get significant rights as this significantly decreases the chance of the root or /usr filesystem becoming corrupted.

Unfortunately, rescue mode did not help fixing the file systems. We managed to get some of our data back, but the vms still won't boot. We're naturally reluctant to install any more linux vms until we know that this won't happen again, that's why I' was curious if anybody else had experienced this.

The error you are getting is critical, but shouldn't be unrecoverable except in the worst cases. There are multiple copies of the group descriptors on disk and you should be able to locate them with dumpe2fs. Worst case you should be able to get your data by using debugfs. I've been doing this a long time, and I've never seen this be unrecoverable, but it is a touchy recovery and it's certainly possible that you have an unrecoverable situation that I've never run across. Just because I haven't seen it doesn't mean it does exist.

I can certainly understand being reluctant to reinstall linux VMs. I'd be that way too if I were in your situation. I will tell you that I personally run over 20 RHEL4 (and now a few RHEL5) VM's running critical production systems and have never seen more than minor corruption, and none since the problems with journal aborts were corrected. Of course, backups are always important, and you'll want to test your setup thoroughly but I believe it is possible to have great success with Linux on VMware.

Later,
Tom

Re: ESX 3.0.1 - Linux Guests go ReadOnly

socius — Fri, 07 Dec 2007 14:49:19 GMT

I'm not really an expert on SAN infrastructure so maybe these are not the answers you were looking for, but I know we are using HP EVA 8000.

The ESX hosts in question are running on HP DL385 G1 hardware and I think the HBAs are Emulex LP1050DC.

Re: ESX 3.0.1 - Linux Guests go ReadOnly

Damin — Fri, 07 Dec 2007 14:24:04 GMT

Yes, they are indeed installed with a single partition and then cloned
from the same installation. What surprised me is that we got this
problem with file system corruption on all of the ones that got mounted
as read only.

Unfortunately, rescue mode did not help fixing the file systems. We
managed to get some of our data back, but the vms still won't boot.
We're naturally reluctant to install any more linux vms until we know
that this won't happen again, that's why I' was curious if anybody else
had experienced this.

Out of curiosity, what sort of SAN setup are you running?

Re: ESX 3.0.1 - Linux Guests go ReadOnly

socius — Fri, 07 Dec 2007 06:58:55 GMT

Re: ESX 3.0.1 - Linux Guests go ReadOnly

tsightler — Thu, 06 Dec 2007 23:13:51 GMT

socius wrote:
The errors look like the attached screenshots in this post:

The first picture is before attempted reboot and the second is after.

I have never seen any of our filesystems corrupted significantly, but some did experience minor corruption and I would suspect it's possible that bad things could happen if the error hit at a critical time. Do you happen to install your systems with only a single partition (or perhaps just root and boot partitions)? That would certainly make it more susceptible to corrupting the root volume.

You can probably boot a CD in rescue mode and run fsck on the root filesystem to get the system booting again.

Later,
Tom

Re: ESX 3.0.1 - Linux Guests go ReadOnly

socius — Wed, 05 Dec 2007 13:39:44 GMT

The errors look like the attached screenshots in this post:

The first picture is before attempted reboot and the second is after.

Re: ESX 3.0.1 - Linux Guests go ReadOnly

sriramrajan — Wed, 05 Dec 2007 13:32:45 GMT

We ran about 20-25 RHEL 3 and 4 VMs using Vmware ESX 3.1 and 3.2

Never had a kernel panic. The read only issue is now fixed with RHEL 5 also.

I have seem kernel panics when using the Redhat cluster services on Vmware (free server) but that may not be related to this issue. Those are mainly with GFS or CLVMD.

Sriram

Re: ESX 3.0.1 - Linux Guests go ReadOnly

socius — Wed, 05 Dec 2007 12:17:01 GMT

Has anybody else had this problem with the addition of corrupted file systems? Almost all our rhel4u3 vms encountered this at some point this autumn and became unusable afterwards.

I only see people in this thread talking about the file systems being mounted as read only, do your vms also receive kernel panics at reboot after this happened?

Re: ESX 3.0.1 - Linux Guests go ReadOnly

rann — Mon, 26 Nov 2007 13:04:12 GMT

Hi All,

For what it's worth, we ran into this issue also. However, in our case this symptom (ext3fs remounts read-only) occured even after applying the mptscsi hotfix to a 2.6.9-42.0.8 kernel (the one with another ext3 + san based storage and ext3fs fix). We also found another thread which talks about increasing vm.min_free_kbytes to 10240 (see links below). Even after making sure all these three things were set and done, we encountered the issue!

However we did finally pinpoint another reason for this occurence. There is a recent update for ESX 3.0.2 (3.0.2 Update 1) and three updates for this release, detailed in ESX-1002424, ESX-1002425 and ESX-1002429. This update fixes a networking latency issue on various hardware that leads to high network latency (high ping response times) and extremely slow san/iscsi/nfs/smb speeds.

To summarize, what we had to do to finally alleviate the issue is this:

1: Set vm.min_free_kbytes = 10240 in /etc/sysctl.conf on guests:
http://www.noah.org/wiki/index.php/VMware_notes
http://communities.vmware.com/message/249823

2: Update to at least kernel 2.6.9-42.0.8 which fixes an issue with ext3fs on san/iscsi/nfs turning read-only by interpreting certain scsi layer messages as severe:
http://kbase.redhat.com/faq/FAQ_85_9610.shtm
https://bugzilla.redhat.com/show_bug.cgi?id=213921

3: Apply the fix mentioned on this page relating to the mptscsi driver and an upstream patch relating to failover/multipath functionality:
http://www.tuxyturvy.com/blog/index.php?/archives/31-VMware-ESX-and-ext3-journal-aborts.html
http://kb.vmware.com/selfservice/microsites/search.do?cmd=displayKC&externalId=51306
http://communities.vmware.com/thread/58121?tstart=0&start=0

4: Update ESX 3.0.2 to 3.0.2 Update 1 + three additional fixes: ESX-1002424, ESX-1002425 and ESX-1002429, fixing high networking latency and other issues:
http://communities.vmware.com/thread/97117
http://www.vmware.com/download/vi/
http://www.vmware.com/download/vi/vi3_patches_302.html

We have pretty thoroughly stress-tested the above configuration with a very high load on io and cpu for about three days now and have survived stuff that would previously lead to remounting read-only pretty quickly (used to be less than an hour in most cases).

Regards,

Rubin.

Re: ESX 3.0.1 - Linux Guests go ReadOnly

tsightler — Wed, 19 Sep 2007 02:42:38 GMT

OK, good, that's what I thought.

So just to clarify for future readers of the thread here is the current status as far as I know:

RHEL4 U3 & U4 -- VMware provides a patch for these OS versions in VMware KB ID 51306.
You can also upgrade to RHEL4 U5 which includes the required fixes.

SLES9 SP3, SLES10 -- VMware provides a patch for these OS versions in VMware KB ID 51306.

RHEL5 FCS (Initial Release) -- VMware provides a patch for these OS versions in VMware KB ID 1001778
The RHEL5 U1 beta kernel includes a fix for this issue and thus it is expected that when RHEL5 U1 final is released (probably sometime in October) it will also include the fix.

For other, unsupported distros (or even the distro's above if you don't want VMware's patches for some reason) you can also continue to use the patched drivers available on my site or follow the generic instructions to manually apply the source patch to the driver included with your distros kernel source package. I have success reports from many Debian and Ubuntu users as well as several other distros.

Later,
Tom

Re: ESX 3.0.1 - Linux Guests go ReadOnly

Damin — Tue, 18 Sep 2007 05:11:14 GMT

Unpatched RedHat 5 kernels (Centos) will break. I've patched several of the virtual machines w/ your Centos patches and so far so good. I guess I was under the mistaken impression that the mainlined redhat kernels in R5 had the patches.

Re: ESX 3.0.1 - Linux Guests go ReadOnly

tsightler — Tue, 18 Sep 2007 00:04:56 GMT

Damin,

Are you saying you are seeing this even with the VMware provided RHEL5 LSILogic patches or the RHEL5 patches from my site? I know that a completely unpatched RHEL5 system still has the problem, that's what I stated above (although perhaps not clearly).

Also, if you have access you may want to test the RHEL5.1 beta kernels as they appear to include the same fixes that Redhat included in RHEL4 U5.

Later,
Tom

Re: ESX 3.0.1 - Linux Guests go ReadOnly

Damin — Tue, 11 Sep 2007 03:30:49 GMT

Tom,
I've been testing the waters on our ESX 3.0.1 (w/ all patches) cluster w/ a bunch of Centos 5 virtual machines, and all of them tend to go ReadOnly at some point. Same behavior as the last time, while the Centos 4 boxes happily crank along. I'm running stock Centos kernels, so I'll look at the patches to see what might need to be done and automate it. I've also been seeing this with Rpath based appliances running the later 2.6.18 kenels as well.

Re: ESX 3.0.1 - Linux Guests go ReadOnly

tsightler — Fri, 31 Aug 2007 21:56:54 GMT

The fix posted by VMware for RHEL5 is equivalent to my fix for RHEL5 which I have posted on my website and is basically just a patch to the LSIlogic SCSI driver which gives the required "retry forever" behavior.

VMware is correct in stating that RHEL5 has another issue caused by code in the SCSI mid-layer that means that, even with the fix in the LSIlogic SCSI driver, it's still possible for RHEL5 to timeout during very long stalls in I/O. This is much more difficult to patch because I believe that it requires changes to code that is compiled directly into the kernel rather than simply loaded as a module.

That being said, the SCSI mid-layer takes a pretty large stall before it times out, enough that I'm not really sure it's a major issue. The old problem was that even fairly short stalls, on the order of 10-30 seconds, could cause the issue, but, so far in my testing, it seems to take minutes before the SCSI mid-layer in RHEL5 times out. If your storage array is pausing for minutes, you've got serious problems anyway.

I've been running RHEL5 through the paces pretty hard for about a month, first using my own patch, and now using VMware's patch, and I haven't been able to trigger this problem just using high loads. I have been able to trigger a timeout by doing things like disconnecting all cables or restarting the storage array, but I would actually expect those things to fail. In my opinion it was always a luxury that RHEL4 with the patch would actually survive a storage array reboot that took minutes.

Later,
Tom

Re: ESX 3.0.1 - Linux Guests go ReadOnly

alvserversupportteam — Fri, 24 Aug 2007 14:47:07 GMT

Has anybody tried this fix yet (for RH5)? Does it work or is tsightler's fix still required?
http://kb.vmware.com/selfservice/microsites/search.do?cmd=displayKC&externalId=1001778

I'm a little confused by the statement: "it also requires a SCSI mid-layer patch"

Where exactly is the SCSI mid-layer patch applied? OS, HBA firmware on my side or SAN side, Fiber Switch? Just a little confused if VMWare isn't providing it then are we waiting on Red Hat, LSI, other?

Re: ESX 3.0.1 - Linux Guests go ReadOnly

kiano — Tue, 24 Jul 2007 16:59:58 GMT

Hi - U5? I am running the 55.0.2 kernel from Red Hat following this link http://kbase.redhat.com/faq/FAQ_85_10846.shtm but have not got any more U5 updates (running up2date), largely because my /usr is almost full on the guest machine. So I am running RHEL 4 AS with a U5 kernel - but will this fix my issue? & are there any other specific RPM's I should get hold hold of?

Thanks

Andy

Re: ESX 3.0.1 - Linux Guests go ReadOnly

GvdWereld — Mon, 23 Jul 2007 17:01:49 GMT

For those running SLES9 or SLES10, see Novell's TID 3584352:

SUSE Linux Enterprise 10
This issue has been fixed as of Service Pack 1.

SUSE Linux Enterprise Server 9 and Open Enterprise Server (Linux based)
This issue has been fixed as of kernel 2.6.5-7.286.

Re: ESX 3.0.1 - Linux Guests go ReadOnly

tsightler — Sat, 14 Jul 2007 01:42:18 GMT

I've been happily running RHEL4 U5 on about a dozen VM's for weeks now, so far with no issues so, while RHEL4 U5 is not yet officially supported it seems to work fine.

That being said I'm sure you can happily run the RHEL4 U4 kernel with all of the U5 updates if you want to.

Later,
Tom

Re: ESX 3.0.1 - Linux Guests go ReadOnly

Ops admin — Tue, 10 Jul 2007 17:09:18 GMT

Is anyone running the latest Update 4 kernel but all the other Update 5 patches from RHN? Was wondering if that may be a good route to go until they fully support Update 5.

Does anyone know a good thread that talks about RHEL 4 Update 5 support? This one has good info, just cant find other ones.

Thanks

Re: ESX 3.0.1 - Linux Guests go ReadOnly

tsightler — Wed, 20 Jun 2007 14:32:39 GMT

I currently believe that the solution in RHEL4 U5 (kernel versions >= 2.6.9-55.EL) does fix this issue acceptably. I still think it may be possible for a very long (say >5 minute) pause in IO to cause the error to occur but if your SAN is pausing/stopping IO for >5 minutes then you probably have other issues you should address.

I've run every stress test I can throw at it, including rebooting my Equallogic array, and removing the zones on our CX700 and so far have not seen the issue with these kernels. Based on my testing results I moved our development Oracle systems, and several other fairly high I/O systems to this kernel, without patches, and they have survived several weeks without issue. These systems would typically fail in days, or even hours with the 2.6.9-42.x seriese of kernels.

Based on this, I believe the issue is "fixed" for RHEL4 Update 5, however, that still leaves a lot of distros that either still need the VMware fix or the manual compiled driver with my small patch.

RHEL5 still needs a "fix" as well, and since it's not yet officially supported from VMware for ESX my guess is it won't get a formal fix until it is certified. I plan to post a patched driver for RHEL5 on my website in the next day or so. I would suspect this newer driver would also work with other "newish" distros based on more recent kernels.

Later,
Tom

Re: ESX 3.0.1 - Linux Guests go ReadOnly

stericho — Mon, 18 Jun 2007 13:25:47 GMT

Anyone try this with RHEL 5 ?

My kernel version is 2.6.18-8.1.1.el5.

I am trying compile the module from source and
getting the following errors.

In file included from
/home/sri/mptscsi_vmware/mptscsi-rhel-3.02.62.01/mptba
se.c:50:
include/linux/config.h:6:2: warning: #warning
Including config.h is deprecated.
/home/sri/mptscsi_vmware/mptscsi-rhel-3.02.62.01/mptba
se.c: In function âmpt_suspendâ:
/home/sri/mptscsi_vmware/mptscsi-rhel-3.02.62.01/mptba
se.c:1646: error: switch quantity not an integer
/home/sri/mptscsi_vmware/mptscsi-rhel-3.02.62.01/mptba
se.c: In function âMakeIocReadyâ:
/home/sri/mptscsi_vmware/mptscsi-rhel-3.02.62.01/mptba
se.c:2510: error: implicit declaration of function
âcrashdump_modeâ
make[2]: ***
[/home/sri/mptscsi_vmware/mptscsi-rhel-3.02.62.01/mptb
ase.o] Error 1
make[1]: ***
[_module_/home/sri/mptscsi_vmware/mptscsi-rhel-3.02.62
.01] Error 2
make[1]: Leaving directory
`/usr/src/kernels/2.6.18-8.1.4.el5-i686'

Sriram

Same problem here. Also the version of the in-kernel driver is somewhat newer than the one provided from vmware.

Re: ESX 3.0.1 - Linux Guests go ReadOnly

stan-whitfield — Fri, 08 Jun 2007 01:50:24 GMT

Ok, I've got the driver installed, and we haven't had the file system (ReiserFS 3) go read only yet. However, I'm concerned. I'm still seeing tons of mpt errors on the SLES 10 server in /var/log/messages. Such as:

Jun 7 01:00:41 MoodleFinal kernel: mptbase: ioc0: IOCStatus(0x0002): Busy
Jun 7 01:01:02 MoodleFinal kernel: mptbase: ioc0: IOCStatus(0x0002): Busy
Jun 7 10:35:15 MoodleFinal kernel: mptscsih: ioc0: attempting task abort! (sc=c1df1980)
Jun 7 10:35:15 MoodleFinal kernel: mptbase: ioc0: IOCStatus(0x004b): SCSI IOC Terminated
Jun 7 10:35:15 MoodleFinal kernel: mptscsih: ioc0: task abort: SUCCESS (sc=c1df1980)

I'm sincerely hoping that even in the presence of these errors that the file system driver doesn' continue to remount the volume as read only. I already announced to our community that the problem has been fixed. Thankfully it hasn't happened yet since the driver update.

Some of you have asked so I'm posting the relevant error entries in ESX /var/log/vmkernel:

Jun 7 20:14:11 esx1 vmkernel: 145:08:22:29.648 cpu3:1687)VSCSI: 1829: Reset request on handle 8335 (0 outstanding commands)
Jun 7 20:14:11 esx1 vmkernel: 145:08:22:29.648 cpu3:1687)VSCSI: 1829: Reset request on handle 8336 (0 outstanding commands)
Jun 7 20:14:11 esx1 vmkernel: 145:08:22:29.648 cpu3:1663)VSCSI: 2028: Resetting handle 8335 [0/0]
Jun 7 20:14:11 esx1 vmkernel: 145:08:22:29.648 cpu3:1663)SCSI: 3222: handle 306590 / orig 0x7d71bb0
Jun 7 20:14:11 esx1 vmkernel: 145:08:22:29.648 cpu3:1663)VSCSI: 1878: Completing reset on handle 8335 (0 outstanding commands)
Jun 7 20:14:11 esx1 vmkernel: 145:08:22:29.648 cpu3:1663)VSCSI: 2028: Resetting handle 8336 [0/0]
Jun 7 20:14:11 esx1 vmkernel: 145:08:22:29.648 cpu3:1663)SCSI: 3222: handle 735951 / orig 0x7d70e08
Jun 7 20:14:11 esx1 vmkernel: 145:08:22:29.648 cpu3:1663)<6>scsi(1:0:0:12): DEVICE RESET SUCCEEDED.
Jun 7 20:14:11 esx1 vmkernel: 145:08:22:29.648 cpu3:1663)VSCSI: 1878: Completing reset on handle 8336 (0 outstanding commands)
Jun 7 20:17:24 esx1 vmkernel: 145:08:25:42.730 cpu2:1040)WARNING: SCSI: 1726: Unexpected status returned: bad000a I/O error

Where should I go from here to eliminate these errors?

Also, we're not currently backing up our Linux VMs (gasp) but we're going to be implementing ESX ranger within a month or two. So heavy disk I/O during backup isn't the cause. And our "read only" issue I don't believe is related to storage load. It seems to happen randomly.

Our hardware:

IBM bladecenter
2 LS20 blades running ESX 3.0.0
Qlogic SFF FC adapters
Bladecenter integrated 14x2 port FC switch module
Qlogic Sanbox 3050
IBM FasTt 600 and Nexsan Satablade SAN arrays

The two FC switches run in tandem with 2 uplinks between them for bandwidth and failover between the switches. The SAN arrays are connected to the 8 port Sanbox 3050. We are not multipathing. Both FC ports on Satablade are connected to the switch and I'm exposing LUNs from both ports. I have mapped raw LUNs exposed out of both ports for bandwidth balancing but I'm only exposing a given LUN on one port since we're not multipathing. The FasTt 600 only has one port connected to the switch.

Re: ESX 3.0.1 - Linux Guests go ReadOnly

stan-whitfield — Wed, 06 Jun 2007 09:47:29 GMT

Turns out swapping to the Buslogic isn't as straightforward as I thought it may be. I can't seem to select a new scsi module from yast if the hardware isn't "present' when I do a scan...

So, I downloaded the LSI logic SLES 10 driver patch linked in this KB:

http://kb.vmware.com/selfservice/microsites/search.do?cmd=displayKC&externalId=51306

However, the darn MD5 sum doesn't match. All the other files linked on that KB page have MD5 sums that check out, with the exceptoin of the SLES 10 download. I'm wary of using it because of this. And darn it I've been up all night working on this and want to have it done before I go into the office today!! I've got half a notion to just throw caution to the wind regarding the MD5 sum mismatch assuming it's a VMware documentation glitch. Something like ... they uploaded a new file and forgot to change the MD5 sum in the text of the KB article.

I've downloaded it twice to two different machines, one Linux one Windows, and the sum doesn't match what's listed in the KB. Each time I downloaded the file the checksum is the same, so I doubt there's a problem with the download. VMware please square this situation. It's annoying.

Update: apparently someone was listening, as after a few more repeated downloads, the MD5 sums now match. Weird...very weird.

Message was edited by:
stan-whitfield

Re: ESX 3.0.1 - Linux Guests go ReadOnly

stan-whitfield — Wed, 06 Jun 2007 07:42:11 GMT

I am so glad I found this thread! We've had this problem with SLES 10 VMs using the LSI Logic adapter on 3.0.0 (we'll be uprading to 3.0.1 soon). After researching this issue from the Linux kernel side, I found that the mptbase LSI Logic SCSI driver code in the 2.6.x kernel is buggy on its own, unrelated to VMware.

http://lkml.org/lkml/2005/10/11/224

I have 2 SLES 10 and 1 SLES 9 VMs all on 2.6.x kernels. The two SLES 10s were built with the LSI Logic adapter and driver and the SLES 9 VM was built with the Buslogic adapter. Both of the SLES 10 VMs experience the massive scsi errors and the read only remounts. The SLES 9 VM using Buslogic goes not.

I'm going to convert to Buslogic on the SLES 10 VMs and see if that helps. I think I saw a procedure for this documented in this thread. If not I'll figure it out, implement it, and I'll post back with results.

Oh, btw, we're all FC SAN at my shop. IBM and Nexsan arrays, Qlogic switches. No multipathing. We have a mix of VMFS volumes with multiple VMs stored on each VMFS. We map RDMs to individual VMs that need lots of storage. As a general rule, for any VM that will need over 10GB (i.e. database server), we create a 10GB virtual disk on a shared VMFS volume and install the base OS on it. We then map an RDM to it, format it with Reiser3, and mount it into the filesystem.

I do see some SCSI errors in our W2K3 VM event logs but we're not having huge issues as a result of them, not like with these Linux VMs. However, if the Buslogic adapter/driver works better in Windows, I'll look into swapping those over as well.

Re: ESX 3.0.1 - Linux Guests go ReadOnly

sriramrajan — Tue, 05 Jun 2007 00:15:08 GMT

Anyone try this with RHEL 5 ?

My kernel version is 2.6.18-8.1.1.el5.

I am trying compile the module from source and getting the following errors.

In file included from /home/sri/mptscsi_vmware/mptscsi-rhel-3.02.62.01/mptbase.c:50:
include/linux/config.h:6:2: warning: #warning Including config.h is deprecated.
/home/sri/mptscsi_vmware/mptscsi-rhel-3.02.62.01/mptbase.c: In function âmpt_suspendâ:
/home/sri/mptscsi_vmware/mptscsi-rhel-3.02.62.01/mptbase.c:1646: error: switch quantity not an integer
/home/sri/mptscsi_vmware/mptscsi-rhel-3.02.62.01/mptbase.c: In function âMakeIocReadyâ:
/home/sri/mptscsi_vmware/mptscsi-rhel-3.02.62.01/mptbase.c:2510: error: implicit declaration of function âcrashdump_modeâ
make[2]: *** [/home/sri/mptscsi_vmware/mptscsi-rhel-3.02.62.01/mptbase.o] Error 1
make[1]: *** [_module_/home/sri/mptscsi_vmware/mptscsi-rhel-3.02.62.01] Error 2
make[1]: Leaving directory `/usr/src/kernels/2.6.18-8.1.4.el5-i686'

Sriram

Re: ESX 3.0.1 - Linux Guests go ReadOnly

Damin — Sat, 02 Jun 2007 12:58:36 GMT

I'm not sure I'd consider it fixed until Tom weighs in w/ more testing. I have about 15 production VMs running Centos that will soon be updated to the Centos 2.6.9-55 kernel. I'm going to do a yum update to a pair of these boxes in the near future, so I'll let you know if I see the problem.

I would reccomend that someone who has a test farm (not production) load up a few VMs with the new kernel and do some insane disk access against the san.

Another interesting way to test it is to reboot your SAN while the boxes are running! With Tom's patch, our Linux boxes just "pause" until the SAN comes back up and then continue on as normal (it's actually kind of cool to see).

Re: ESX 3.0.1 - Linux Guests go ReadOnly

jatwell — Fri, 01 Jun 2007 20:02:39 GMT

So now, do we consider this fixed? Just wondering if it would be wise to recompile the driver for the new kernel or if the update already included in the new kernel definitely fixes this issue.

After updating to the newer kernel I have not seen this issue, but I only have 2 linux VM's. Soon, I plan on helping someone roll out 20-30 so I'd like to get this right and not run into the issue down the road on heavy hit production machines.

Re: ESX 3.0.1 - Linux Guests go ReadOnly

ibm-sorcerer — Wed, 23 May 2007 00:49:59 GMT

I also have the issue showing up on 2 of my RHEL5 systems. So we are all in the same boat...

Re: ESX 3.0.1 - Linux Guests go ReadOnly

tsightler — Wed, 09 May 2007 03:53:35 GMT

Well, as stated above, RHEL4 U5 does include a patch which is supposed to correct the issue. It's not the same patch that is on my website (which is really just a hack) but is supposed to be a more "correct" fix for the problem. So far testing is looking very good.

I've now upgraded two VM's to RHEL4 U5, one connected to an EMC AX150i (iSCSI) that was always easy to reproduce the issue with RHEL4 U3 and U4 in just a couple of hours (sometimes minutes), and another connected to an EMC CX700 (Fiber Channel) which I could usually make fail in 8-10 hours.

I've now hammered on both of these for almost 48 hours with no signs of problems so it seems like the "fix" in RHEL4 U5 looks pretty good, but I'm conservative, I'll need a few weeks of production runtime without errors, or reports from others, before I'll consider it completely safe.

Later,
Tom

Message was edited by:
tsightler

Re: ESX 3.0.1 - Linux Guests go ReadOnly

jatwell — Mon, 07 May 2007 19:43:57 GMT

I would be interested in knowing if the RHEL4 U5 kernel has the fix too.

Re: ESX 3.0.1 - Linux Guests go ReadOnly

tsightler — Mon, 07 May 2007 19:03:14 GMT

U5 does include a patch that is supposed to correct this issue. I've only got one VM running the new kernel so far but I hammered on it this weekend and it passed without any special patches so it looks promising. I've got some systems on a much slower EMC AX150i that were always great and showing the problem. I'll try to upgrade those and see how they behave in the next few days.

Interestingly, the problem does seem to still be an issue for RHEL5 (admittedly not supported by VMware yet anyway, but we've got a few test systems running). I'm hoping to update my page with a workaround for RHEL5 soon.

Later,
Tom

Re: ESX 3.0.1 - Linux Guests go ReadOnly

soleblazer — Sat, 05 May 2007 20:16:52 GMT

update 5 is out now, anyone know if this solves the issue?

Re: ESX 3.0.1 - Linux Guests go ReadOnly

tsightler — Sun, 01 Apr 2007 16:29:07 GMT

Has anyone tested RHEL4 U5 Beta? Kernels newer than 2.6.9-45 are supposed to have a more "correct" fix for this issue built in but I haven't had time to test. I'm going to try to find time this week, but I'd love to hear if anyone else has given them a spin.

Later,
Tom

Re: ESX 3.0.1 - Linux Guests go ReadOnly

sjc_dogs — Tue, 06 Mar 2007 23:04:11 GMT

Hi,

I am having problem on RHEL4U3, RHEL3U8 and SUSE9 SP3 VMs. This is a NAS / SAN system and I am running Connectathon test which is an NFS test for NAS side testing. When running the special test using NFS version 3 , using UDP protocol, it gets stuck at a point where it says "write/read 30 MB file". On RHEL3U8 and SUSE9 SP3, it gets stuck for 5-7 min, then continues fine. On RHEL4U3, it gets stuck forever.

Is my problem and the problem you discussed can be related? Plesae let me know.

Re: ESX 3.0.1 - Linux Guests go ReadOnly

tsightler — Thu, 01 Mar 2007 22:27:36 GMT

The "latest kernel" does not fix the problem, at least as of 2.6.9-42.0.8, so, yes, if you are going to update to the latest kernel then you also need either VMware's fix or my fix.

On the other hand, if you are on kernel 2.6.9-22.0.2 or earlier, and you're not planning to upgrade your kernel, then you don't need to do anything as those kernels don't have the problem.

Later,
Tom

Re: ESX 3.0.1 - Linux Guests go ReadOnly

cuzic4n — Wed, 28 Feb 2007 15:46:30 GMT

So, to summarize should we get the kernel updated to 2.6.9-42.0.8 AND apply the vmware patch via http://kb.vmware.com/vmtnkb/search.do?cmd=displayKC&docType=kc&externalId=51306&sliceId=SAL_Public
OR do we just apply the kernel patch and we should be good to go since the problem really was in rhel4?

I was set on just going to the latest kernel, but then "Ops Admin" replied 2 posts before this one and said that he was on the latest kernel and still had problems..

Re: ESX 3.0.1 - Linux Guests go ReadOnly

tsightler — Mon, 26 Feb 2007 14:41:28 GMT

We have 20+ other systems that are running the stock
stuff "2.6.9-22.ELsmp #1 SMP" and "mptlinux-3.02.18".
We need to update the kernel to apply the patch.

Does anyone know if we need to apply the patch to the
stock kernel systems? I am just wondering if the
problem wont show itself with what we have on the
older systems.

For RHEL4 the problem only occurs with U3 and above. The 2.6.9-22.EL kernel is, I believe, the kernel from RHEL4 U2 so it doesn't actually have the problem. I'm not 100% sure, but I believe that the first "released" kernel that included this problem is 2.6.9-34.EL. Any non-development kernel older than that should be safe.

I actually think the update that broke this was added in the U3 development chain around kernel 2.6.9-22.25.EL, but it's been a long time so I may not have the exact correct release. For example, I have some 2.6.9-28.EL U3 beta kernels which definitely have the "broken" driver.

Later,
Tom

Later,
Tom

Re: ESX 3.0.1 - Linux Guests go ReadOnly

Ops admin — Sat, 24 Feb 2007 04:54:39 GMT

This came up on our farm.

In fact one of the servers was rebooted, when it came back up all that was left of the "/" filesystem was "lost&found" The entire rest was gone!!

One thing we have noticed, the 3 systems that have the problem all have the updated Redhat kernel from RHN "2.6.9-42.0.8.ELsmp #1 SMP" and the following entry for the driver "mptlinux-3.02.62.01rh". We can apply the patch to these servers.

We have 20+ other systems that are running the stock stuff "2.6.9-22.ELsmp #1 SMP" and "mptlinux-3.02.18". We need to update the kernel to apply the patch.

Does anyone know if we need to apply the patch to the stock kernel systems? I am just wondering if the problem wont show itself with what we have on the older systems.

Re: ESX 3.0.1 - Linux Guests go ReadOnly

tsightler — Wed, 14 Feb 2007 20:43:15 GMT

Our company had a thorough discussion and meeting
sessions setup with redhat on this 'readonly issue'
not sure if its identical but we actually had a
hot-fix and new kernel created from our testing. If
you using RHEL your going to need to lay down
2.6.9-42.0.8 kernel. 42.0.3 is not sufficient enough.
Bugzilla 213921 is a good reference to the trouble we
had.

The bug fixed in 2.6.9-42.0.8, and documented in Bugzilla 213921, is an issue with ext3 and is not really related to VMware, although I suspect that VMware may make the problem slightly more likely to occur.

The "VMware" bug is an issue with the mptscsih driver, it's interaction with VMware server and how it reports timeouts to the SCSI midlayer in the guest.

While the failure mode of both bugs does cause a nearly indetical end result (ext3 filesystem going read-only) but key differences can be noted in the log files in most cases.

Later,
Tom

Re: ESX 3.0.1 - Linux Guests go ReadOnly

petr — Wed, 14 Feb 2007 16:52:10 GMT

Do you have logs from guest & vmkernel from around time of failure? You should hit this problem only if I/O takes longer than 30 seconds to finish (and you'll hit it on both real hardware and in the VM). Perhaps your SAN decided to do something else in the middle of test?

Re: ESX 3.0.1 - Linux Guests go ReadOnly

ddecker — Wed, 14 Feb 2007 16:35:09 GMT

Our company had a thorough discussion and meeting sessions setup with redhat on this 'readonly issue' not sure if its identical but we actually had a hot-fix and new kernel created from our testing. If you using RHEL your going to need to lay down 2.6.9-42.0.8 kernel. 42.0.3 is not sufficient enough. Bugzilla 213921 is a good reference to the trouble we had.

Re: ESX 3.0.1 - Linux Guests go ReadOnly

magusnet — Wed, 14 Feb 2007 15:56:27 GMT

This is just what I need.
Thanks for the assist

If I come up with any variations of:
'iozone -r 1m -s 1024m -t 8'
with iozone or any other tools, I'll post back here.

Re: ESX 3.0.1 - Linux Guests go ReadOnly

tsightler — Wed, 14 Feb 2007 00:28:47 GMT

1. Can someone provide a clear set of steps to
reproduce this in a lab?

You mean reproduce the error? It's a little difficult to provide clear steps since it can be more difficult to trigger on some hardware, however, the basic formula in my case is, run a continuous loop of ' iozone -r 1m -s 1024m -t 8' in 2 VM's against a single VMFS LUN. Given enough time this has failed on every one of my SAN's without doing much of anything else, usually within just a few hours.

Damin seems to have a more formal test that he feels comfortable with on his setup.

Later,
Tom

Re: ESX 3.0.1 - Linux Guests go ReadOnly

magusnet — Mon, 12 Feb 2007 22:09:18 GMT

The VMware patch won't help us since we are on SLES9_SP3 kernel build -282.

1. Can someone provide a clear set of steps to reproduce this in a lab?
2. Has anyone gotten any information from Novell for SLES9 on a fix like
RedHat has done?

Re: ESX 3.0.1 - Linux Guests go ReadOnly

egr — Fri, 09 Feb 2007 14:13:54 GMT

Hi,

VMware has updated the article:
http://kb.vmware.com/vmtnkb/search.do?cmd=displayKC&docType=kc&externalId=51306&sliceId=SAL_Public

Now a hotfix for SLES 9 SP3 is available.
Does anyone if this does work for SLES 10, too?

@doomdevice : Thanks for your work!
But ask that because we are looking for an offiicially supported solution.

Thanks in advance.
/egr

Re: ESX 3.0.1 - Linux Guests go ReadOnly

doomdevice — Tue, 06 Feb 2007 16:01:29 GMT

If you use Novell SLES, I´ve created an short guideline how to change the lsi driver based on Tom´s RedHat guide (Thanks Tom!).
You can find it in here:
http://www.vmachine.de/kb/index.php/Linux_Kernel_2.6_Problem_-_Read-Only_Filesystem_nach_Path_Failover

It´s in German but the commands should be understandable by everyone.

Dennis

Re: ESX 3.0.1 - Linux Guests go ReadOnly

tsightler — Mon, 05 Feb 2007 15:47:04 GMT

So, as I suspected, the patch is question is simply not enough. Of interest however is that Redhat has commited a patch to the U5 stream which does include a supposed fix (and one that looks "sufficient" to me). Test kernels that include this fix are available at the following location:

http://people.redhat.com/~jbaron/rhel4/RPMS.kernel/

It would be interesting to see what you tests turn up with those kernels.

Later,
Tom

Re: ESX 3.0.1 - Linux Guests go ReadOnly

Damin — Mon, 05 Feb 2007 04:17:45 GMT

After running several tests w/ the latest kernel, I've reverted back to using Tom's patched drivers. The .08 release will eventually fail under stress.

Sigh..

Re: ESX 3.0.1 - Linux Guests go ReadOnly

tsightler — Sat, 03 Feb 2007 21:55:14 GMT

The patch in question appears to be as follows:

--- linux-2.6.9/fs/ext3/dir.c~ 2007-01-08 16:19:36.000000000 -0500
+++ linux-2.6.9/fs/ext3/dir.c 2007-01-08 16:22:10.000000000 -0500
@@ -155,7 +155,7 @@ static int ext3_readdir(struct file * fi
brelse (tmp);
}
if (num) {
- ll_rw_block (READA, num, bha);
+ ll_rw_block (READ, num, bha);
for (i = 0; i < num; i++)
brelse (bha);
}

I'm not sure that this is enough to correct the issue in VMware because the Bugzilla seems to imply the issue was only seen in cases that were using PowerPath or dm-multipath. When using READA request under heavy load the bio request can sometimes return EAGAIN which is basically saying "I had a temporary error, please try again" but the ext3 code did not actually expect to handle this condition.

Redhat initially looked at backporting the code in more recent 2.6 kernels but this appeared to have more complex interactions and they opted for the simple, low-risk fix, which appears to basically forgoe readahead on directory lookups. It's possible that this might have a minor impact on performance, but probably not noticable in most workloads.

I'll be interested in seeing how your tests turn out so let us know.

Later,
Tom

Re: ESX 3.0.1 - Linux Guests go ReadOnly

Damin — Thu, 01 Feb 2007 05:07:12 GMT

Alright people.. The new kernel update for RedHat claims to fix the issue:

http://rhn.redhat.com/errata/RHSA-2007-0014.html

Specifically:

213921 - SAN file systems becoming read-only

I'm going to pop it on a few VMs and beat the crap out of them to see.

Anyone care to look at the patches applied to this update to see how they fixed it?

Re: ESX 3.0.1 - Linux Guests go ReadOnly

tsightler — Sat, 20 Jan 2007 19:40:23 GMT

OK, I know you didn't ask me this question, but I thought my experiences might be interesting to you. We have ESX servers running against several different storage arrays, a CX400 via FC, a CX700 via FC, a AX150i via iSCSI, and an Equalogic PS300E via iSCSI (and also a few systems running on local storage like IBM ServeRAID and Dell PERC controllers).

We've found the problem to be fairly easy to reproduce on the AX150i, as well as the CX700. For whatever reason it's actually more difficult to trigger it on the CX400, we theorize it has to do with the smaller write cache and thus lower latency during heavy writes but it may also be related to the fact that the CX400 simply has less contention because it services fewer hosts.

The Equalogic PS300E is by far the most difficult for us to reproduce the issue. We did mange to create the issue even when running against this array but it took a pretty crazy level of fake I/O load running in multiple VM's to do it.

On the other hand, if you pull the plug on a network cable, it's likely that you'll see the issue on every Linux guest by the time the iSCSI connection fails over to another port. My opinion is that this fix is an important proactive fix if you want your guest to continue running during any failure scenario, especially since that's usually the reason you invest in all of the redundancy.

Later,
Tom

Re: ESX 3.0.1 - Linux Guests go ReadOnly

tsightler — Sat, 20 Jan 2007 19:26:41 GMT

I'm seeing this behavior (Linux guest goes read-only
suddenly) on a guest OS that is running on _local_
storage. (I.e., not on a SAN, and not on iSCSI.)
Has anyone seen this?

Oh yeah, I can definitely reproduce this issue on local storage, basically, anything that causes a delay of greater than 5 seconds in reading data from the disks. It's especially easy to trigger if you have a storage system with large amounts of cache that creates long pauses during forced flushes. We've reproduced the issues on IBM ServeRAID 6i and Dell PERC 4/Di controllers using locally attached UltraSCSI 320 drives.

Re: ESX 3.0.1 - Linux Guests go ReadOnly

letoatrads — Sat, 20 Jan 2007 13:31:31 GMT

Do you use NIC teaming?. If yes, could you check if
the problem happen when there is only one uplink?.

Is this being asked because there is a possibility that having a NIC bond going to the ISCSI target could mitigate this issue? The reason I ask is I have several ESX Hosts hitting an EQL ISCSI SAN, and I have several RHEL4 UP4 guests with Sybase Databases running in the on the guest and I've never seen this issue. Even when I've spent 9 or 10 hours loading 1TB+ database.

Reading this thread makes me wonder if I should proactively apply the 'fix', but since I haven't ran into the issue, I'm reluctant to mess with something that is working.

Re: ESX 3.0.1 - Linux Guests go ReadOnly

osterber — Sat, 20 Jan 2007 04:23:17 GMT

I'm seeing this behavior (Linux guest goes read-only suddenly) on a guest OS that is running on _local_ storage. (I.e., not on a SAN, and not on iSCSI.) Has anyone seen this?

-Rick

Re: ESX 3.0.1 - Linux Guests go ReadOnly

tonywieczorek — Thu, 18 Jan 2007 19:15:17 GMT

For those who come to this forum and need a VMWare fix, after much searching I've come up with this link: http://kb.vmware.com/vmtnkb/search.do?cmd=displayKC&docType=kc&externalId=51306&sliceId=SAL_Public

We've been running this patch on several of our production Oracle and web servers for a few days now, and no more of those messages.

Hope this saves some trouble for people!

Re: ESX 3.0.1 - Linux Guests go ReadOnly

CMCC — Thu, 18 Jan 2007 16:54:34 GMT

thanks, I really need your help 'cause it's a critical machine and I have just one more day to find a solution...

Re: ESX 3.0.1 - Linux Guests go ReadOnly

tsightler — Thu, 18 Jan 2007 16:41:41 GMT

No, probably not.

Your problem appears to be that you updated your system and the initrd image does not contain the proper scsi drivers. You will need to boot your RHEL4 U4 CD into rescue mode and fix modprobe.conf file and rebuild your initrd. I'll try to reply with more details in your thread.

Later,
Tom

Re: ESX 3.0.1 - Linux Guests go ReadOnly

CMCC — Wed, 17 Jan 2007 20:29:16 GMT

is this solution useful for my problem???
http://www.vmware.com/community/thread.jspa?threadID=67781&tstart=0

Re: ESX 3.0.1 - Linux Guests go ReadOnly

tsightler — Wed, 17 Jan 2007 01:42:16 GMT

VMware has now posted their "solution" in the KB article, basically just an RPM with new mptscsih modules and a script which makes installing the drivers easier. So far it is only for 32-bit RHEL4 U3 or U4 (looks like that's all they've made binaries for). Still, for people who only want to run "supported" solutions then at least you finally have an option.

Later,
Tom

Re: ESX 3.0.1 - Linux Guests go ReadOnly

Daniel Herrero — Mon, 15 Jan 2007 17:39:28 GMT

Hello, have the same problem. Thanks to your comments, we found a solution.

Many thanks to all of you who got involved in this thread, this has been really really helpful.
You rule!

Re: ESX 3.0.1 - Linux Guests go ReadOnly

tsightler — Fri, 12 Jan 2007 22:12:23 GMT

That is pretty funny. I'm sure glad that lots of users reported it and thus allowed them to "discover" it. I also like the note that was added to the install documentation telling you to look at the KB article which basically says, "if you use RHEL4 U3 or U4 then don't upgrade 3.x" but they still list RHEL4 U3 or U4 as supported for 3.x.

So basically their saying, "We support it, but that doesn't mean we can make it work". Gee thanks, I always wondered what I was paying for.

Oh well, I guess we can look forward to a fix eventually.

Later,
Tom

Re: ESX 3.0.1 - Linux Guests go ReadOnly

Damin — Fri, 12 Jan 2007 20:15:56 GMT

Vmware uncovered this? Hahahahaha.. that's spin! Vmware did not uncover this.. their users did!

Makes you wonder just what goes on when they certify a specific OS distribution to work on Virtual Infrastructure....

Re: ESX 3.0.1 - Linux Guests go ReadOnly

CMCC — Fri, 12 Jan 2007 19:56:05 GMT

VMware has uncovered a problem with RHEL4 U3, RHEL4 U4 or SLES9 SP3 virtual machines running on SAN or iSCSI storage. File systems could become read-only in the event of busy I/O retry or path failover. VMware does not recommend ESX Server 3.x customers upgrade virtual machines to these operating systems until this issue is resolved.

From: http://kb.vmware.com/vmtnkb/dynamickc.do?externalId=51306&sliceId=SAL_Public&command=show&forward=nonthreadedKC&kcId=51306

Re: ESX 3.0.1 - Linux Guests go ReadOnly

tsightler — Sat, 09 Dec 2006 00:17:29 GMT

You should be able to look at the source and make the appropriate change as the code in FC6 is not significantly different in this particular area but I think the file is called mptscsih.c in the newer kernels.

I don't have the FC6 source handy, but basically, if you simply simply take the FC6 source and look in drivers/message/fusion there should be a file called mptscsih.c Open this file and do a search for DID_BUS_BUSY. Approximately the third hit from the top of the file should be a line that looks like this:

sc->result = (DID_BUS_BUSY << 16) | scsi_status;

Simply change the DID_BUS_BUSY to DID_OK so that the line looks like this:

sc->result = (DID_OK << 16) | scsi_status;

And then remcompile and you should be good. If you can't get this to work I can try to put some more specific instructions together in another day or so, perhaps even a download, I just don't have time at the moment.

Later,
Tom

Re: ESX 3.0.1 - Linux Guests go ReadOnly

Damin — Fri, 08 Dec 2006 18:13:41 GMT

Not that I am aware of. I try to stay away from Fedora for production use, as it is a testing ground for new technology. RedHat EL and derivatives aren't quite so cutting edge, but their bugs are well known.

Re: ESX 3.0.1 - Linux Guests go ReadOnly

jarjona — Fri, 08 Dec 2006 17:52:22 GMT

Hello:

I am working on a Fedora Core 6 box, and have the same issue.
Currently running 2.6.18-1.2849. When I try to compile the patch, of course, it breaks horribly:

make -C /lib/modules/2.6.18-1.2849.fc6/build SUBDIRS=/usr/src/fusion clean
make[1]: Entering directory `/usr/src/kernels/2.6.18-1.2849.fc6-i686'
make[1]: Leaving directory `/usr/src/kernels/2.6.18-1.2849.fc6-i686'
make -C /lib/modules/2.6.18-1.2849.fc6/build SUBDIRS=/usr/src/fusion modules
make[1]: Entering directory `/usr/src/kernels/2.6.18-1.2849.fc6-i686'
CC [M] /usr/src/fusion/mptbase.o
/usr/src/fusion/mptbase.c:50:26: error: linux/config.h: No such file or directory
In file included from /usr/src/fusion/mptbase.c:71:
/usr/src/fusion/mptbase.h:1088:28: error: linux/diskdump.h: No such file or directory
/usr/src/fusion/mptbase.c: In function âmpt_suspendâ:
/usr/src/fusion/mptbase.c:1646: error: switch quantity not an integer
/usr/src/fusion/mptbase.c: In function âMakeIocReadyâ:
/usr/src/fusion/mptbase.c:2510: error: implicit declaration of function âcrashdump_modeâ
/usr/src/fusion/mptbase.c:2510: error: implicit declaration of function âdiskdump_mdelayâ
make[2]: *** [/usr/src/fusion/mptbase.o] Error 1
make[1]: *** [_module_/usr/src/fusion] Error 2
make[1]: Leaving directory `/usr/src/kernels/2.6.18-1.2849.fc6-i686'
make: *** [default] Error 2

Any patches for this kernel?

Thanks,

-- Juan

Re: ESX 3.0.1 - Linux Guests go ReadOnly

Damin — Fri, 24 Nov 2006 01:18:10 GMT

Just a followup to this thread. So far, I've had good luck with Tom's patch and haven't had any guests go read-only since. As a result, we put the system back into uber-aggresive DRS mode and it's done 798 migrations of virtual machines so far.

Still no word from Vmware or RedHat on an "official" fix yet.

Re: ESX 3.0.1 - Linux Guests go ReadOnly

Damin — Tue, 14 Nov 2006 02:02:46 GMT

BTW, I'd be interested in hearing more about your
56/60 byte problem. Sounds like you have a switch
that is not adding the padding if it strips the VLAN
tag, which is actually very common. Do you have
another thread about it? Did you come up with the
patch yourself or is it something from VMware?

I don't know where the initial information came from. I suspect our Operations Manager found the information somewhere in these forums and/or out on the net.

The specific "patch" is nothing more than changing

net.h:#define ETH_MIN_FRAME_LEN 60

to a value of 56 in the vmxnet driver source.

And yes, this is related to a specific vendors equipment not properly padding VLAN tagged packets. We've had discussions with them, but they seemt to just not get it.

Re: ESX 3.0.1 - Linux Guests go ReadOnly

tsightler — Tue, 14 Nov 2006 01:40:51 GMT

Excellent, thanks Damin.

BTW, I'd be interested in hearing more about your 56/60 byte problem. Sounds like you have a switch that is not adding the padding if it strips the VLAN tag, which is actually very common. Do you have another thread about it? Did you come up with the patch yourself or is it something from VMware?

Later,
Tom

Re: ESX 3.0.1 - Linux Guests go ReadOnly

Damin — Fri, 10 Nov 2006 08:18:14 GMT

Here are the links to the kernels:

http://www.nacs.net/yum/kernel-2.6.9-42.0.3.VM.src.rpm

http://www.nacs.net/yum/kernel-2.6.9-42.0.3.VM.x86_64.
rpm
http://www.nacs.net/yum/kernel-debuginfo-2.6.9-42.0.3.
VM.x86_64.rpm
http://www.nacs.net/yum/kernel-devel-2.6.9-42.0.3.VM.x
86_64.rpm
http://www.nacs.net/yum/kernel-largesmp-2.6.9-42.0.3.V
M.x86_64.rpm
http://www.nacs.net/yum/kernel-largesmp-devel-2.6.9-42
.0.3.VM.x86_64.rpm
http://www.nacs.net/yum/kernel-smp-2.6.9-42.0.3.VM.x86
_64.rpm
http://www.nacs.net/yum/kernel-smp-devel-2.6.9-42.0.3.
VM.x86_64.rpm

Did some testing, and the patch that I applied when I rolled up these kernels does NOT give the desired behaviour. Guests go read-only.

However, on a separate note, I did follow the process outlined in Tom's article using his driver archive and I am happy to report that guests with the patched drivers have sustained 24 hours of straight Iozone abuse, with hundreds of AsyncIO timeouts on the Iscsi layer. We're hammering the living crap out of the Iscsi bus and it's working like a champ.

I've created an automated script for building / patching our RedHat boxes. You can review the script here:

http://www.nacs.net/n2net-vmfix.sh

This does two things;
1. Grabs, builds and installs the patched Fusion MPT drivers
2. Grabs, builds and installs a patched Vmxnet driver

The Vmxnet driver deals with a specific issue that we have related to runt-packets on a Vlan tagged network, whereby if a piece of hardware does not properly pad a tagged packet to 60 bytes, it will be dropped. In our case, it lowers the threshold to 56 bytes. It's a simple, one line change, but it allows the networking to function while the vendor scratches their head and tries to figure out why their hardware is broken.

Let me know if the script works for you. Simply run it, answer any questions it asks, then reboot. Do a "dmesg" and search for "vmfix" and you should see the driver version listed.

Re: ESX 3.0.1 - Linux Guests go ReadOnly

soleblazer — Thu, 09 Nov 2006 23:17:48 GMT

I havn't done the fix, but I can confirm that I am suffering from this same issue. Its not necessarily under load for me though, I have used iozone alot and no issues. The problem creeps up for me when esx does hba failover. Doesnt always happen, but in my tests today I had on AS3update6 vm, ES4update3 vm, and a ES4update4 vm. I pulled the fiber out of one of the hba's, took about 7 times, but eventually when problems did happen, they only happed on the ES4 vm's. THe AS3 vm stayed up throughout. I duplicated this 4 seperate times on two different ESX servers.

Anybody have any idea when any sort of fix is out for this? I guess its redhat that needs to fix, but this sucks big time for me as I have to put a total halt on all vmware use.

Re: ESX 3.0.1 - Linux Guests go ReadOnly

Damin — Mon, 06 Nov 2006 16:22:46 GMT

Alright.. Those of you that are feeling adventurous...

Until Vmware, LSI Logic and RedHat figure out an "official" solution to this that makes it into the mainline kernel, (and no, running the BusLogic driver is not an acceptable solution), I guess it is incumbent on the community to work around the issue.

To that end, I've taken the patch discussed in Tom's Ramblings here:
http://www.tuxyturvy.com/blog/index.php?/archives/31-VMware-ESX-and-ext3-journal-aborts.html

and I have rolled it into a custom kernel RPM for Centos 4.4. If the patch works, it should revert the behavior of the driver to pre U3 behavior and if there is an underlying storage timeout, the VM should pause indefinitely instead of timing out. It's not pretty, but it should solve the problem as others have noted earlier in this thread.

I've only created x86_64 kernels currently, as I run on Opteron processors, but if things work, I'll roll up some i686 versions.

Here are the links to the kernels:

http://www.nacs.net/yum/kernel-2.6.9-42.0.3.VM.src.rpm
http://www.nacs.net/yum/kernel-2.6.9-42.0.3.VM.x86_64.rpm
http://www.nacs.net/yum/kernel-debuginfo-2.6.9-42.0.3.VM.x86_64.rpm
http://www.nacs.net/yum/kernel-devel-2.6.9-42.0.3.VM.x86_64.rpm
http://www.nacs.net/yum/kernel-largesmp-2.6.9-42.0.3.VM.x86_64.rpm
http://www.nacs.net/yum/kernel-largesmp-devel-2.6.9-42.0.3.VM.x86_64.rpm
http://www.nacs.net/yum/kernel-smp-2.6.9-42.0.3.VM.x86_64.rpm
http://www.nacs.net/yum/kernel-smp-devel-2.6.9-42.0.3.VM.x86_64.rpm

NOTE: I didn't take the time to sign these RPMS, and while there is yum repodata that you can use, it will complain about the RPMS not being signed and refuse to install them. This is probably configurable, but this is only designed to be a temporary measure anyway, so just let me know if they work for you!

Re: ESX 3.0.1 - Linux Guests go ReadOnly

tsightler — Fri, 03 Nov 2006 03:55:12 GMT

Good, I was pretty sure I didn't miss this but couldn't be sure.

I'm very curious if others are seeing success with the patch. Both of my most troublesome systems just hit the 10 day mark with no issues. Previous record was 7 days and typically only 2-3 days so it seems to be working great for me.

I've also had two other users who have reported at least initial success but more datapoints and a little more history would be nice.

Thanks,
Tom

Re: ESX 3.0.1 - Linux Guests go ReadOnly

paithal — Thu, 02 Nov 2006 22:23:24 GMT

It was my mistake, my rpmbuild had failed to update the source. The problem seems to exist on U4 too.

Re: ESX 3.0.1 - Linux Guests go ReadOnly

tsightler — Thu, 26 Oct 2006 19:47:41 GMT

Where did you decide that U4 rolled back this change? The code I am providing is against the mptscsih driver in U4, specifically, I took the mptscsih driver from the 2.6.9-42.0.3 kernel, the most recent public release, and patched that.

I'd be interested to know why you think this was rolled back in U4.

Later,
Tom

Re: ESX 3.0.1 - Linux Guests go ReadOnly

paithal — Thu, 26 Oct 2006 08:22:31 GMT

Well done Tom. I looked the the mptscsi source too. Yes, U3 added the code that introduced this incorrect behavior. Looks like U4 rolledback the change.

For others who do not want to upgrade to U4 or patch this driver module with the fix, you can use BusLogic emulation as the guest SCSI controller.

Re: ESX 3.0.1 - Linux Guests go ReadOnly

Damin — Tue, 24 Oct 2006 22:38:32 GMT

I am currently in Dallas at Astricon, however, David Coulson our Operations Manager has patched a virtual machine or two with the driver. I will ask him to post his experience here.

Re: ESX 3.0.1 - Linux Guests go ReadOnly

tsightler — Tue, 24 Oct 2006 22:36:15 GMT

We have now survived 24 hours running our simple iozone benchmark which used to trigger the issue in only a few minutes so I'm very interested in your results.

If you do have the problem, if you could possibly capture the output of dmesg that would be great.

Later,
Tom

Re: ESX 3.0.1 - Linux Guests go ReadOnly

paithal — Tue, 24 Oct 2006 20:25:18 GMT

Thanks. I like to hear from Damin too regarding his configuration and how was the usage of iSCSI.

Re: ESX 3.0.1 - Linux Guests go ReadOnly

CTeague — Tue, 24 Oct 2006 16:04:36 GMT

I have updated my development CentOS 4.4 VM with your proposed fix. Usually it takes me 24-48 hours to encounter a read-only file system. So far it's been working well since 9:00 AM EST.

If the VM is stable after a week I'll be fairly certain the cause of this problem has been found.

Thank you Tom for the patch and directions.

Re: ESX 3.0.1 - Linux Guests go ReadOnly

tsightler — Tue, 24 Oct 2006 13:06:35 GMT

Wow, you sure ask a lot of questions!

What kind of test you are doing to get to the read-only situation?.

Basically, anything that eventually leads to a timeout on the ESX host. For the most part this simply involves running disk intensive benchmarks on multiple guests. We've been using iozone, but I would suspect you could use almost anything that generates a lot of I/O.

I assume the boot disk is on iSCSI volume.

No, boot disk is on locally attached RAID storage (Dell PERC 4)

Do you use virtual disk in VMFS volume or RDM for guest's boot disk?. If using VMFS, do you perform any I/O's to VMFS volume apart from booting (and guest doing I/O's)?.

We use a virtual disk in a VMFS volume. Yes, of course there are I/O's other than booting since the entire virtual disk for the guest is on the VMFS volume.

Are you sharing the LUN with any other server?.

I can't remember, but did you mention which target array you are using?.

One other ESX server.

Do you use NIC teaming?. If yes, could you check if the problem happen when there is only one uplink?.

Yes, but the problem will happen even with a single uplink.

You seem to be under the impression that the problem can only happen with iSCSI, however, we have reproduced the issue with locally attached RAID, and with fiber channel attached storage. Basically, if you can genereate enough I/O that you get an Async I/O error, the chances of seeing your ext3 volumes go read only is high.

If you see a message like this in your log:

Oct 22 01:28:19 esxhost1 vmkernel: 13:20:19:56.753 cpu3:1028)SCSI: 3731: AsyncIO timeout (5000); aborting cmd w/ sn 399366, handle b78/0x6a028a8

You are probably going to have a problem. With the small change I made to the LSI Logic driver the linux systems seem to be able to survive these timeouts.

Later,
Tom

Re: ESX 3.0.1 - Linux Guests go ReadOnly

paithal — Tue, 24 Oct 2006 07:38:13 GMT

What kind of test you are doing to get to the read-only situation?.

I assume the boot disk is on iSCSI volume. Do you use virtual disk in VMFS volume or RDM for guest's boot disk?. If using VMFS, do you perform any I/O's to VMFS volume apart from booting (and guest doing I/O's)?.

Are you sharing the LUN with any other server?.

I can't remember, but did you mention which target array you are using?.

Do you use NIC teaming?. If yes, could you check if the problem happen when there is only one uplink?.

Re: ESX 3.0.1 - Linux Guests go ReadOnly

tsightler — Mon, 23 Oct 2006 22:31:03 GMT

OK, today I decided to try patching the mptscsi driver to revert it's behavior to RHEL4 U2 and prior. So far, this seems to be working. I've run significant tests on several systems today, including a set of my most troublesome boxes, connected to a lowly AX150i, that previously would fail in 5-10 minutes. So far the boxes have survived the day without issue.

There have been several iSCSI timeouts issues on the hosts (as noted in the vmkernel logs) which would normally have been propagated as SCSI timeouts in the guest but with the patched driver the Linux system seems to just pause, and the return to normal operation.

I've posted the patched files and some crude instructions at http://www.tuxyturvy.com/blog/index.php?/archives/31-VMware-ESX-and-ext3-journal-aborts.html so others can give it a try if they want too.

I'm not willing to say this is 100% yet, perhaps the systems are just behaving today, but so far it looks good.

Later,
Tom

Re: ESX 3.0.1 - Linux Guests go ReadOnly

tsightler — Mon, 23 Oct 2006 03:50:03 GMT

I've been battling this issue for weeks so it's good to know I'm not alone.

In researching the issue tonigh I found the following bug which looks like it might be related:

https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=197158

It looks like this problem may be caused by a fairly recent (in the last few months) change in the upstream LSI Logic driver. For RHEL4 systems this code was changed between U2 and U3. I wonder if we could simply recompile the mptscsih driver from U2 for use in U3 and newer as a temporary solution. Has anybody tried that?

Basically, the newer driver adds an extra DID_BUS_BUSY status to a SCSI command failure of , MPI_SCSI_STATUS_BUSY which causes the SCSI mid-layer only tries 5 times before reporting a timeout to the upper layers. With previous versions without this extra return status the SCSI mid-layer would try indefinitely.

If this is really the issue it would be pretty simple to patch and compile a new mptscsih driver to temporarily work around this issue. Maybe I'll give that a try tomorrow.

Later,
Tom

Re: ESX 3.0.1 - Linux Guests go ReadOnly

Damin — Sun, 22 Oct 2006 23:56:03 GMT

I'm wondering if this fact is a "known issue" as in
"we will/must do something to fix it", or if it's an
expected behaviour.

From what I understand, when the SAN times out (Which SANS can/will do occasionally!) Vmware is sending a signal to the Guest. This signal can either be "Storage Unavailable "or "Storage Timeout". The guest storage drivers can react differently to these signals. Under Linux, the LSI driver freaks out and tells the OS that storage is unavailable, and Ext3 goes into read-only mode.

According to the engineer that I spoke with, this was fixed for Windows guests, but not yet for Linux. I'm not sure how they plan on fixing it.

Re: ESX 3.0.1 - Linux Guests go ReadOnly

manuel.wenger — Sun, 22 Oct 2006 07:21:41 GMT

I'm wondering if this fact is a "known issue" as in "we will/must do something to fix it", or if it's an expected behaviour.

Re: ESX 3.0.1 - Linux Guests go ReadOnly

Damin — Sun, 22 Oct 2006 00:20:31 GMT

#316658

BTW.. this was closed by Vmware because the "iSCSI Target you are using is not supported".

Never mind that the behavior occurs on HCL supported iSCSI Targets as well.

The basic cause of this problem, as explained to me by Vmware engineering, is that if a SAN timeout happens, the Vmkernel will send a message up to the guest reflecting the timeout. I was told that this is a known issue.

Re: ESX 3.0.1 - Linux Guests go ReadOnly

paithal — Sat, 21 Oct 2006 17:42:25 GMT

Can you post the SR number?. We will take a look at the vm-support logs.

Re: ESX 3.0.1 - Linux Guests go ReadOnly

Damin — Wed, 18 Oct 2006 01:30:40 GMT

1. Vmkernel log was provided to Vmware support.
2. Yes I have an SR#.

Re: ESX 3.0.1 - Linux Guests go ReadOnly

paithal — Tue, 17 Oct 2006 21:05:08 GMT

Who did you provide the vmkernel log?. Did you file an SR?. Do you have SR number?.

Re: ESX 3.0.1 - Linux Guests go ReadOnly

Damin — Tue, 17 Oct 2006 18:15:10 GMT

Can you see if there are any errors reported in ESX's
/var/log/vmkernel?.

Yep. And those have been provided to Vmware. When using the Software iSCSI initiator, the problem happens much more rapidly than using an HBA. This makes a lot of sense, as the HBA is interrupt driven, while the Software is system-load dependent. On a highly utilized Host, the Software iSCSI initiator is NOT going to be able to service the data as quickly as an HBA will.

So.. back to your question.. We see buffer overruns for reserve/release activity, and several iSCSI timeouts. The timeouts appear to indicate a networking issue such as dropped packets (none showd on either end, or on either port!!!), but we can account for every packet back and forth between the client and the target and there ARE NO DROPPED PACKETS!

We've even removed the switch from the mix and gone to crossover cables between the host and the target. Same problem.

Also, make sure network connectivity is healthy and
there are no intermittent failures. Path to storage
being down for extended amount of time (for RH it is
typically 1min, I guess) can take the FS read-only.
Which target are you using?.

IETD, specifically patched w/ Reserve/Release and several other patches that have been submitted back to the project as a result of 9 months of testing w/ Vmware.

Is this a shared network or dedicated to iSCSI only?.

Dedicated to iSCSI only.

Are any VLAN configuration in the picture?.

No.

Any
traffic shaping or bandwidth allocation restrictions
placed on the physical switch that is connected to
iSCSI?.

On a crossover cable? No.

Re: ESX 3.0.1 - Linux Guests go ReadOnly

Damin — Tue, 17 Oct 2006 18:10:08 GMT

No. This happens on a direct crossover cable (I.E. no switch) between the HBA and the iSCSI Target box. The ports indicate no errors, dropped packets etc.. so the underlying network layer (all Cat-5E 6 feet of it) is fine.

This might expose some
issues with the underlying iSCSI stack which are in
turn getting exposed to the filesystem ?

I suspect that it is related to how Vmware passes SAN timeouts to it's Guests. I was told by a Second Level support that Vmware is aware of the problem, that they have fixed it for Windows in 3.0.1, but that they have NOT fixed it yet for Linux. I strongly, strongly, strongly, STRONGLY disagree that this is a problem with the iSCSI target. In all of our testing over 12 months, we've never had this issue w/ any other Clients. Our iSCSI target is used by more than just Vmware here.

Might be
worth trying to get an idea of the health of the
network which you are sending your iSCSI i/o to see
if something is underperforming/failling under load.

The network is 100% clean. No dropped packets, no resends, no errors, no overruns. Nothing. It's clean.

I totally agree that this problem should not be
happening tho...

Re: ESX 3.0.1 - Linux Guests go ReadOnly

manuel.wenger — Sun, 15 Oct 2006 20:25:39 GMT

There's another thread discussing the same problem, in case you missed it:

http://www.vmware.com/community/thread.jspa?threadID=58081

Re: ESX 3.0.1 - Linux Guests go ReadOnly

CTeague — Sun, 15 Oct 2006 18:44:37 GMT

Wow, having this same problem I didn't even have to search to find another with it!

I'm running CentOS 4.3 (4x guests) on ESX 3.0 and 3.0.1 w/ VMWare Tools installed.

I experience the same exact read-only problem....I have had this happen on three occasions now and a reboot temporarily fixed the issue. The problem is that I don't have any heavy loads going to any of these guests (Apache web servers / mySQL servers, currently with little web traffic / db traffic as it's still heavy in development).

I'm using PE2850 (Dual Duo-Core 2.8ghz / 8gb Ram / 2x QLogic PCI-E HBA) on a Sun based SAN. The SAN is controlled by a higher technical department within my school so the details I can give right now are lacking.

Any ideas what could be causing this?

Re: ESX 3.0.1 - Linux Guests go ReadOnly

paithal — Fri, 13 Oct 2006 16:29:18 GMT

Can you see if there are any errors reported in ESX's /var/log/vmkernel?.

Also, make sure network connectivity is healthy and there are no intermittent failures. Path to storage being down for extended amount of time (for RH it is typically 1min, I guess) can take the FS read-only. Which target are you using?.

Is this a shared network or dedicated to iSCSI only?. Are any VLAN configuration in the picture?. Any traffic shaping or bandwidth allocation restrictions placed on the physical switch that is connected to iSCSI?.

Re: ESX 3.0.1 - Linux Guests go ReadOnly

jonhutchings — Fri, 13 Oct 2006 14:26:25 GMT

It's similar to issues with redhat scsi drivers in a multipath environment, where SAN switch reset confuse the mutlipath driver and cause the filesystem to flag a corrupt journal. Is it possible that under heavy load your iSCSI network is dropping packets or performing badly in some way. This might expose some issues with the underlying iSCSI stack which are in turn getting exposed to the filesystem ? Might be worth trying to get an idea of the health of the network which you are sending your iSCSI i/o to see if something is underperforming/failling under load.

I totally agree that this problem should not be happening tho...

Message was edited by:
jonhutchings

ESX 3.0.1 - Linux Guests go ReadOnly

Damin — Fri, 13 Oct 2006 02:45:32 GMT

Hello,
My hopes for the 3.0.1 release solving this problem have been dashed. Per the suggestion of Vmware support, we ripped out our Qlogic HBAs and went w/ software iSCSI and upgraded our 3.0.0 machine to 3.0.1.

Same problem. Under heavy load, the Linux guests go into Read Only mode on their filesystem.

Consoles indicate the following:
SCSI Error : <0 0 0 0> return code = 0x20008
end_request: I/O error, dev sda, sector 4928181
Aborting journal on device dm-0
ext3_abort called.
EXT3-fs error (device dm-0): ext3_journal_start_sb: Detected aborted journal
Remounting filesystem read-only.

All guests are using the LSI-Logic driver and have the latest Vmware tools installed.

This is very clearly a Vmware issue, and we've now proven that it IS NOT related to the Qlogic QLA4010 HBAS, as we migrated to Software iSCSI and are getting the same problem.