Can you see if there are any errors reported in ESX's
/var/log/vmkernel?.
Yep. And those have been provided to Vmware. When using the Software iSCSI initiator, the problem happens much more rapidly than using an HBA. This makes a lot of sense, as the HBA is interrupt driven, while the Software is system-load dependent. On a highly utilized Host, the Software iSCSI initiator is NOT going to be able to service the data as quickly as an HBA will.
So.. back to your question.. We see buffer overruns for reserve/release activity, and several iSCSI timeouts. The timeouts appear to indicate a networking issue such as dropped packets (none showd on either end, or on either port!!!), but we can account for every packet back and forth between the client and the target and there ARE NO DROPPED PACKETS!
We've even removed the switch from the mix and gone to crossover cables between the host and the target. Same problem.
Also, make sure network connectivity is healthy and
there are no intermittent failures. Path to storage
being down for extended amount of time (for RH it is
typically 1min, I guess) can take the FS read-only.
Which target are you using?.
IETD, specifically patched w/ Reserve/Release and several other patches that have been submitted back to the project as a result of 9 months of testing w/ Vmware.
Is this a shared network or dedicated to iSCSI only?.
Dedicated to iSCSI only.
Are any VLAN configuration in the picture?.
No.
Any
traffic shaping or bandwidth allocation restrictions
placed on the physical switch that is connected to
iSCSI?.
On a crossover cable? 
No.
Wow, you sure ask a lot of questions!
What kind of test you are doing to get to the read-only situation?.
Basically, anything that eventually leads to a timeout on the ESX host. For the most part this simply involves running disk intensive benchmarks on multiple guests. We've been using iozone, but I would suspect you could use almost anything that generates a lot of I/O.
I assume the boot disk is on iSCSI volume.[/b]
No, boot disk is on locally attached RAID storage (Dell PERC 4)
Do you use virtual disk in VMFS volume or RDM for guest's boot disk?. If using VMFS, do you perform any I/O's to VMFS volume apart from booting (and guest doing I/O's)?.
We use a virtual disk in a VMFS volume. Yes, of course there are I/O's other than booting since the entire virtual disk for the guest is on the VMFS volume.
Are you sharing the LUN with any other server?.
I can't remember, but did you mention which target array you are using?.
One other ESX server.
Do you use NIC teaming?. If yes, could you check if the problem happen when there is only one uplink?.
Yes, but the problem will happen even with a single uplink.
You seem to be under the impression that the problem can only happen with iSCSI, however, we have reproduced the issue with locally attached RAID, and with fiber channel attached storage. Basically, if you can genereate enough I/O that you get an Async I/O error, the chances of seeing your ext3 volumes go read only is high.
If you see a message like this in your log:
Oct 22 01:28:19 esxhost1 vmkernel: 13:20:19:56.753 cpu3:1028)SCSI: 3731: AsyncIO timeout (5000); aborting cmd w/ sn 399366, handle b78/0x6a028a8
You are probably going to have a problem. With the small change I made to the LSI Logic driver the linux systems seem to be able to survive these timeouts.
Later,
Tom