I guess this is the thread that just won't die!

I'm running a pre-production, test environment in my lab consisting of HCL SuperMicro boxes connected to a Wasabi Systems iSCSI SAN box. The purpose of this testing is to really put the Vmware multipath round-robin load balancing under the microscope and see how it performs.

My test bed consists of 2 SuperMicro boxes w/ 2 x Quad Core Intel boxes each connected via Gig-E iSCSI to a dedicated target NIC on the Wasabi box. The Wasabi box has 16 1 TB Sata drives configured in RAID-10 (it's a screamer!).

I am running 32 bit Centos 4 (latest release) VMs inside of the Vmware infrastructure. Kernel is the latest available from Centos (2.6.9-67.0.1.EL). Each VM is running w/ a single processor and 256 megs of memory, w/ a 4 Gig filesystem. I run 32 total VMs accross the two SuperMicro boxes.

I was under the assumption that the released kernel for the later Centos versions had the fixes to prevent this from happening in the event of a SAN failure, but I am still able to cause the guests to go readonly w/ the stock kernel. So I downloaded and applied Tom's patch, and verified that the driver is being loaded (rhel-vmfix in version string). However, I'm STILL getting timeouts even w/ the patched driver.

What I find particularly confusing, however, is that nowhere in the Vmware logs is there ANY indication of SAN timeouts. Everything looks clean as a whistle. Am I missing something really simple? Inquiring minds want to know..

Well.. now I'm posting again to the thread that just will not die.

I recently switched from running our production SAN on IET (Iscsi Enterprise Target) to the commercial Wasabi Systems iSCSI target. My production VMs are a handful of Centos 4.6 boxes, running the latest 2.6.9-67.0.4.EL kernels. It was my understanding that this contained patches that would prevent the ext3_fs aborts and resulting read-only issues.

I reviewed the code in the centos repository and saw that the Fusion Drivers had been updated to version "3.02.99.00rh". This appears to include the changes to the SCSI mid-layer to support conditional requeue of the timeouts. This patch appears to have gone mainline here: http://www.mail-archive.com/linux-scsi@vger.kernel.org/msg05338.html

After making the change to the new SAN, and updating the kernels on my Centos box, all looked good a while. Then we started getting 5-6 instances of boxes going ReadOnly a day. So I storage vmotioned everything off of the SAN and spent another week testing things to be absolutely sure that the SAN was "clean". I've not had any issues w/ Windows box, and I've checked every inch of the Network (switch ports, duplex settings, error levels, collisions etc..). I can get great performance from the SAN w/ very minimal latency. One caveat, is that I am using Vmware's Round Robin Load balancing w/ a pair of NICs on the servers, so I'm not sure if this plays into the issue.

Basically, I'm stumped on this one. Any suggestions? It looks like a SCSI timeout, but I can't see anything in the Vmware logs or the kernel logs to indicate it. The network is CLEAN and SOLID.. I can achieve 150MB / second sustained reads/writes from the SAN using the 2 Gig-E nics. When I was running on IET, I had NO such issues. The difference here seems to the be Wasabi target. Is there something that I'm overlooking?

P.S. And yes, this is ESX 3.5