Path thrashing with DS3300 and iSCSI hw initiators

[Stan01]

We have 2 ESX 3.5 U5 hosts - IBM x3650, connected to DS3300 (model 1726) using iSCSI hardware initiators - 2 Qlogic QLE406x HBAs on each server.

We are experiencing dramatically reduced disk performance of the VMs when both ESX hosts access the SAN. If all VMs run on just one of the hosts, the performance is fine.

While both hosts are accessing the SAN, there are a lot of errors in vmkernel :

Feb 9 07:03:18 MMMESX002 vmkernel: 1:09:18:50.547 cpu3:1052)WARNING: SCSI: 4634: Manual switchover to path vmhba3:0:2 begins.

Feb 9 07:03:19 MMMESX002 vmkernel: 1:09:18:50.974 cpu2:1154)StorageMonitor: 196: vmhba2:0:2:0 status = D:0x2/H:0x0 0x5 0x94 0x1

Feb 9 07:03:19 MMMESX002 vmkernel: 1:09:18:50.974 cpu2:1154)SCSI: 5420: vml.0200020000600a0b80003ad15c00000a6e48b1ec9d313732362d33: Requeuing command that failed during failover process.

Feb 9 07:03:19 MMMESX002 vmkernel: 1:09:18:51.024 cpu2:1129)StorageMonitor: 196: vmhba2:0:2:0 status = D:0x2/H:0x0 0x5 0x94 0x1

Feb 9 07:03:19 MMMESX002 vmkernel: 1:09:18:51.024 cpu2:1129)SCSI: 5420: vml.0200020000600a0b80003ad15c00000a6e48b1ec9d313732362d33: Requeuing command that failed during failover process.

Feb 9 07:03:19 MMMESX002 vmkernel: 1:09:18:51.062 cpu3:1052)WARNING: SCSI: 1091: Could not write sense data for vmhba3:0:2

Feb 9 07:03:19 MMMESX002 vmkernel: 1:09:18:51.062 cpu3:1052)WARNING: SCSI: 3259: Manual switchover(Force=0) failed. Trying forced SP failover for FAStT device vmhba3:0:2

Feb 9 07:03:19 MMMESX002 vmkernel: 1:09:18:51.595 cpu3:1052)WARNING: SCSI: 1091: Could not write sense data for vmhba3:0:2

+Feb 9 07:03:19 MMMESX002 vmkernel: 1:09:18:51.595 cpu3:1052)WARNING: SCSI: 3277: Manual switchover(Force=1) attempt 1 failed for FAStT device vmhba3:0:2, try again

for bigger part of the log, please see the attachment.

It looks like path thrashing, every time both ESX access the SAN. I called VMware and I know that DS3300 is not officially supported with HW initiators for ESX 3.5 U5 : http://www.vmware.com/resources/compatibility/detail.php?device_cat=san&device_id=3160&release_id=31

But I believe that this was working in the past and I think there is a big chance that this can be improved.

I was also reading the following posts :

http://communities.vmware.com/message/986306

http://communities.vmware.com/thread/115843

and I see that someone has successfully used HW initiators with DS3300 already.

Could you please give me some piece of advice, where could the problem be, what should I check next?

Thank you.