ESX 3.5 Update 3 - Slow SAN performance via Qlogic 2460

[IDC]

Hardware:

IBM 3850M2 40 Gb

Qlogic 2460 x 2

IBM DS4700

We are experiencing poor performance of our SAN when used with ESX 3.5 Update 3 (same issue with Update 2). The performance is poor in both the VMWare Console and VM's themselves. Poor performance is relative to any other OS. By way of example, I can dd a 2 Gb file in ESX (or any Guest VM) in about 30 seconds. If I boot into a Gentoo LiveCD/Knoppix, you name it (with NO tweaking of the qlogic driver AT all), then the same write to the same logical drive is completed in less than 13 seconds. Read performance has a similar spread.

I have duplicated this on other machines with different HBA's. I have peformed the test when I am the only user of the entire SAN and and all machines connected to it.

Thank you for any assistance.

[okeedokee]

Have you disabled Auto Volume Transfer on the storage processors? See page 60 of the following.

Fibre Channel SAN Configuration Guide

Regards

[okeedokee]

The following will tell you if disabling AVT will help...

*

IBM TotalStorage DS4000 and Path Thrashing

*

When path thrashing is detected on a DS 4000 or compatible SAN array, the following warning is logged to the vmkernel log.

FAStT SAN is path thrashing with another system. Check AVT setting.

.

[bobross]

What virtual FC HBA are you using inside of ESX? This can and does make a difference...several previous threads on it.

[IDC]

As per the IBM docs, the Host type is set to LNXCLVMWARE which is supposed to disable AVT automatically.

[IDC]

bobross:

The ESX host itself is slow to read/write on the SAN, not just VM's that are hosted on it.

I can boot into the Console, access a dedicated logical drive that is only available to this specific HBA and create/format an ext2 partition. Using the exact same partition for both the Console and Gentoo (for example), the console (and any VM's hosted on it) is slow to read/write from the disk, and the boot liveCD is fast.

The VM's do not interface directly the Qlogic cards, but only via the Buslogic SCSI device that the ESX hosts provides for them.

Thanks

[JBradford]

Not that this helps any but I'm having an almost identical problem with my Gen1 IBM 3850s that are on an IBM DS4300. It seems like most of the time it's all I can do to get desktop type performance from my SAN within my VMs as well as from the console. The thing is, whenever I do any sort of file copy from one VMDK to another within a VM (the VMDKs are on different LUNs) my CPU load within that VM will hit and stay at 100% for as long as the copy is running. And that copy will run about as quick as what I'd expect from an old laptop...