Short explanation:

It seems that I'm seeing an extreme slowdown only when it comes to a vmfs3 volume running on my iscsi san. vmfs on local storage, iscsi volumes that aren't vmfs3, and mixes of the two are fast.

Long Explanation:

I was originally benchmarking vm performance. We have them running on a lefthand iSCSI SAN with 6 lefthand nsm160 units. The SAN screams with non-vmware servers, but we noticed that our vms were getting horrible iometer results, with read/writes around 20-30MB/s.

If I storage vmotion the vm to the local disk of the blade servers that are the host, the vm's disk I/O speeds up a lot to like 50-60MB/s. So originally I thought it was a network issue with bladecenter we're running our hosts on or soemthing like that. Timing the storage vmotion, or disk cloning from local storage to the san which uses vmkernel showed the similar 20-30MB/s speeds.

So for the heck of it I decided to test the iscsi performance using the guest's software initiator to access a non vmfs3 volume. So I have a linux vm running on local storage of the blade server (vmfs3), and it uses open-iscsi to mount a volume on the san which is formatted with ext3. doing tests with dd, file copies, etc... I was able to see the 50-60MB/s I see with other non vm servers and local storage. So the guest OS is getting 2x the performancde on the san as the host is.

However, this is only when it's located on the local storage of the blade. I once again storage vmotion'd the vm back to a vmfs3 san volume and booted it up. Once again I used the software initiator to mount an ext3 volume. So now the vm is running from the san, and is mounting another volume on hte same san. Now all my tests are back down to 20-30MB/s.

So I am only seeing decreased performance when it comes to reading or writing with a vmfs3 volume on the iscsi san. The virtual machine can get good speed with vmfs locally, and good speed with the san if it's an ext3 volume, so why can't it get good speed with a san volume thats vmfs?