Re: vmfs3 + iscsi = slowness

jackpal — Thu, 10 Apr 2008 14:28:27 GMT

I saw the same type of behavior a few months ago. In my case we only had one NSM160 to work with. Initially I used the NSM to host all vmdks but performance was just horrible. It tooks days literally to convert 5 vms and afterword the vms just were very laggy. The NSMs logs showed 100% usage most of the time so we were working it too hard. Also during the convertions we noticed some port errors between the switch and the NSM. Creating a channel on the switch fixed that but performance was still pretty bad. Since then I've load balanced the vms across with a RHEL box configured as a NFS/NAS device and things are better.

Things to check:

make sure there are no port errors happening between the NSM and the switch

make sure there are no port errors happening between the esx host and the switch

make sure the RAID array is not degraded

make sure the NSMs are up to v7. Check ESX HCL for this.

check the logs on the NSM for utilization. It does keep track of this data

vmfs3 + iscsi = slowness

nappyrat — Thu, 10 Apr 2008 03:14:28 GMT

ESX3.5, HP Proliant 460c, Lefthand iSCSI SAN

Short explanation:

It seems that I'm seeing an extreme slowdown only when it comes to a vmfs3 volume running on my iscsi san. vmfs on local storage, iscsi volumes that aren't vmfs3, and mixes of the two are fast.

Long Explanation:

I was originally benchmarking vm performance. We have them running on a lefthand iSCSI SAN with 6 lefthand nsm160 units. The SAN screams with non-vmware servers, but we noticed that our vms were getting horrible iometer results, with read/writes around 20-30MB/s.

If I storage vmotion the vm to the local disk of the blade servers that are the host, the vm's disk I/O speeds up a lot to like 50-60MB/s. So originally I thought it was a network issue with bladecenter we're running our hosts on or soemthing like that. Timing the storage vmotion, or disk cloning from local storage to the san which uses vmkernel showed the similar 20-30MB/s speeds.

So for the heck of it I decided to test the iscsi performance using the guest's software initiator to access a non vmfs3 volume. So I have a linux vm running on local storage of the blade server (vmfs3), and it uses open-iscsi to mount a volume on the san which is formatted with ext3. doing tests with dd, file copies, etc... I was able to see the 50-60MB/s I see with other non vm servers and local storage. So the guest OS is getting 2x the performancde on the san as the host is.

However, this is only when it's located on the local storage of the blade. I once again storage vmotion'd the vm back to a vmfs3 san volume and booted it up. Once again I used the software initiator to mount an ext3 volume. So now the vm is running from the san, and is mounting another volume on hte same san. Now all my tests are back down to 20-30MB/s.

So I am only seeing decreased performance when it comes to reading or writing with a vmfs3 volume on the iscsi san. The virtual machine can get good speed with vmfs locally, and good speed with the san if it's an ext3 volume, so why can't it get good speed with a san volume thats vmfs?

VMware Communities: Message List - vmfs3 + iscsi = slowness

Re: vmfs3 + iscsi = slowness

vmfs3 + iscsi = slowness