iSCSI horrific slowness - how to benchmark?

[amills]

We implemented a LeftHand NSM 2060 a few months ago in a 3 server ESX 3.5 cluster and have been seeing horrific performance ever since. I've been testing network bandwidth through hdparm on a linux VM, and when things are bad I see around 100KB/sec on buffered disk reads. This is usuallt accompanied by tons of LUN contention issues in the logs.

I've gone through and implemented all of the best practices ( < 10 VMs per LUN, NIC teaming on the LeftHand). I'm curious, though, is there a way to gauge the actual throughput each ESX box has to the LeftHand. I'm trying to figure out exactly how much bandwidth I'm using. The best data I get right now is file copy tests on VMs, but with ~20 VMs per ESX host, I can''t account for any IO the other VMs are using. Can I get this info at the ESX level?

Any advice is appreciated.

[Rob_Bohmann1]

Have you tried running esxtop in the service console?

[amills]

I've run esxtop, which shows high IO on many of the VMs. It gives me an idea of which VMs are using a lot of resources, but I haven't yet been able to get a picture of how much bandwidth I'm using. Is there a similar utility for this other than just looking at the performance graphs in the GUI?

[anujmodi1]

Before doing anytesting on the iSCSI, You have verify the firmware versions of the ESX host server and iSCSI box. Most of the performace issues are taken care by upgrading the latest version as they will fix most of the issues.

If you want to verify the benchmark on the iSCSI, just present a LUN on the ESX Server double the size of VM and check that if you still see the slow performance. Also verify that if you see any logs related to iSCSI in vmkernel.

Regards,

AM

PS: If you find this information useful, please award points for "correct" or "helpful".

[Rob_Bohmann1 · Accepted solution]

esxtop can show you MB/s received and transferred on each vSwitch and individual nic. Type n to get to network stats. From that you can get some idea of how much bandwidth is being used. Its real-time so to get a more friendly display or possibly a chart, you could check with your network and /or storage people to see what they can see and share in terms of bandwidth usage on the switch port and interface on the storage. You could also (temporarily) increase your virtual center logging level to see if the graphs there provide the information you are looking for.

Have you reached out to the storage vendor? I am not familiar with them, do they have some performance monitoring tool for the actual array to look at?

I know you can add fields in the disk stats display to not only see throughput stats but also latencies. Check the help. I don't remember if there are latency metrics for the network stats.

There are 3rd party monitoring tools out there you maybe be able to demo/trial them to see if they help you determine where the problem lies. Good luck!

[JDLangdon]

Can you provide an overview on how you have the environment configured?

________________________________

Jason D. Langdon

[amills]

I've got 3 ESX boxes in a cluster. 1 dedicated NIC each to the LeftHand. The LH is presenting 7 luns to the cluster with 3-9 VMs per lun (as per recommendations).

Running esxtop in disk mode seems to provide a pretty good sense of how much data is being read/written to the disk (I guess it helps to RTFM). I'm still seeing horribly slow performance due to lun reservation issues between the 3 ESX boxes, but it does help give me a sense of how much bandwidth I'm pushing between each.

[JDLangdon]

Are you using hard or soft iniators on ESX? Are you using iniators in your VM's?

________________________________

Jason D. Langdon

[amills]

We're using the software initiators. All 7 luns are setup as VMFS volumes on

the cluster, the VMs are just kept on each data store.