Appear to have poor storage performance. Best test(s) to confirm?

[MLCatRDP]

I am not certain, but I think that I am having performance issues with my storage array.  My current environment consists of 2 Dell R510's as my ESXi hosts, and an Equallogic PS6100XV.  My data-store is configured as raid 50 across (12) 10active/2standy 15K 600GB SAS drives.  iSCSI is running across Dell PowerConnect 6224, on it's own vlan and the only other traffic on the switch is vMotion.  Jumbo Frames enabled, Flow control enabled, paths are set as Round Robin.

When I am running similar operations on one of my live vs test servers, performance is much more favored on the test server.  The operation I am basing this performance question on is a database file rebuild (Sage 100/MAS 90).  On my live server (in virtual environment) the operation takes about 5 hours.  On my test server (PE2950) it takes about 30 minutes.  My test server is running on (4) 120Gb 0K scsi drives in raid 5.  The Live server has 28GB more memory (memory is faster too), and 4X more cpu's.  So the only thing I can think of that could possibly slower is the disk speed.

When I run the file rebuild and watch resource monitors, I can see the disk IO stay pretty consistent at 10MB.  Sometimes with will spike to 15/20MB.  On my testserver I see it stay pretty consistent at 40MB, with spikes up to 120MB.

So after all that big story, I guess what I'm asking is what kind of tool(s) should I use to verify the performance on my virtual server's data-store?  What kind of throughput should I be expecting with my hardware?

[memaad]

Hi,

Can check this on your physical ESXi host    http://kb.vmware.com/kb/1018206

Regards

Mohammed Emaad

[MLCatRDP]

Both of my hosts were set to balanced, so I changed to High Performance.  I will rerun my database operation and report back.

[JoJoGabor]

On your array check the throughput and IOPS going to every physical disk - are these getting close to the mximums?

on array check CPU of controllers?

Check throughput of iSCSI NICs on your VMware host.

VMotion the VM in question to another host - does it change?

[MLCatRDP]

No noticeable different in performance with the power settings changed between Balanced & High Performance, +/- within 1%.

Here is what CrystalDiskMark is telling me:

[MLCatRDP]

JoJoGabor, is this what you were looking for?  I'm new the the Equallogic so I'm not sure how to get all the results you're looking for.

[memaad]

HI,

I would like to know if you have performed the actual task and check how much about of time it takes to complete. Also I would like to know what you see in esxtop when you perform the actual task. Need to see CPU and Disk latency.

Regards

Mohammed

[memaad]

Hi,

What is PSP you are using in this host.

Regards

Mohammed Emaad

[MLCatRDP]

I'm not familiar with esxtop or psp.

On my live server, the task takes about 5 hours.

On my test server, the task takes about 30 minutes.

[MLCatRDP]

I started running Iometer.  It appears my IOPS are ok, but I'm surprised my MBs/second is as low as it is.

Does this look correct?  How does this compare to other san implementations that are similar to mine?

[JoJoGabor]

PHysical disks on array are running fine. Check your controller load on equalogic now.

NEtwork seems ok if you're running 1gb, yoscan get 120MB/sec max which you have on sequential. It's likely to be storage array, but also check vmkernel.log (does this still exist?) for any scsi errors.

[MLCatRDP]

I don't seem to know where to find the controller load on my equallogic, Can you point me in the right direction?

I am running 1Gb network.  But I have 4 connections on each controller, and each ESXi host has 4 iscsi connections so shouldn't the throughput increase?

[JoJoGabor]

IN general terms, no, don't expect to get more than 120MB/sec throughout for a single disk on a VM. You can do clever stuff like manually balancing paths to different data stores to get increased throughput for a VM with multiple disks, or some other settings, but you have more fundamental problems here. I don't know equal pic arrays so can't advise on that. If you have local storage on your host, as a test you can storage vmotion your vmdk to that to see if throughput increases, proving nets storage array or network.

[mcowger]

"IN general terms, no, don't expect to get more than 120MB/sec throughout for a single disk on a VM."

I'm sorry, but this is just stupid.  It is TRIVIAL to get better performance than that from even the most basic enterprise arrays.  Heck - my company demonstrated over 10GB/s to a single VM 2 years ago.

[Josh26]

mcowger wrote:

Heck - my company demonstrated over 10GB/s to a single VM 2 years ago.

On a 1Gb network?

[JoJoGabor]

He's using a 1Gbps network, hence the 120MB/sec, assuming a basic setup

You got 10GB/sec to a VM or 10Gbps?

10GB/sec is very impressive if so.

[mcowger]

No - of course not.

[mcowger]

When I originally wrote my reply, the user hadn't specified that he was on a 1GbE network.

We got 10GB/s to a single host, running 4 VMs.

[MLCatRDP]

Sorry in my first post I mentioned my switches model but didn't specify Gbe.

I'm happy with the IOPS results (5000+)

I am still baffled on my throughput.  using iometer the most throughput I can get is about 75 MB/s (combined read/write).  60% of the theoretical limit.  Why am I losing 40%?

[tomtom901]

Did you enable Jumbo Frames on the switch, the vSwitch and the port group? The SAN itself runs at 9000 MTU but if your physical switches, virtual switches and port groups segmenting of the packets will occur.  I've still got a whitepaper from Dell on how to configure an Equallogic array with VMware. If you're interested, send me a PM, I'll be sure to send it to you. Perhaps you can review the configuration then.