vSphere Performance Issues - Disk Latency

[huntercr]

Hi All,

First time poster here..  Just got a question regarding our vSphere setup and some performance issues we are having.  It seems to be a storage related issue, and with no storage background I thought I would ask the experts here.

We have a EMC CX4 SAN with 2 Raid 5 LUNs setup on FC disks, and 2 Raid 5 LUNs setup on a SATA expansion unit.  I wasn't involved in the setup of this environment, and in my previous role the setup was different and we had no performance issues.

After a little research I have narrowed the issue down to the 2 SATA LUNs having high read latency.  Both LUNs are averaging approx 70ms disk read latency and physical device read latency, with maximums of around 1100ms.  The FC LUNs have averages of around 4ms, with maximums of around 25.

From what I understand, the types of data being stored on the SATA drives should not be anything which has high disk I/O (so no databases, log files, etc).  I am trying to organise the VM's better to enable better use of the FC storage space and use SATA for less I/O intensive applications such as backups, archives, file serer, etc.  Presently the virtual disks are spread out across the FC and SATA LUNs with little consistency (ie system drives for different VM's on both, database drives on both, etc).

Anyone have any advice/guidelines I could look at to see how things should be setup?  And is this NORMAL for this type of setup? Could there be any hardware issues?  Is there any other tools I could use to monitor/assess the situation to find the exact cause?

Thanks in advance.

[chriswahl]

70ms is a bit high (I'm averaging 7ms read, 1ms write on my busiest SATA volume). There's a lot of ways to attack this problem. Thin provisioned disks causing "hot zones" on the SATA LUNs, misaligned disks (typical with Windows Server 2003 and older), etc. Too much to really go in to in the VMTN - I'd check with your storage admin or vendor.

I'd pop the hood and check out performance in esxtop for a better idea as to where the latency is occuring. This esxtop community article is a great place to start. Specifically, Section 4.2.2 Latency Statistics. It will help isolate where exactly the latency is taking place (examples being at the guest, kernel, or disk level).

[huntercr]

Thanks Chris, I will check out some of those latency counters in esxtop and see what it comes back with.

Edit: Ok, I have contacted EMC and they are going to be looking at the log files for me to see if they can see any issues on the SAN side.