VMware

This Question is Possibly Answered

1 "correct" answer available (10 pts) 2 "helpful" answers available (6 pts)
1 Replies Last post: Nov 6, 2009 10:34 AM by Scissor  

Device latency - EVA8000 posted: Nov 4, 2009 1:53 PM

Click to view houghtp's profile Enthusiast 144 posts since
Jan 22, 2008
Hi,

We've recently changed our VMFS layout and are seeing higher latencies on the new VMFS from an ESX standpoint..

We used to have out datastores layed out so each VM's VMDK's were on a seperate datastore. we decided to change this as we had reached the maximum number of paths to the SAN and also to simplify things such as backups. We now have all disks for each VM in a smaller number of larger datastores (1TB). We have apporx 20 VM's per datastore which is less than we had in the datastores containing OS vmdk's in the previous design, which was up to approx 25.

We are now seeing latencies averaging from 10ms up to 40ms with spikes upto 300ms. I wanted to know why this would be. I'm not seeing any storage related errors in vmkernel log such as reservation conflicts or aborts. I've checked the queues on the HBA's and they seem fine. Is it just the fact that we have consolodated more I/O onto larger LUNS? The thing thast confusing me is the Disk Group on the EVA (spindle count) and the load on the VM's hasn't changed. The one thing we haven't doen is load balance the storage paths but we hadn't done that in our previous design and we had no latency issues.

Thanks in advance

Re: Device latency - EVA8000

1. Nov 6, 2009 10:34 AM in response to: houghtp
Click to view Scissor's profile Master 1,249 posts since
Oct 8, 2007
houghtp wrote:
Is it just the fact that we have consolodated more I/O onto larger LUNS?

I think this might be it. I believe that on SANs, ESX by default allows 32 outstanding I/Os per LUN (you can raise that up to a max of 64 if you want to). So if you have many VM's all performing IO to the same LUN you will run into Guest IO slowness. You can verify this by running esxtop -a, d, u from the console and viewing the DQLEN, ACTV, QUED, and %USD columns.

I have a IBM DS3400 SAN and believe I ran into this problem when I started with 1 TB LUNS. Guest IO was horribly slow even though the SAN didn't show any problems. I am now in the process of moving Guests over to 200 GB LUNs in order to spread the I/O load. Easy to do with Storage VMotion -- too bad I can't use Storage VMotion on my Lab Manager setup.

VMware Developer

SDKs, APIs, Videos, Learn and much more in the Developer community.

Learn More

Developer Sample Code

Increase your developer productivity with VMware API sample code.

Learn More

VMworld Sessions & Labs

Online access to the latest VMworld Sessions & Labs and online services.

Learn more

Purchase PSO Credits Online

Purchase credits to redeem training and consulting services online.

Buy Now

Community Hardware Software

View reported configurations or report your own.

Learn More

VMware vSphere

Come witness the next giant leap in virtualization.

Register Today

Communities