VMware Cloud Community
VMwareMessage
Contributor
Contributor
Jump to solution

The storage I/O latency too high,it will affect the virtual machines cpu usage?

Our storage due to the recent utilization , resulting in latency too high.

Average latency is between 42216~1463757 microseconds.(The SAN  environment)

I found that the increase or decrease of each delay time will affect the utilization rate of CPU.

Sometimes the latency too high,it will cause the virtual machine crash!

I don't found the VMware KB or document to narrations this reason.

What is the reason for this?

Is there a description or document for how long the Storage latency too high will result in system instability?

(PS.Our storage for a long time in a state of high latency. Maybe average latency is between 100000~300000 microseconds.)

1 Solution

Accepted Solutions
jhague
VMware Employee
VMware Employee
Jump to solution

‌Basically if your VM is not getting responses back quick enough from the storage it will queue requests and the CPU will have to work harder to manage these queues. The queue can only handle so many requests itself so you will potentially see instability as requests are frozen or dropped or whatever. There are queues right along the path from OS, drivers, ESXi kernel, HBAs, storage processors etc. so with a big issue you can observe problems in multiple areas. There is a KB which explains how to troubleshoot- Using esxtop to identify storage performance issues for ESX / ESXi (multiple versions) (1008205) | V...

Your latency figures look very high!

John Hague http://linkedin.com/in/john-hague | twitter @jhague10 VCIX-DCV | VCP-DCV 3/4/5/6 | VCP6-NV | VCP7-CMA | VCAP7-CMA Design

View solution in original post

1 Reply
jhague
VMware Employee
VMware Employee
Jump to solution

‌Basically if your VM is not getting responses back quick enough from the storage it will queue requests and the CPU will have to work harder to manage these queues. The queue can only handle so many requests itself so you will potentially see instability as requests are frozen or dropped or whatever. There are queues right along the path from OS, drivers, ESXi kernel, HBAs, storage processors etc. so with a big issue you can observe problems in multiple areas. There is a KB which explains how to troubleshoot- Using esxtop to identify storage performance issues for ESX / ESXi (multiple versions) (1008205) | V...

Your latency figures look very high!

John Hague http://linkedin.com/in/john-hague | twitter @jhague10 VCIX-DCV | VCP-DCV 3/4/5/6 | VCP6-NV | VCP7-CMA | VCAP7-CMA Design