I have 4 ESX 4.0 hosts running around 100 VMs with vCenter and have been experiencing performance issues for a couple of weeks.
All VMs are very slow during certain period about 1 or 2 hours every day, and it changes sometimes.
After having analysized with SAN tools, esxtop tools and some scripts inside VMs, I identified the problem. It's about High Disk Avg. Latency on a Dell PS6000E Storage.
the Latency pattern presented in esxtop tools matched SAN Utility tools, sometime with high IOPS, sometime with high MB Read/sec and MB Written/sec, but not easy to identify which VM or group of VMs caused the problem.
I wrote a script on every ESX host, collecting esxtop every 10 seconds for 1 hour whenever a High "Physical Disk\(vmhba36\)\\Average Driver MilliSec" detected.
And, now I found out when it becomes slow, the output of esxtop got something interesting every time. I awk the output for Human read, only taking several measurements.
Machine Commands Reads Writes Read MBs Written MBs Latency ESX Host
VM-name1 5462836927972282368 5462836927972308992 5462836927972466688 5.20977e+12 5.20977e+12 17318.8 ESX01
VM-name2 3518437297766401024 3518437297766401024 3518437297766402048 3.35544e+12 3.35544e+12 12120.1 ESX02
helper 7.04 2334.75 2.89 2331.87 0.01 103.46 ESX01
VM-name3 49458.6 41820.9 7637.62 689.17 274.14 722.07 ESX03
VM-name4 17584.6 11340 6244.58 463.47 342 12301.6 ESX04
............
Does anyone see this before?
