High latency on VMs but nothing odd in ESXTOP

[SyApps]

I've noticed some spiky latency on some of my VMs. Below is an image of the performance graphs as well as a quick look at esxtop, the red line being the VM. Is this at a level where I should be concerned? I'm surprised there's latency at all. This is happening with quite a few of my production VMs. Any help would be appreciated. Thanks!

vSphere Version: 5.5

Cores for this VM: 2:2

No resource pools configured. There just isn't enough demand imo.

Here's the VM graph. Sometimes latency can be as high as 30% throughout the day:

Thanks in advance!

[sxnxr]

For me 95% of all latency on a cpu has been the power policy on my physical hosts. Check to see if they are set to balanced and if so change them to high performance (making sure it wont cause any rack power problems)

I cant realy tell for you screenshots as i can compare the two. ESXiTOP is real-time and the graph is not.

When the latency happens you relay want to look at ready%, contention and co stop at the same time.

[SyApps]

Thanks for the reply. This weekend I was able to bring the hosts down and change the power settings to "Operating System" as I read somewhere else in the community that this was something someone did and it helped. I then hopped into each host and changed the power management settings to High Performance. I guess it has only been one day in production, but it does seem as though latency has dropped to about 5-10% for some VMs. Others are still up there. I'm wondering if this helped or not. Proper metrics gathering can be so fleeting.

[daphnissov]

Changing your physical hosts' power policy to OS Controlled and leaving the ESXi setting to Balanced is what's actually going to give you the best performance. It allows ESXi to use its own internal scheduling more effectively and will further reduce some of your latency numbers.

[sxnxr]

I have to disagree. High performance will provide more absolute performance over balanced.

Taken from the 6.5 resource management guide

Balanced (default) The vmkernel uses the available power management features conservatively to reduce host energy consumption with minimal compromise to performance

High Performance The VMkernel detects certain power management features but will not use them unless the BIOS requests them for power capping or thermal events

Even tho it is minimal there is still performance impact using balanced. I would always recommend using high performance if power usage is not a problem

I have seen in our environment a reduction of ready from 35% to less than 5 and contention reduce from 30 to less than 2% just by changing from balanced to high performance. Again each environment is different

[daphnissov]

This is a subject for some debate and considerable "your mileage may vary" statements. In general, though, I'm talking about setting the BIOS power policy of a given ESXi host to be OS Controlled. Vendors call this different things. From there, changing the vSphere host power policy to either balanced or high performance. The benefits you will receive depend on several factors including workload type. To learn more about this fascinating subject, I'd recommend reading Frank Denneman's and Neils Hagoort's book VMware vSphere 6.5 Host Resources Deep Dive. They dedicate a chapter solely to host power management. Specifically, read pages 187-194 which discuss the pros and cons of the various ESXi-defined power policies.

[sxnxr]

apologises syapps for hijacking your thread

I am curious to why you would set the power policy to balanced (OS controlled or in the BIOS) if you dont have any power constraints in your datacenter. Would you not get more benefit having all set to high performance regardless of the workload?

Granted some workloads wont care if it is balanced or high performance and some would care if it was balanced and not high performance would it not be better to have at high performance and cater for all workloads. Lets be realistic who knows what workload requirement would benefit from in there environment for all VMs.

I am genuinely curious and not trying to prove you wrong (and dont want to buy a book) If i have no power constraints in my datacenter why would i go balanced? I ask because we just change 400 hosts to high performance last month.

[daphnissov]

Just to be clear, setting the BIOS power policy to "Balanced" or some other vendor equivalent is not the same thing as setting the vSphere host power policy to "Balanced". Regardless if you choose Balanced or High Performance inside of vSphere-which is always the best practice-you should set the BIOS power policy mode to "OS Controlled" or whatever the terminology is. From there, balanced or high performance will provide different results based on the workloads running on hosts in that cluster. For example, if you're running vSAN or low-latency applications such as stock or trading applications, you'll probably want to stick with high performance. If your workloads are mixed in set and you don't know how those applications behave, it's best to leave it at balanced. It's important to note that balanced always strives to achieve performance over power economy and not the other way around. As an example of how the balanced profile may potentially provide better performance (which is subjective to begin with) over the high performance profile, the authors ran a series of tests using the industry-standard DVD Store 3.0 benchmark software. In random test results, they found that setting the host power profile to balanced resulted in an average of almost 2,300 more orders processed during the 5-minute test period. Understand that this isn't conclusive evidence, but it shows that, again, depending on workload, balanced may offer better and more predictable performance in some cases. The only way you'll know for sure in your environment is to test between the two. This is another benefit for having the BIOS policy set to OS Controlled and configuring the host power policy inside ESXi/vSphere is that it can be changed on the fly, even programmatically, without downtime.