I have 1 VM that has a significant %lat_c/CPU Latency and it makes no sense.
I have found that CPU latency is made up of Ready, Swap Wait, and Power control issues. But all of these are under control:
No Swap wait at all, very low ready, and power management is fully off. (C-state and CE1 are both off in the bios and not reporting in esxtop) (Power is set for OS Control, esxi is set at High Performance and esxtop is reporting all CPUs at 100% P0 level) So it can not be power related. It is just this one VM. Not other VMs are showing this level of latency. Host totals are:
There are only 10 VMs on this host, 16 vCPUs, 704 worlds. The host is a dual Hex-core with hyperthreading (24 threads possible), 192GB of RAM. It is not the newest thing in the world, it is 2.5GHz Westmere processors. It is running ESXi 5.5 (without R1 so far). We have run it on a v55R1 host with the same results.
The VM is 80GB RAM and Dual proc averaging 65% to 68% CPU with a few spikes to 100%. So it really does need 2 procs and we are monitoring it to see when it will need a third. It is an SQL server and is using all 80GB of RAM and could actually use more. It is hammering the SAN pretty hard reading indexes (99% reads from SSD cache on the SAN). Active Memory is averaging over 70 GB. The entire hosts active memory is barely over 80GB. The other VMs are all pretty small.
What else could be causing 10% CPU Latency. Does this really mean that this VM is waiting for physical CPUs 10% of the time when it should be doing work? The performance of the MS CRM Dynamics application is not what we would like. We have been contemplating adding an SSD to the host to use vFlash Cache to bring the entire 350GB database to the host to offload the reads from the SAN, but while that would offload the SAN we don't really figure it will help performance of this VM that much since we are seeing nearly 100% of the reads coming from the SSD cache that is on the Nimble SAN.
Can anyone give me any idea what is causing this latency and how to resolve it.
I see that this thread has now been viewed 99 times, but not a single response with any idea of what this %LAT_C counter is all about. I now have another example that is even worse. This VM is a Windows iSCSI target server (VSAN) with %LAT_C numbers from 60 to 88% on the VMX process. CPU is about 40%. the total of the two is consistently 110% when the VM is under load. The screenshot below totals 113.9%. See the esxtop entry below:
Power saving by the Motherboard has been suggested in other threads as being a component of this counter. However, power has been optimized:
Both of the VMs in question are high disk I/O, and while there have been spikes in Disk Latency numbers, they are averaging 7 ms or less under heavy load. This is for the VMs accessing the targets on the VSAN that is shown in the ESXTOP entry above. The VSAN itself is getting access times to the physical SAN on which it resides averaging 2-7 ms with hardly any spikes at all. I am seeing more latency introduced by this VSAN than I would like. Performance is not what it should be.
Note that the $USED of the vmx process is equal to the %LAT_C. This is creating a high $USED across the board that appears to be bogus. Even causing alarms on the VM.
You can see that it is showing 100% utilization, generating an alarm, even though the CPU utilization is only 938 Mhz. It is interesting to not that the chart does not allow higher than 100% even though the actual total utilization that we see in esxtop is 110% between these two processes. The 938 Mhz correlates to the 39.92% vcpu-0 numbers in esxtop. The screenshots were not taken at the exact same moment so the numbers are not exact.
These numbers are no longer unique on this host. High %LAT_C is being seen on all the VMs on this host. Sometimes as high as 150%. This screenshot shows a couple in the 90% range:
Can someone respond?
The %LAT_C just doesn't make sense. It is 4:00 am and I have 3 VMs doing anything, the other 49 on the host are all idle. Total CPU 14GHz out of 100GHz available on the host. 80vCPUs between the 52 VMs on 80 logical procs. These 3 VMs are all showing High %LAT_C on only the vmx process. This screenshot actually caught it at 89.1%. High Disk I/O, but disk latency (all components) under 2ms. %LAT_C also shows up as %SYS and %USED on the vmx process and is rolled up into %Usage on the VM. So is it using or latency? The dictionary definition of Latency would indicate a waiting time, not a time doing something. %Usage is doing something. It is all bogus.
this is the same VM a few second before it spiked quite so high;
As you can see, if you look at the 0/1 cpus, the Usage in MHz of the VM is way higher than the total of its vCPUs. That is the %LAT_C being rolled up into the total usage numbers. It is warping all the reports and charts. We really need someone from VMware to tell us if this is real. Is the CPU actually doing something during this %LAT_C time on the vmx process? What? Is this an indication of a slowdown of the actual VM or not. If I look at the vCPU numbers I would have to say that it is not effecting performance of the running processes inside the VM. But I can't be sure unless I know what external services might be responding slowly due to this %LAT_C on the vmx process. I can't find any other counters that are indicating slowdowns. Just what is vmx doing? Why does it have %LAT_C and not any other processes?
This problem is still driving us crazy. At the time of these screenshots, I am down to 17 VMs on a machine with 40 cores (with Hyper-threading 80) and 512GB of RAM. Total vCPUs is less than 34. Total vRAM ~ 120GB.
Highlighted line on this graph is CPU Latency. Actually pegged at 100% for almost 10 minutes. Entire time is during high disk I/O as I was cloning a VM from one datastore on this virtual SAN to another.
As you can see by this graph and table below it, vCPU 0 maximum usage is 431 Mhz, but the CPU Latency was pegged at 100% for almost 10 minutes. And since Usage includes this Latency number, I am getting CPU alarms right and left on my VMs. As you can see from earlier posts, the cpu usage of the vmx process includes CPU Latency on that process and Total Usage of the VM is a total of usage on all the worlds.
As stated earlier, %Lat_C is coming from the vmx process. Actual vCPU utilization maximum is about 400 Mhz. Yet performance of this virtual SAN is terrible.
N%L is 100%, so vCPUs are not moving around, all data is local to the NUMA node. There is no reason for CPU Latency in this environment. Every vCPU has it's own core, let alone Thread.
