This document is a living, up-to-date version of the performance analysis methods whitepaper.
Host memory utilization represents the entirety of memory usage due to the VM and all tasks required by ESX Server to manage and provide control of the VMs. Using ESX Server's monitoring capabilities there is no visibility into improper usage of configuration of memory within the guest. Continue to use traditional monitoring tools in the guest to identify memory-hungry applications or shortages that lead to in-guest swapping.
As before, bring up esxtop to inspect system specifics. Hitting the ‘m' key will display the memory counters.
Once running, the following can be observed from the esxtop report:
Type | VirtualCenter | esxtop | Details |
Total memory size | MEMSZ | The is the amount of memory that the VM has been sized to. The VM will never get more than this but most of the time will be using far less than this amount due to sharing, ballooning, and swapping. | |
Memory target | SZTGT | The amount of memory that the kernel would like to provide to the VM. This number is calculated by on the guest's memory usage. When memory is over-committed, it may not equal the amount of memory that is actually provided due to ballooning and swapping. | |
Granted memory | mem.granted.average | The amount of memory that has been provided to the VM. Memory is not granted to the VM until it has been touched once. In the case of Linux, which does not zero out pages upon boot, a 4G VM will only be granted the small portion (100M or so) needed to run the OS until the OS or applications start to access more. | |
Touched memory | TCHD | The amount of memory (in MB) that has been "touched" (read from or written to) in the past X minutes. | |
Consumed memory | mem.consumed.average | The amount of machine memory allocated to the VM. For instance, a Linux VM might have been sized to 4G. Half of the pages may not yet have been used by the OS. Perhaps 1G of this remaining 2G can be shared. That leaves a consumed memory of only 1G. | |
Shared memory | mem.shared.average | Shared memory represents the entire pool of shareable memory. For instance, if two VMs each have 500M of identical memory, the shared memory is 1G. | |
Shared common memory | mem.sharedcommon.average | Shared common memory represents the footprint in machine memory as a result of memory sharing. For instance, if two VMs each have 500M of identical memory, the shared common memory is 500M. | |
Active memory | mem.active.average | %ACTV, %ACTVS, %ACTVF | The amount of memory (as a percentage of the entire host's memory) that has been used by the VM in the past sample period. %ACTVS and %ACTVF are slow and fast counters showing recent and long-term averages. |
Ballon driver usage | mem.vmmemctl.average | MCTLSZ | The amount of memory claimed by the balloon driver for us in other VMs. |
Swap rate | SWW/s SWR/s | The rates at which memory is swapped out (written) or in (read). | |
Swap Totals | mem.swapout.average, mem.swapin.average | These are cumulative amounts of swapping that has occurred since the VM was powered on. It's important to check if swapin and swapout are increasing, rather than just seeing if they are nonzero. Because if they are non-zero, it could be the result of swapping in the past, and not swapping at the present time. | |
NUMA migrations | NMIG | The number of NUMA migrations that have occurred since the VM's creation. | |
NUMA memory | NLMEM, NRMEM | The amount of the VM's memory that is on the local and remote NUMA nodes. | |
Overhead | mem.overhead.average | OVHD | The amount of memory required by the VMkernel to maintain and execute the VM. |
Memory analysis on an ESX Server means not just investigation of server-side statistics but also a solid understanding of the application that is running in the VM. When memory is short on the host, ballooning and swapping may be visible in esxtop, with swapping having a great impact on performance. When memory is short within the VM the guest will swap.
The prescriptive advice for memory shortages is fairly simple: use less memory or buy more. The following recommendations are variations on this theme:
One cannot fully optimize an ESX Server's memory without understanding the performance implications of page sharing. VMware's page sharing algorithm was presented at EMC World 2008 as resulting in a 2% increase in CPU load. But the benefits of page sharing have been demonstrated to provide overcommitment of memory safely to 2X and beyond.
The value of page sharing can be seen int the following counters:
esxtop | VirtualCenter | Description |
SHRD | memory.shared | The amount of memory in the VM that is sharable. |
SHRDSVD | No equivalent. | The amount of memory saved due to page sharing. |
No equivalent. | memory.sharedcommon | The size of the memory after redundant pages have been removed. |
Note that missing counters can be calculated using the other two. Shared memory minus shared common memory equals shared savings.
The top-level Performance Monitoring and Analysis paper.
The esxtop Performance Counters index.
The Understanding VirtualCenter Performance Statistics page.