Introduction
A common question that crosses my desk is, "how do I analyze and correct performance?" There are a variety of techniques for doing this and many sources of information spread around the interweb, so I'm going to collect a few thoughts here.
Monitoring
Guest-based performance monitoring is an inaccurate and unhelpful means of evaluating performance in virtual deployments. See Guest-based Performance Measurement for more inforamtion. Monitoring and analysis of VMware ESX Server should be performed with esxtop and VirtualCenter.
esxtop
esxtop is the tried-and-true means of collecting every performance stat needed and making it available in a way that is conducive to analysis. The best source of information on launching esxtop can be found in the Resource Management Guide (page 159). It's worth nothing that the directions in that guide came from ESX Server 3.0.1. Since then, the developers have been kind enough to simplify the process of including all performance counters. This is done with the "-a" switch. So,
esxtop -a -b > analysis.csv
runs esxtop in batch mode and prints all performance counters. Let me give you the quick "do-s" and "don't-s" of esxtop:
- With esxtop on ESX Server 3.5 and newer, always include the "-a" option to display all counters.
- With esxtop on versions of ESX Server prior to 3.5 always follow the resource management guide to enable all counters. The storage latency statistics, for instance, are not displayed by default!
- Always start your VMs before running esxtop in batch mode. If you start them after starting "esxtop -b", then esxtop will only produce VM data based on the VMs that were running at the time of its start.
VirtualCenter
VirtualCenter doesn't provide all of the performance counters you might need to analyze performance. But, it provides more than you might think! The default setup for VirtualCenter performance counter collection is fairly minimal. But this can be expanded by reconfiguring VC's performance counter collection. This is done as follows:
- From the VI client, Administration->VirtualCenter Management Server Configuration...
- On the left, click "Statistics".
- Now increase the stats level as you see fit.
More information on VC performance counters and archival can be found in the Understanding VirtualCenter Performance Statistics wiki article. A few things are worth considering before going crazy with this:
- You probably will never need level four stats. Those are mainly for debugging.
- Use the DB size estimator after you monkey around with these levels. The DBs can get quite large and you want to know this before it happens and your VC system performance suffers.
Analysis
Performance analysis techniques were previously detailed in the performance analysis whitepaper. That has been migrated to this wiki and new material will appear here.
The following pages will provide guidance for identifying and correcting performance problems on ESX-based systems. We recommend following each of these pages in the order they're presented here.
- Check and correct CPU utilization: CPU Performance Analysis and Monitoring
- Identify memory bottlenecks and remove: Memory Performance Analysis and Monitoring
- Characterize storage performance and correct: Storage Performance Analysis and Monitoring
- Understand and improve the network utilization profile: Network Performance Analysis and Monitoring
Within each of these pages are techniques for using counters from VirtualCenter and esxtop. Information on those counters is provided in vCenter Performance Counters and esxtop Performance Counters, respectively.
Also, note that, while useless in collecting performance data, Perfmon can help with analysis of large esxtop output files. See Using Perfmon for esxtop-based Performance Analysis for more information.