Performance Monitoring and Analysis


    A common question that crosses my desk is, "how do I analyze and correct  performance?" There are a variety of techniques for doing this and many  sources of information spread around the interweb, so I'm going to  collect a few thoughts here.


    Guest-based performance monitoring is an inaccurate and unhelpful means of evaluating performance in virtual deployments.  See Guest-based Performance Measurement for more inforamtion.  Monitoring and analysis of VMware ESX Server should be performed with esxtop and VirtualCenter.


    esxtop is the tried-and-true means of collecting every performance stat  needed and making it available in a way that is conducive to analysis.  The best source of information on launching esxtop can be found in the Resource Management Guide (page 159).  It's worth nothing that the directions in that guide came from ESX  Server 3.0.1. Since then, the developers have been kind enough to  simplify the process of including all performance counters. This is done  with the "-a" switch. So,

    esxtop -a -b > analysis.csv

    runs esxtop in batch mode and prints all performance counters.  Let me give you the quick "do-s" and "don't-s" of esxtop:

    1. With esxtop on ESX Server 3.5 and newer, always include the "-a" option to display all counters.
    2. With esxtop on versions of ESX Server prior to 3.5 always follow the  resource management guide to enable all counters.  The storage latency  statistics, for instance, are not displayed by default!
    3. Always start your VMs before running esxtop in batch mode.  If you  start them after starting "esxtop -b", then esxtop will only produce VM  data based on the VMs that were running at the time of its start.


    VirtualCenter doesn't provide all of the performance counters you might  need to analyze performance. But, it provides more than you might think!  The default setup for VirtualCenter performance counter collection is  fairly minimal. But this can be expanded by reconfiguring VC's  performance counter collection. This is done as follows:

    1. From the VI client, Administration->VirtualCenter Management Server Configuration...
    2. On the left, click "Statistics".
    3. Now increase the stats level as you see fit.

    More information on VC performance counters and archival can be found in the Understanding VirtualCenter Performance Statistics wiki article.  A few things are worth considering before going crazy with this:


    • You probably will never need level four stats. Those are mainly for debugging.
    • Use the DB size estimator after you monkey around with these levels.  The DBs can get quite large and you want to know this before it happens  and your VC system performance suffers.



    Performance analysis techniques were previously detailed in the  performance analysis whitepaper.  That has been migrated to this wiki  and new material will appear here.

    The following pages will provide guidance for identifying and correcting  performance problems on ESX-based systems.  We recommend following each  of these pages in the order they're presented here.

    1. Check and correct CPU utilization: CPU Performance Analysis and Monitoring
    2. Identify memory bottlenecks and remove: Memory Performance Analysis and Monitoring
    3. Characterize storage performance and correct: Storage Performance Analysis and Monitoring
    4. Understand and improve the network utilization profile: Network Performance Analysis and Monitoring

    Within each of these pages are techniques for using counters from  VirtualCenter and esxtop.  Information on those counters is provided in vCenter Performance Counters and esxtop Performance Counters, respectively.

    Also, note that, while useless in collecting performance data, Perfmon  can help with analysis of large esxtop output files.  See Using Perfmon for esxtop-based Performance Analysis for more information.