VMware Communities : Document List - Performance & VMmark

IOmega ix4-200d IOmeter results (100MB/Full Network)

Thu, 15 Oct 2009 21:07:59 GMT

Advanced Networking Performance Options

Wed, 14 Oct 2009 16:58:33 GMT

Some of the advanced networking options available in vSphere 4.0 are reviewed in this paper. Many of these options control trade-offs between latency, throughput, CPU utilization, and reliability (e.g., dropped packets). It is not possible to optimize all of these at the same time, so option defaults are chosen to be suitable for the vast majority of applications. These options are provided to meet the stricter requirements of other applications. Advanced options often have subtle side effects, or merely move an issue from one area to another. Therefore it is recommended that VMware Support be engaged before changing such options, especially for production machines.

There are over 100 options that can be set under Configuration → Advanced Settings → Net. Of these, the ones listed below are most likely to be useful for tuning networking performance. Many of the others are for internal testing or enable unreliable features.

All of the options listed here take integer values. For the “Boolean” ones only the default value is shown: 0 for “false”, and 1 for “true”. Other parameters are shown with their default, minimum, and maximum values.

Parameter Name	(Default, Minimum, Maximum)	Description
MaxPortRxQueueLen	(80, 1, 500)	Maximum length of the Rx queue for virtual ports whose clients support queueing. Possibly should be increased if Rx packet drops are seen in the port connected to a VM. Relevant only for e1000 vNICs used with Fault Tolerance (FT) and VLANs.
MaxNetifTxQueueLen	(500, 1, 1000)	Maximum length of the Tx queue for the physical NICs. Increase if Tx packet drops are seen in uplink port to the pNIC.
GuestTxCopyBreak	(64, 60, 4294967295)	Packet header transmits smaller than this in bytes will be copied rather than mapped. More security and functionality than performance implications.
VmxnetTxCopySize	(256, 0, 4294967295)	Transmits smaller than this in bytes will be copied rather than mapped. Copying costs CPU but puts lets pressure on the Tx queue and doesn’t require completion.
VmxnetWinUDPTxFullCopy	(1)	Enable full copy of Windows vmxnet UDP Tx packets. Might disable to save CPU, especially for jumbo frames, at the cost of risking more packet drops.
NetTxDontClusterSize	(0, 0, 8192)	Tx packet size (in bytes) smaller than this are transmitted immediately (coalescing options are over-ruled for these packets). Used to ensure good latency for small packets.
CoalesceTxTimeout	(4000, 1, 4294967295)	The coalesce timeout in micro-seconds, or effectively the maximum latency without transmitting. Smaller values can reduce the packet latency at the cost of CPU. Risky to go below 1000.
CoalesceDefaultOn	(1)	Enable dynamic coalescing. Disable to test if issues are related to coalescing.
CoalesceHandlerPcpu	(1, 0, 128)	pCPU that coalesce timeout handler runs on. May be important to set this if VM CPU pinning is used.
CoalesceTxQDepthCap	(40, 0, 80)	Maximum number of “normalized” Tx packets to coalesce. Reduce if Tx coalescing appears to be too aggressive.
CoalesceRxQDepthCap	(40, 0, 80)	Maximum number of “normalized” Rx packets to coalesce. Reduce if Rx coalescing appears to be too aggressive.
vmxnetThroughputWeight	(0, 0, 255)	How far to favor Tx throughput for vmxnet 2 & 3. “0” is dynamic, otherwise this is a weight where a lower value favors latency and a higher value favors throughput.
TcpipHeapSize	(24, 24, 120)	Initial size of the TCP/IP module heap in megabytes. May need to increase if there are many vmkernel connections (NFS, iSCSI, etc.).
TcpipDefLROMaxLength	(16000, 1, 65535)	Maximum length for the LRO aggregated packet for vmkernel connections. Increasing this reduces the number of acknowledgments, which improves efficiency but may increase latency.
E1000TxZeroCopy	(0)	If disabled copy UDP or non-TSO Tx packets for e1000.
E1000TxTsoZeroCopy	(1)	If enabled do not copy TSO Tx packets for e1000.
E1000IntrCoalesce	(1)	Enable interrupt coalescing for e1000. Disabling can improve latency at the expense of CPU.
MaxPktRxListQueue	(3500, 0, 200000)	Maximum number of packets queued in vmkernel. Increasing this can reduce the number of dropped packets but at the cost of increased vmkernel memory and queuing latency.
Vmxnet3RSSHashCache	(1)	Enable RSS hash cache for vmxnet3 in Windows guests.
VmklnxLROEnabled	(0)	Enable large packets for recent Linux guests with vmxnet 2 & 3. Most likely to benefit hosts with small number of VMs with few sessions each, where each session has a heavy Rx load (more than 1 MB/sec). This is an experimental feature and has not been tested extensively.
VmklnxLROMaxAggr	(6, 0, 24)	Maximum aggregation count in number of packets for vmklinux LRO.

rename_vm.sh

Wed, 07 Oct 2009 14:00:46 GMT

clone_vm2.sh

Wed, 07 Oct 2009 13:57:43 GMT

VAM Doc

Mon, 14 Sep 2009 12:52:51 GMT

This About Business Marketing.

Memory Performance Chart Metrics in the vSphere Client

Thu, 23 Jul 2009 22:05:34 GMT

The vSphere Client exposes several memory performance statistics for users to identify VM memory usage.

Some of the important memory performance metrics follow. Each metric name appears under the Measurement column of the Performance Chart Legend, as shown in the following screenshot:

Active: The amount of guest physical memory that is being used by the VM. Active memory may be different from what is seen inside the guest operating system. This is because the guest operating system generally has a more precise view about what memory is “active” than the hypervisor because it knows when applications allocate or deallocate memory. In addition, the sampling technique used by ESX often takes time to converge, so the memory usage measured in the guest operating system may be more accurate when the workload memory usage is fluctuating.
Shared: The amount of guest physical memory shared through transparent page sharing. This includes the memory shared with other VMs and the memory shared within the VM.
Consumed: The amount of host physical memory allocated to the VM, accounting for saving from memory sharing with other VMs. When multiple VMs share a host memory region, each VM is accounted to consume the shared memory proportionally based on the total references to that host memory. For example, if a VM has 100MB host memory equally shared with the other three VMs, the Consumed memory only accounts for 25MB. If the 100MB memory is only shared within the VM, the Consumed memory accounts for 100MB.

Note that for a host that is not memory overcommitted, the Consumed memory represents a “high water mark” of the memory usage by the VM. It is possible that in the past, the VM was actively using a large amount of host physical memory but currently it is not. Because host memory is not overcommitted, the Consumed memory will not be shrunk through ballooning or swapping. Hence, the Consumed memory could be much higher than the Active memory when host memory is not overcommitted.
Granted: The amount of guest physical memory currently backed by the host physical memory. Due to memory sharing, the Granted memory is greater than or equal to the Consumed memory. For instance, assuming a guest allocates 100MB memory while the whole memory are zeroes, once all the zeroed pages are shared, the VM’s Granted memory is 100MB but the VM’s Consumed memory is only 4k.
Overhead: The extra host physical memory used by the ESX to run a VM. The Overhead memory has two components: 1) System wide overhead from VMkernel; 2) Additional overhead for each VM, including the space reserved for the VM frame buffer and various virtualization data structures. Since the Overhead memory always resides in host memory, ESX must reserve memory for it. Thus a VM’s memory reservation has two individual components: user-specified memory reservation and overhead memory reservation. For example, if the user specifies a 1GB reservation and the Overhead memory for the VM is 100MB, the VM’s memory reservation when powered on would be 1.1GB.
Balloon: The amount of guest physical memory that is currently reclaimed through the balloon driver.
Swapped: The amount of guest physical memory swapped out to the VM’s swap device by ESX.
Swapped in rate: The rate at which the host physical memory is being swapped in from the host swap device.
Swapped out rate: The rate at which the host physical memory is being swapped out to the host swap device.

Performance Troubleshooting for VMware vSphere 4 and ESX 4.0

Mon, 13 Jul 2009 14:03:44 GMT

Performance problems can arise in any computing environment. Complex application behaviors, changing demands, and shared infrastructure can lead to problems arising in previously stable environments. Troubleshooting performance problems requires an understanding of the interactions between the software and hardware components of a computing environment. Moving to a virtualized computing environment adds new software layers and new types of interactions that must be considered when troubleshooting performance problems.

The attached document is the first installment in a guide covering performance troubleshooting in a vSphere environment. It uses a guided approach to lead the reader through the observable manifestations of complex hardware/software interactions in order to identify specific performance problems. For each problem covered, it includes a discussion of the possible root-causes and solutions. Topics covered include performance problems arising from issues in the CPU, memory, storage, and network subsystems, as well as in the VM and ESX host configuration. Guidance is given on relevant performance metrics to observe using the vSphere Client and esxtop in order to isolate specific performance issues.

This first installment of Performance Troubleshooting for VMware vSphere 4 covers performance troubleshooting on a single VMware ESX 4.0 host. It focuses on the most common performance problems which affect an ESX host. Future updates will add more detailed performance information, including troubleshooting information for more advanced problems and multi-host vSphere deployments.

This is a living document. Reader comments, questions, and suggestions are encouraged.

Storage Workload Characterization and Consolidation in Virtualized Enviornments

Wed, 03 Jun 2009 00:21:48 GMT

vscsiStats: Fast and Easy Disk Workload Characterization on VMware ESX Server

Mon, 01 Jun 2009 22:53:39 GMT

Using vscsiStats for Storage Performance Analysis

Fri, 29 May 2009 22:28:27 GMT

Introduction

esxtop is a great tool for performance analysis of all types. However, with only latency and throughput statistics, esxtop will not provide the full picture of the storage profile. Furthermore, esxtop only provides latency numbers for Fibre Channel and iSCSI storage. Latency analysis of NFS traffic is not possible with esxtop.

Since ESX 3.5, VMware has provided a tool specifically for profiling storage: vscsiStats. vscsiStats collects and reports counters on storage activity. Its data is collected at the virtual SCSI device level in the kernel. This means that results are reported per VMDK (or RDM) irrespective of the underlying storage protocol. The following data are reported in histogram form:

IO size
Seek distance
Outstanding IOs
Latency (in microseconds)
More!

Running vscsiStats

vscsiStats collection and analysis requires two steps:

Start statistics collection.
View accrued statistics.

Documentation on command-line parameters are available when running '/usr/lib/vmware/bin/vscsiStats -h'.

Starting and Stopping vscsiStats Collection

The tool is started with the following command:

/usr/lib/vmware/bin/vscsiStats -s -w <world_group_id>

This command starts the process that will accrue statistics. The world group ID must be set to a running virtual machine. The running VMs' IDs can be obtained by running '/usr/lib/vmware/bin/vscsiStats -l'.

After about 30 minutes vscsiStats will stop running. If the analysis is needed for a longer period, the start command should be repeated above in this window. That will defer the timeout and termination by another 30 minutes.

Since results are accrued and reported out in summary, the histograms will include data since collection was started. To reset all counters to zero, run '/usr/lib/vmware/bin/vscsiStats -r'.

Viewing Statistics

Counters are displayed by using the following command:

/usr/lib/vmware/bin/vscsiStats -p <histo_type> [-c]

The histogram type is used to specify either all of the statistics or one group of them. Options include all, ioLength, seekDistance, outstandingIOs, latency, interarrival.

Results can be produced in a more compact comma-delimited list by adding the optional "-c" above.

Using vscsiStats Results

Use Case 1: Identifying Sequential IO

Storage arrays can process sequential IO much faster than random IO. You can therefore improve the performance of a sequential workload by placing it on a dedicated LUN to allow the array to optimize access. vscsiStats can help you identify your sequential workloads even if you don't understand anything about the application in the VM.

Take the following graph as example, which I generated by running '/usr/lib/vmware/bin/vscsiStats -p seekDistance':

This graph shows that most of the commands are being issued a great distance from the previous command. It looks like all of the commands were 50,000 or more logical blocks away from the previous command. When I looked at the raw data, I saw that over 99% of the commands were more than 128 blocks away from the previous command. That's random access if I've ever seen it. Here's the opposite example:

In this case the logical block number (LBN) of each command is most frequently exactly one larger than the previous command. That's the signature of a heavily sequential workload. It shouldn't surprise you to learn that both of these profiles were generated by Iometer using random and sequential writes, respectively.

Use Case 2: Optimizing for IO Sizes

The IO size is an important characteristic of storage profiles. A variety of best practices have been provided by storage vendors to enable customers to tune their storage to a particular IO size. As an example, it may make sense to optimize an array's stripe size to its average IO size. vscsiStats can provide a histogram of IO sizes to help this process. The following graph was generated by '/usr/lib/vmware/bin/vscsiStats -p ioLength':

From these results I can see that about a quarter of the commands came in IOs smaller than 4k. About half of the commands were sized to 4k commands. The minute number of remaining IOs were larger than 4k. This signature is common of a VMDK formatted to 4k blocks and supporting OS and application execution. The storage array should be optimized for 4k blocks if this disk's performance is a priority.

Use Case 3: Storage Latency Analysis (Including NFS!)

esxtop is a terrific tool for latency-based storage analysis. Fibre Channel and iSCSI HBAs have device and kernel latencies in esxtop's storage panel. Software iSCSI initiators will show up as vmhba32 (ESX 3.5 and earlier) and vmhba33 (ESX 4.0 and later.) But esxtop does not provide latency statistics for NFS stores.

Because vscsiStats collects its results where the guest interacts with the hypervisor, it is unaware of the storage implementation. Latency statistics can be collected for all storage configurations with this tool.

The above graph shows that the server in my office with a single direct-attached SCSI disk is performing as I would expect. About half of all the operations are completing in under 5 ms. The other half take 5-15 ms to complete. A few commands took longer than 15 ms, but the number is so small that it doesn't concern me. Similar results can be seen with NFS arrays.

vscsiStats on ESXi

vscsiStats can be installed on ESXi hosts after putting the host into tech support mode. More information on this process is availalble on Scott's blog on the subject on vPivot.

Additional Resources

My colleagues Ajay Gulati, Chethan Kumar, and Irfan Ahmad presented at VPACT 09 Storage Workload Characterization and Consolidation in Virtualized Enviornments. This paper serves as an excellent example of vscsiStats in action.

I learned vscsiStats by reviewing Irfan's VMworld 2007 presentation (vscsiStats: Fast and Easy Disk Workload Characterization on VMware ESX Server) and playing with the tool. Check out his presentation if you'd like more detail.

Meet the Engineer Series: VMware Performance Advancements

Thu, 21 May 2009 23:41:52 GMT

Real people, real faces, discussing VMware vSphere topics...

In this video, VMware's Chief Performance Architect discusses why you should seriously consider virtualizing all of your applications on VMware:

{youtube}http://www.youtube.com/watch?v=I-W0ZWm5Jf4{youtube}

ESX Monitor Modes

Tue, 28 Apr 2009 20:40:05 GMT

VMware has supported Intel and AMD's virtualization assist since 2006. Long before then we were using an all-software approach that we call binary translation (BT). With the benefit of years of development and optimization, BT outperformed the early versions of hardware assist. But as hardware assist evolved the use of these new features became more attractive.

Because our support for hardware assist is rich and BT is heavily optimized, the monitor can benefit from using either technology in different situations. The following tables detail the defaults in ESX 4.0, which can be changed through VM settings if desired.

Monitor Defaults with Intel Processors

VM Configuration	Core-i7 (Nehalem)	45nm Core2 with VT-x	65nm Core2 with VT-x and FlexPriority	65nm Core2 with VT-x and No FlexPriority	P4 with VT-x	EM64T without VT-x	No EM64T
FT enabled	VT-x + SPT	VT-x + SPT	VT-x + SPT	VT-x + SPT	Not runnable	Not runnable	Not runnable
64-bit guests	VT-x + EPT	VT-x + SPT	VT-x + SPT	VT-x + SPT	VT-x + SPT	Not runnable	Not runnable
VMI enabled	BT + SPT	BT + SPT	BT + SPT	BT + SPT	BT + SPT	BT + SPT	BT + SPT
OpenServer, UnixWare, OS/2	VT-x + EPT	VT-x + SPT	VT-x + SPT	VT-x + SPT	VT-x + SPT	BT + SPT	BT + SPT
32-bit Linux and 32-bit FreeBSD	VT-x + EPT	VT-x + SPT	BT + SPT (*)	BT + SPT (*)	BT + SPT (*)	BT + SPT	BT + SPT
32-bit Windows XP, Windows Vista, Windows Server 2003, Windows Server 2008	VT-x + EPT	VT-x + SPT	VT-x + SPT	BT + SPT (*)	BT + SPT (*)	BT + SPT	BT + SPT
Windows 2000, Windows NT, DOS, Windows 95, Windows 98, Netware, 32-bit Solaris	BT + SPT (*)	BT + SPT (*)	BT + SPT (*)	BT + SPT (*)	BT + SPT (*)	BT + SPT	BT + SPT
All other 32-bit guests	VT-x + EPT	VT-x + SPT	VT-x + SPT	VT-x + SPT	VT-x + SPT	BT + SPT	BT + SPT

(*) When we use BT on an Intel system with VT-x capability, we dynamically switch to VT-x if the guest enters long mode.

Monitor Defaults with AMD Processors

Configuration	Barcelona, Phenom, and Newer	AMD64 pre-Barcelona	No AMD64
FT enabled	AMD-V + SPT	Not runnable	Not runnable
64-bit guests	AMD-V + RVI	BT + SPT	Not runnable
VMI enabled	BT + SPT	BT + SPT	BT + SPT
OpenServer, UnixWare, OS/2	AMD-V + RVI	BT + SPT	BT + SPT
32-bit Linux and 32-bit FreeBSD	AMD-V + RVI	BT + SPT	BT + SPT
32-bit Windows XP, Windows Vista, Windows Server 2003, Windows Server 2008	AMD-V + RVI	BT + SPT	BT + SPT
Windows 2000, Windows NT, DOS, Windows 95, Windows 98, Netware, 32-bit Solaris	BT + SPT	BT + SPT	BT + SPT
All other 32-bit guests	AMD-V + RVI	BT + SPT	BT + SPT

Legend

VT-x: Intel's virtualization hardware assist.
EPT: Extended Page Tables. Intel's on-board, virtualization-aware memory management unit (MMU).
EM64T: Intel's 64-bit extensions to the x86 architecture.
SPT: Shadow page tables. ESX's software memory management unit (i.e., not EPT or RVI.)
BT: Binary translation. ESX's software virtualization capability (i.e., not VT or AMD-V)
AMD-V: AMD's virtualization hardware assist.
RVI: Rapid Virtualization indexing. AMD's on-board, virtualization-aware memory management unit (MMU).

Storage Performance: VMFS and Protocols

Fri, 13 Mar 2009 00:12:37 GMT

Introduction

VMware's customers are always asking us about the storage stack. Without exception, the two most common questions about our storage system performance are:

Which storage protocol performs best?
Does VMFS scale to meet the demands of many servers and VMs?

This document will contain a few of the points needed to help understand this issue.

Storage Protocols

VMware published a paper comparing storage protocols in 2008. This paper detailed the two key characteristics of ESX's storage stack:

The hypervisor is easily able to drive the storage connection to link speed.
Configurations where protocol management happens in the HBA (Fibre Channel and HW iSCSI) are more CPU efficient.

On the first note, take the following graph, taken from page three of the paper:

Note that in this case all four test cases drive the storage to link speed. That's 2 Gb/s with the Fibre Channel HBA and 1 Gb/s with the other three. In short, if throughput is your goal, make decisions based on link speed. If you check through the rest of the paper, you'll see that response time is similar for all of the configurations, as well. But you will see slight differences in throughput in some of the protocols.

This brings us to the second point from above: less work is done by the CPU when protocol management can be off-loaded to the HBA. This means that FC and HW iSCSI HBAs will have additional CPU cycles for the VMs' work. It can also explain the slight differences in throughput in the other graphs in the paper. The efficiency results quoted in the paper are here:

The increased overheads of running software iSCSI or NFS are due to the VMkernel managing those protocols. It's worth noting that the proliferation of iSCSI in the enterprise has led VMware to spend considerable effort to improve the efficiency of SW iSCSI. Expect its efficiency to improve dramatically in the following releases.

VMFS Scalability

Many in the industry erroneously believe that VMFS won't scale as storage demands grow. Often SCSI reservations and disk locking are cited as the technical-sounding but vaguely-supported reason for this claim. It's worth sampling data from our scalable storage performance paper to debunk this myth.

This chart is a favorite in our world-wide tours as we address VMFS scalability. It's was first introduced in a VMFS scalability blog article that went live in February of 2008. It shows the results of using 64 hosts to generate a variety of traffic on a single VMFS volume. And it's a wealth of information on VMFS and storage access patterns. For instance:

The aggregate number of random writes, in cyan in the middle, maintains perfectly flat linear scalability as the host count grows from 1 to 64.
The aggregate number of random reads is initially limited by the few disks being accessed but ultimately matches the throughput of random writes as many disks come to bear to serve the large number of random reads.
The sequential read activity, which highlights the strengths of today's arrays, demonstrates the largest total throughput which only slightly drops as the array manages so many connections.
But the sequential read activity drops off dramatically as hosts are added.

This last example showing degradation in aggregate sequential read capability is an artifact of the workload that is very important to database administrators: multiple sequential reads approximate random activity. Why is this? As many hosts request more and more sequential data, the array interleaves these requests to maintain response times. This means that the sequential accesses get "shuffled" which results in a random access pattern.

In short, VMFS has no scalability problems as many hosts drive tremendous amounts of traffic to a single volume. If the data isn't convincing enough, consider the following: there are no SCSI reservations used during normal data access. This means that there are no scalability limitations as a result of virtual machine storage access. A word of caution, though: the file system is locked during administrative operations that change the metadata on the volume. This means that virtual machine creation or destruction can will result in file system locks. Perform these operations off of peak hours.

Best Practices for IBM Lotus Domino

Mon, 09 Mar 2009 22:19:19 GMT

Introduction

This page provides the best practices for virtualizing IBM Lotus Domino using VMware Infrastructure. This list is based on my VMworld 2008 session EA2348.

General Recommendations

Use newer hardware

Supports latest hardware assist technologies, larger on-processor cache
64-bit may not perform better with older hardware

Use VMware ESX , that uses bare-metal or hypervisor architecture

You can start with VMware ESXi - the free version.
Do not use the VMware Workstation or Server, that use the hosted architecture.

VMware ESX allows you the choice of virtualization technology best suited for your workload

Hardware Assist (AMD, Intel) (both CPU and MMU virtualization) if your hardware supports it
Paravirtualization (if you use SLES for your Domino deployment)
Binary Translation

Migrate to latest version of ESX

E.g. ESX 3.5 defaults to 2nd Generation Hardware Assist if available, has several I/O Performance improvements

Lotus Domino: Plan to migrate to version 8.0

Significant performance improvements, specially disk I/O

Provide Redundancy to the ESX host

Power supplies, HBAs, NICs, Network and SAN switches
E.g. NIC teaming, HBA multi-pathing

Leverage VMotion, Storage VMotion, DRS and HA for higher Domino availability

VM configuration

64-bit OS recommended

VI3 supports all x86 OSs that Domino supports: Windows, SLES, RHEL
Improved memory limits in 64-bit OS helps cache more data, and thus avoid disk IO. Reduces response times, and hence increasing the number of users
Increase VM memory when running in 64-bit guest OS

64-bit may not perform better with older hardware

E.g. 64-bit Windows more sensitive to onboard L2/L3 chip caches
Microsoft reports 10-15% degradation with older hardware

Guest Operating System:

Windows: Use 2003 SP2
- Microsoft eliminated most APIC TPR accesses, improves virtual performance
Linux: Use 2.6.18-53.1.4 kernel or later to use divider patch
- Some older Linux versions have a 1Khz timer rate
- Put divider=10 on the end of the kernel line in grub.conf and reboot

VM Time Synchronization

Use VMware Tools time synchronization within the virtual machine
Enable ESX server NTP daemon to sync with external stratum NTP source (VMware Knowledge Base ID# 1339)
Disable OS Time Service
- Windows: w32time service
- Linux: NTP daemon

Storage

Storage configuration is absolutely critical; most performance problems traced to this

Number of spindles, RAID configuration, drive speed, controller cache settings, queue depths – all make a big difference

Align partitions

For VMFS, use Virtual Center to create partitions
Check Recommendations for Aligning VMFS Partitions

Use separate, dedicated LUNs for OS/Domino, data and transaction logs

Separate the IO at physical disk level, not simply logical LUNs
Make sure these LUNs have enough spindles to support the IO demands
Fewer spindles or too many VMDK files on single VMFS LUN can substantially increase disk IO latencies
Check Scalable Storage Performance to understand the details

RAID configuration

RAID 1+0 for Data, RAID 0 for Log

Cache settings

Write policy to "write back“, read policy to "read ahead“

Queue Depths

Increase to 255

Storage Protocol: Fibre Channel or iSCSI

Link speed typically limits the scalability, NOT VMware ESX
Link speed is maintained up to 32 virtual machines for each storage connection option
For details check Comparison of Storage Protocol Performance

Storage Partition: VMFS or RDM

VMFS is recommended
- Leverage templates and quick provisioning
- Fewer LUNs means you don’t have to watch Heap
Performance difference between VMFS and RDM not significant
- For details check Performance Characterization of VMFS and RDM Using a SAN

VI3 supports latest storage technologies: leverage these if you have already invested or plan to invest

Fibre channel – 8Gbps connectivity
ISCSI – 10GigE network connectivity, Jumbo Frames
Infiniband support

Virtual CPUs

The number of vCPUs per VM depends on the number of users to be supported

Start with uni-processor, may be enough
Try not to over-provision vCPUs in the guest CPU

Verify CPU compatibility for VMotion

Memory

Increasing memory to avoid disk I/O is most technique to improve performance
More available memory = more Lotus Domino Cache

Increase NSF_DbCache_Maxentries value

Leverage the higher VI 3.5 support 64GB memory limit per VM in VI 3.5 when using 64-bit guest OS for Domino

64-bit OSs can take advantage of larger memory limits for file caching

Leverage NUMA optimizations in VI3

When using NUMA, try to fit the VM within a single node to avoid latencies accessing memory on remote nodes

Networking

Use dedicated NICs based on the network traffic

E.g. separate NICs for mail and replication traffic

Use NIC Teaming & VLAN Trunking

Use Enhanced VMXNET driver with TSO and Jumbo Frames support

Enable TCP transmit coalescing

Check the SPECweb paper for details.

Co-located VMs outperform physical 1Gbps network speed

Resource Management

Use proportional and absolute mechanisms to control VM priorities

Shares, reservations, and limits for CPU and memory
Shares for virtual disks
Traffic shaping for network

Faster migration resulting in better load balancing when using

Smaller VMs
Lesser memory reservations for VMs

Affinity rules for VM placement

E.g. Directory, Mail Server VMs on same ESX

Deployment

Virtualization Assessment

Capacity Planner
Benchmark against Information Warehouse

Easy migration

VMware Converter – both hot and cold cloning
Start with RDM to point to existing data/ transaction log LUNs, but move to VMFS later

Easier change management and quicker provisioning

Templates and clones for easy provisioning

Interpreting esxtop Statistics

Wed, 31 Dec 2008 22:48:19 GMT

Section 1. Introduction
Section 2. CPU
Section 2.1 Worlds and Groups
Section 2.2 Global Statistics
Section 2.3 World Statistics
Section 3. Memory
Section 3.1 Machine Memory and Guest Physical Memory
Section 3.2 Global Statistics
Section 3.3 Group Statistics
Section 4 Disk
Section 4.1 Adapter, Device, VM screens
Section 4.2 Disk Statistics
Section 4.2.1 I/O Throughput Statistics
Section 4.2.2 Latency Statistics
Section 4.2.3 Queue Statistics
Section 4.2.4 Error Statistics
Section 4.2.5 PAE Statistics
Section 4.2.6 Split Statistics
Section 4.3 Batch Mode Output
Section 5 Network
Section 5.1 Port
Section 5.2 Port Statistics
Section 6. Interrupt
Section 7. Batch Mode

Section 1. Introduction

Esxtop allows monitoring and collection of data for all system resources: CPU, memory, disk and network. When used interactively, this data can be viewed on different types of screens; one each for CPU statistics, memory statistics, network statistics and disk adapter statistics. In addition to the disk adapter statistics in earlier versions, starting with ESX3.5, disk statistics at the device and VM level are also available. Starting with ESX 4.0, esxtop has an interrupt statistics screen. In the batch mode, data can be redirected to a file for offline uses.

Many esxtop statistics are computed as rates, e.g. CPU statistics %USED. A rate is computed based on the refresh interval, the time between successive snapshots. For example, %USED = ( CPU used time at snapshot 2 - CPU used time at snapshot 1 ) / time elapsed between snapshots. The default refresh interval can be changed by the command line option "-d", or the interactive command 's'. The return key can be pressed to force a refresh.

In each screen, data is presented at different levels of aggregation. It is possible to drill down to expanded views of this data. Each screen provides different expansion options.

It is possible to select all or some fields for which data collection is done. In the case of interactive use of esxtop, the order in which the selected fields are displayed can be selected.

In the following sections, this document will describe the esxtop statistics shown by each screen and their usage.

Section 2. CPU

Section 2.1 Worlds and Groups

Esxtop uses worlds and groups as the entities to show CPU usage. A world is an ESX Server VMkernel schedulable entity, similar to a process or thread in other operating systems. A group contains multiple worlds.

Let's use a VM as an example. A powered-on VM has a corresponding group, which contains multiple worlds. In ESX 4.0, there is one vcpu (hypervisor) world corresponding to each VCPU of the VM. The guest activities are represented mostly by the vcpu worlds. (In ESX 3.5, esxtop shows a vmm world and a vcpu world for each VCPU. The guest activities are represented mostly by the vmm worlds.) Besides the vcpu worlds, there are other assisting worlds, such as a MKS world and a VMX world. The MKS world assists mouse/keyboard/screen virtualization. The VMX world assists the vcpu worlds (the hypervisor). The usage of the VMX world is out of the scope of this document. In ESX 4.0, there is only one vmx world. (In ESX 3.5, there are two vmx worlds for each VM.)

There are other groups besides VM groups. Let's go through a few examples:

The "idle" group is the container for the idle worlds, each of which corresponds to one PCPU.
The "system" group contains the VMKernel system worlds.
The "helper" group contains the helper worlds that assist VMKernel operations.
In classic ESX, the "console" group is for the console OS, which runs ESX management processes. In ESXi, these ESX management processes are running as user worlds directly on VMKernel. So, on an ESXi box you can see much more groups than on a classic ESX, but not the "console" group.

Note that groups can be organized in a hierarchical manner in ESX. However, esxtop shows, in a flat form, the groups that contain some worlds. More detailed discussion on the groups are out of the scope.

Q: Why can't we find any vmm worlds for a VM in ESX 4.0?
A: Before ESX 4.0, each VCPU has two worlds "vmm" and "vcpu". In ESX 4.0, cpu scheduler merges their statistics to one vcpu world. So, CPU stats won't show vmm worlds. This is not a problem.

Section 2.2 Global Statistics

"up time"

The elapsed time since the server has been powered on.

"number of worlds"

The total number of worlds on ESX Server.

"CPU load average"

The arithmetic mean of CPU loads in 1 minute, 5 minutes, and 15 minutes, based on 6-second samples. CPU load accounts the run time and ready time for all the groups on the host.

"PCPU(%)"

The percentage CPU utilization per physical CPU.

Q: What does it mean if PCPU% is high?
A: It means that you are using lots of resource. (a) If all of the PCPUs are near 100%, it is possible that you are overcommiting your cpu resource. You need to check RDY% of the groups in the system to verify cpu overcommitment. Refer to RDY% below. (b) If some PCPUs stay near 100%, but others are not, there might be an imbalance issue. Note that you'd better monitor the system for a few minutes to verify whether the same PCPUs are using ~100% CPU. If so, check VM CPU affinity settings.

"used total"

Sum( PCPU(%) ) / number of PCPUs

"LCPU(%)"

The percentage CPU utilization per logical CPU. The CPU used percentages for the logical CPUs belonging to a package add up to 100%. This line is displayed only if hyper-threading is present and enabled.

"CCPU(%)"

Percentages of total CPU time as reported by the ESX Service Console. "us" is for percentage user time, "sy" is for percentage system time, "id" is for percentage idle time and "wa" is for percentage wait time. "cs/sec" is for the context switches per second recorded by the ESX Service Console.

Q: What's the difference of CCPU% and the console group stats?
A: CCPU% is measured by the COS. "console" group CPU stats is measured by VMKernel. The stats are related, but not the same.

Section 2.3 World Statistics

A group statistics is the sum of world statistics for all the worlds contained in that group. So, this section focuses on worlds. You may apply the description to the group as well, unless stated otherwise.

ESX can make use of the Hyperthreading technology, so, the performance counters takes Hyperthreading into consideration as well. But, to simplify this document, we will ignore HT related issues. Please refer to "Resource Management Guide" for more details.

"%USED"

The percentage physical CPU time accounted to the world. If a system service runs on behalf of this world, the time spent by that service (i.e. %SYS) should be charged to this world. If not, the time spent (i.e. %OVRLP) should not be charged against this world. See notes on %SYS and %OVRLP.

%USED = %RUN + %SYS - %OVRLP

Q: Is it possible that %USED of a world is greater than 100%?
A: Yes, if the system service runs on a different PCPU for this world. It may happen when your VM has heavy I/O.

Q: For an SMP VM, why does VCPU 0 have higher CPU usage than others?
A: The system services are accounted to VCPU 0. You may see higher %USED on VCPU 0 than others, although the run time (%RUN) are balanced for all the VCPUs. This is not a problem for CPU scheduling, but only the way VMKernel does the CPU accounting.

Q: What is the maximum %USED for a VM group?
A: The group stats is the sum of the worlds. So, the maximum %USED = NWLD * 100%. NWLD is the number of worlds in the group.

Typically, worlds other than VCPU worlds are waiting for events most of time, not costing too much CPU cycles. Among all the worlds, VCPU worlds represent best the guest. Therefore, %USED for a VM group usually do not exceed Number of VCPUs * 100%.

Q: What does it mean if %USED of a VM is high?
A: The VM is using lots of CPU resource. You may expand to worlds to see what worlds are using most of them.

"%SYS"

The percentage of time spent by system services on behalf of the world. The possible system services are interrupt handlers, bottom halves, and system worlds.

Q: What does it mean if %SYS is high?
A: It usually means that your VM has heavy I/O.

Q: Are %USED and %SYS similar to user time and system time in Linux?
A: No. They are totally different. For Linux OS, user (system) time for a process is the time spent in user (kernel) mode. For ESX, %USED is for the accounted time and %SYS is for the system service time.

"%OVRLP"

The percentage of time spent by system services on behalf of other worlds. In more detail, let's use an example.

When World 'W1' is running, a system service 'S' interrupts 'W1' and services World 'W2'. The time spent by 'S', annotated as 't', is included in the run time of 'W1'. We use %OVRLP of 'W1' to show this time. This time 't' is accounted to %SYS of 'W2', as well.

Again, let's take a look at "%USED = %RUN + %SYS - %OVRLP". For 'W1', 't' is included in %RUN and %OVRLP, not in %SYS. By subtracting %OVRLP from %RUN, we do not account 't' in %USED of 'W1'. For 'W2', 't' is included in %SYS, not in %RUN or %OVRLP. By adding %SYS, we accounted 't' to %USED of 'W2'.

Q: What does it mean if %OVRLP of a VM is high?
A: It usually means the host has heavy I/O. So, the system services are busy handling I/O. Note that %OVRLP of a VM group may or may not be spent on behalf of this VM. It is the sum of %OVRLP for all the worlds in this group.

"%RUN"

The percentage of total scheduled time for the world to run.

Q: What is the difference between %USED and %RUN?
A: %USED = %RUN + %SYS - %OVRLP. (%USED takes care of the system service time.) Details above.

Q: What does it mean if %RUN of a VM is high?
A: The VM is using lots of CPU resource. It does not necessarily mean the VM is under resource constraint. Check the description of %RDY below, for determining CPU contention.

"%RDY"

The percentage of time the world was ready to run.

A world in a run queue is waiting for CPU scheduler to let it run on a PCPU. %RDY accounts the percentage of this time. So, it is always smaller than 100%.

Q: How do I know CPU resource is under contention?
A: %RDY is a main indicator. But, it is not sufficient by itself.

If a "CPU Limit" is set to a VM's resource settings, the VM will be deliberately held from scheduled to a PCPU when it uses up its allocated CPU resource. This may happen even when there is plenty of free CPU cycles. This time deliberately held by scheduler is shown by "%MLMTD", which will be describe next. Note that %RDY includes %MLMTD. For, for CPU contention, we will use "%RDY - %MLMTD". So, if "%RDY - %MLMTD" is high, e.g., larger than 20%, you may experience CPU contention.

What is the recommended threshold? Well, it depends. As a try, we could start with 20%. If your application speed in the VM is OK, you may tolerate higher threshold. Otherwise, lower.

Q: How do we break down 100% for the world state times?
A: A world can be in different states, either scheduled to run, ready to run but not scheduled, or not ready to run (waiting for some events).
100% = %RUN + %READY + %CSTP + %WAIT.
Check the description of %CSTP and %WAIT below.

Q: What does it mean if %RDY of a VM is high?
A: It means the VM is possibly under resource contention. Check "%MLMTD" as well. If "%MLMTD" is high, you may raise the "CPU limit" setting for the VM. If "%RDY - %MLMTD" is high, the VM is under CPU contention.

"%MLMTD"

The percentage of time the world was ready to run but deliberately wasn't scheduled because that would violate the "CPU limit" settings.

Note that %MLMTD is included in %RDY.

Q: What does it mean if %MLMTD of a VM is high?
A: The VM cannot run because of the "CPU limit" setting. If you want to improve the performance of this VM, you may increase its limit. However, keep in mind that it may reduce the performance of others.

"%CSTP"

The percentage of time the world spent in ready, co-deschedule state. This co-deschedule state is only meaningful for SMP VMs. Roughly speaking, ESX CPU scheduler deliberately puts a VCPU in this state, if this VCPU advances much farther than other VCPUs.

Q: What does it mean if %CSTP is high?
A: It usually means the VM workload does not use VCPUs in a balanced fashion. The VCPU with high %CSTP is used much more often than the others. Do you really need all those VCPUs? Do you pin the guest application to the VCPUs?

"%WAIT"

The percentage of time the world spent in wait state.

This %WAIT is the total wait time. I.e., the world is waiting for some VMKernel resource. This wait time includes I/O wait time, idle time and among other resources. Idle time is presented as %IDLE.

Q: How do I know the VCPU world is waiting for I/O events?
A: %WAIT - %IDLE can give you an estimate on how much CPU time is spent in waiting I/O events. This is an estimate only, because the world may be waiting for resources other than I/O. Note that we should only do this for VMM worlds, not the other kind of worlds. Because VMM worlds represent the guest behavior the best. For disk I/O, another alternative is to read the disk latency stats which we will explain in the disk section.

Q: How do I know the VM group is waiting for I/O events?
A: For a VM, there are other worlds besides the VCPUs, such as a mks world and a VMX world. Most of time, the other worlds are waiting for events. So, you will see ~100% %WAIT for those worlds. If you want to know whether the guest is waiting for I/O events, you'd better expand the group and analyze the VCPU worlds as stated above.

Since %IDLE makes no sense to the worlds other than VCPUs, we may use the group stats to estimate the guest I/O wait by "%WAIT - %IDLE - 100% * (NWLD - NVCPU)". Here, NWLD is the number of worlds in the group; NVCPU is the number of VCPUs. This is a very rough estimate, due to two reasons. (1) The world may be waiting for resources other than I/O. (2) We assume the other assisting worlds are not active, which may not be true.

Again, for disk I/O, another alternative is to read the disk latency stats which we will explain in the disk section.

Q: Why do I always see a high %WAIT for VMX/mks worlds?
A: This is normal. That means there are not too much activities on them.

Q: Why do I see a high %WAIT for a VM group?
A: For a VM, there are other worlds besides the VCPUs, such as a mks world and VMX worlds. These worlds are waiting for events most of time.

"%IDLE"

The percentage of time the VCPU world is in idle loop. Note that %IDLE is included in %WAIT. Also note that %IDLE only makes sense to VCPU world. The other worlds do not have idle loops, so, %IDLE is zero for them.

"%SWPWT"

The percentage of time the world is waiting for the ESX VMKernel swapping memory. The %SWPWT (swap wait) time is included in the %WAIT time. This is a new statistics added in ESX 4.0.

Q: Why do I see a high %SWPWT for a VM group?
A: The VM is swapping memory.

Section 3. Memory

Section 3.1 Machine Memory and Guest Physical Memory

It is important to note that some statistics refer to guest physical memory while others refer to machine memory. "Guest physical memory" is the virtual-hardware physical memory presented to the VM. "Machine memory" is actual physical RAM in the ESX host. Let's use the following figure to explain. In the figure, two VMs are running on an ESX host, where each block represents 4 KB of memory and each color represents a different set of data on a block.

Inside each VM, the guest OS maps the virutal memory to its physical memory. ESX Kernel maps the guest physical memory to machine memory. Due to ESX Page Sharing technology, guest physical pages with the same content can be mapped to the same machine page.

Section 3.2 Global Statistics

"MEM overcommit avg"

Average memory overcommit level in 1-min, 5-min, 15-min (EWMA).

Memory overcommit is the ratio of total requested memory and the "managed memory" minus 1. VMKernel computes the total requested memory as a sum of the following components: (a) VM configured memory (or memory limit setting if set), (b) the user world memory, (c) the reserved overhead memory. (Overhead memory will be discussed in more detail for "OVHD" and "OVHDMAX" in Section 3.3.)

"managed memory" will be defined in "VMKMEM" section.

Q: What does it mean if overcommit is not 0?
A: It means that total requested guest physical memory is more than the machine memory available. This is fine, because ballooning and page sharing allows memory overcommit.

This metric does not necessarily mean that you will have performance issues. Use "SWAP" and "MEMCTL" to find whether you are experiencing memory problems.

Q: What's the meaning of overcommit?
A: See above description for details. Roughly speaking, it reflects the ratio of requested memory and the available memory.

"PMEM" (MB)

The machine memory statistics for the host.

"total": the total amount of machine memory in the server. It is the machine memory reported by BIOS.

"cos" : the amount of machine memory allocated to the ESX Service Console.

"vmk" : the amount of machine memory being used by the ESX VMKernel. "vmk" includes kernel code section, kernel data and heap, and other VMKernel management memory.

"other": the amount of machine memory being used by everything other than the ESX Service Console and ESX VMKernel. "other" contains not only the memory used by VM but also the user worlds that run directly on VMKernel.

"free" : the amount of machine memory that is free.

Q: Why is total not the same as RAM size plugged in my memory slots?
A: This is because some memory range is not available for use. It is fine, if the difference is small. If the difference is big, there might be some hardware issue. Check your BIOS.

Q: Why can't I find the cos part?
A: COS is only available in classic ESX. You are using ESXi.

Q: How do I break down the total memory?
A: total = cos + vmk + other + free

Q: Which one contains the memory used by VMs?
A: "other" contains the machine memory that backs guest physical memory of VMs. Note that "other" also includes the overhead memory.

Q: How do I know my "free" memory is low? Is it a problem if it is low?
A: You could use the "state" field, which will be explained next, to see whether the free memory is low. Basically, it is fine if you do not experience memory swapping or ballooning. Check "SWAP" and "MEMCTL" to find whether you are experiencing memory problems.

"VMKMEM" (MB)

The machine memory statistics for VMKernel.

"managed": the total amount of machine memory managed by VMKernel. VMKernel "managed" memory can be dynamically allocated for VM, VMKernel, and User Worlds.

"minfree": the minimum amount of machine memory that VMKernel would like to keep free. This is because VMKernel needs to keep some amount of free memory for critical uses.

"rsvd" : the amount of machine memory that is currently reserved. "rsvd" is the sum of three parts: (a) the reservation setting of the groups; (b) the overhead reservation of the groups; (c) "minfree".

"ursvd" : the amount of machine memory that is currently unreserved. It is the memory available for reservation.

Please note that the VM admission control is done at resource pool level. So, this statistics is not used directly by admission control. "ursvd" can be used
as a system level indicator.

"state" : the free memory state. Possible values are high, soft, hard and low. The memory "state" is "high", if the free memory is greater than or equal to 6% of "total" - "cos". If is "soft" at 4%, "hard" at 2%, and "low" at 1%. So, high implies that the machine memory is not under any pressure and low implies that the machine memory is under pressure.

While the host's memory state is not used to determine whether memory should be reclaimed from VMs (that decision is made at the resource pool level), it can affect what mechanisms are used to reclaim memory if necessary. In the high and soft states, ballooning is favored over swapping. In the hard and low states, swapping is favored over ballooning.

Please note that "minfree" is part of "free" memory; while "rsvd" and "ursvd" memory may or may not be part of "free" memory. "reservation" is different from memory allocation.

Q: Why is "managed" memory less than the sum of "vmk", "other" and "free" in the PMEM line? Is it normal?
+A: It is normal, just the way we do accounting. A more precise definition for "managed" is the free memory after VMKernel initialization. So, this amount of memory can be dynamically allocated for use of VMs, VMKernel, and user worlds. "managed" = "some part of vmk" + "other" + "free".+

+So, "managed" < "vmk" + "other" + "free". Or, in an equivalent form, "managed" < "total" - "cos".+

Q: How do I break down the managed memory in terms of reservation?
A: "managed" = "rsvd" + "ursvd" + "vmkernel usage"

VMKernel machine memory manager needs to use some part of memory, which should not be subject to reservation, so, it is not in "rsvd", nor in "ursvd". In the above equation, we put this part under "vmkernel usage". Unfortunately, it is not shown directly in esxtop.

Note that the vmkernel usage in managed memory is part of "vmk".

Q: What does it mean if "ursvd" is low?
A: VMKernel admission control prohibits a VM PowerOn operation, if it cannot meet the memory reservation of that VM. The memory reservation includes the reservation setting, a.k.a. "min", and the monitor overhead memory reservation. Note that even if "min" is not set, VMKernel still needs to reserve some amount
of memory for monitor uses.

So, it is possible that even though you have enough free memory, a new VM cannot power on due to the violation of memory reservation.

Q: Why do I fail admission control even though "ursvd" is high?
A: The VM admission control is done at resource pool level. Please check the "min" setting of all its parent resource pools.

Q: Why is "managed" greater than the sum of "rsvd" and "ursvd"? Is it normal?
A: It is normal. See above question. VMKernel may use some of the managed memory. It is not accounted in "rsvd" and "ursvd".

Q: What is the meaning of "state"?
A: See the description of "state" above.

Q: How do I know my ESX box is under memory pressure?
A: It is usually safe to say the ESX box is under memory pressure, if "state" is "hard" or "low". But, you need also check "SWAP" and "MEMCTL" to find whether you are experiencing memory problems. Basically, if there is not enough free memory and ESX are experiencing swapping or ballooning, ESX box is under memory pressure.

Note that ballooning does not have as big performance hit as swapping does. Ballooning may cause guest swapping. ESX swapping means host swapping.

Also note that A VM may be swapping or ballooning, even though there is enough free memory. This is due to the reservation setting.

"COSMEM" (MB)

The memory statistics reported by the ESX Service Console.

"free" : the amount of idle machine memory.

"swap_t": the total swap configured.

"swap_f": the amount of swap free.

"r/s" : the rate at which memory is swapped in from disk.

"w/s" : the rate at which memory is swapped out to disk.

Note that these stats essentially come from the COS proc nodes.

Q: What does it mean if I see a high r/s or w/s?
A: Your console OS is swapping. It is highly likely that your COS free memory is low. You may either configure more memory for COS and restart your ESX box, or stop some programs running inside your COS.

Q: Why can't I see this COSMEM line?
A: You are using ESXi not classic ESX.

"NUMA" (MB)

The ESX NUMA statistics. For each NUMA node there are two statistics: (1) the "total" amount of machine memory managed by ESX; (2) the amount of machine memory currently "free".

Note that ESX NUMA scheduler optimizes the uses of NUMA feature to improve guest performance. Please refer to "Resource Management Guide" for details.

Q: Why can't I see this NUMA line?
A: You are not using a NUMA machine, or your BIOS disables it.

Q: Why is the sum of NUMA memory not equal to "total" in the PMEM line?
A: The PMEM "total" is the memory reported by BIOS, while the NUMA "total" is the memory managed by VMKernel machine memory manager. There are two major parts of memory seen by BIOS but not given to machine memory manager: (1) COS uses, and (2) VMKernel uses during early initialization.

So, Sum("NUMA total") < "PMEM total" - "cos".

Note that the free memory on all the nodes can be added up as the "free" memory in the PMEM line.

"PSHARE" (MB)

The ESX page-sharing statistics.

"shared": the amount of guest physical memory that is being shared.

"common": the amount of machine memory that is common across World(s).

"saving": the amount of machine memory that is saved due to page-sharing.

The monitor maps guest physical memory to machine memory. VMKernel selects to map guest physical pages with the same content to the same machine page. In other words, those guest physical pages are sharing the same machine page. This kind of sharing can happen within the same VM or among the VMs.

Since each VM's "shared" memory measures guest physical memory, the host's "shared" memory may be larger than the total amount of machine memory if memory is overcommitted. "saving" illustrates the effectiveness of page sharing for saving machine memory.

"shared" = "common" + "saving".

Note that esxtop only shows the pshare stats for VMs, excluding the pshare stats for user worlds.

"SWAP" (MB)

The ESX swap usage statistics.

"curr" : the current swap usage. This is the total swapped machine memory of all the groups. So, it includes VMs and user worlds.

"target": the swap usage expected to be. This is the total swap target of all the groups. So, it includes VMs and user worlds.

"r/s" : the rate at which machine memory is swapped in from disk.

"w/s" : the rate at which machine memory is swapped out to disk.

Note that swap here is host swap, not guest swap inside the VM.

Q: What does it mean if "curr" is not the same as "target"?
A: It means ESX will swap memory to meet the swap target. Note that the actual swapping is done at the group level. So, you should check "SWCUR" and "SWTGT" for each group. We will discuss this in the next section.

Q: Is it bad if "r/s" is high?
A: Yes, it is very bad. This usually means that you have memory resource contention. Because swapin is synchronous, it will hurt guest performance a lot.

Do two things: (1) Check your "free" memory or "state" as mentioned above. If free memory is low, you need to move VMs to other hosts or add more memory to the host. (2) If free memory is not low, check your resource setting of your VMs or user worlds. You may have set a low "limit", which causes swapping.

Q: Is it bad if "w/s" is high?
A: Yes, it is also very bad. This usually means that you have memory resource contention. Do the similar actions as mentioned above.

"MEMCTL" (MB)

The memory balloon statistics.

"curr" : the total amount of physical memory reclaimed by balloon driver. This is the total ballooned memory by the VMs.

"target": total amount of ballooned memory expected to be. This is the total ballooned targets of the VMs.

"max" : the maximum amount of physical memory reclaimable.

Note that ballooning may or may not lead to guest swapping, which is decided by the guest OS.

Q: What does it mean if "curr" is not the same as "target"?
A: It means ESX will balloon memory to meet the balloon target. Note that the actual ballooning is done for the VM group. So, you should check "MCTLSZ" and "MCTLTGT" for each group. We will discuss this in the next section.

Q: How do I know the host is ballooning memory?
A: If the "curr" is changing, you can know it is ballooning. Since ballooning is done at VM level, a better way is to monitor "MCTLSZ" for each group. We will discuss this in the next section.

Q: Is it bad if we have lots of ballooning activities?
A: Usually it is fine. Ballooning tends to take unused memory from one VM and make them available for others. The possible side effects are (a) reducing the memory cache used by guest OS, (b) guest swapping. In either cases, it may hurt guest performance. Please note that (a) and (b) may or may not happen, depending on your workload inside VM.

On the other hand, under memory contention, ballooning is much better than swapping in terms of performance.

Section 3.3 Group Statistics

Esxtop shows the groups that use memory managed by VMKernel memory scheduler. These groups can be used for VMs or purely for user worlds running directly on VMKernel. You may see many pure user world groups on ESXi, not on classic ESX.

Tip: use 'V' command to show only the VM groups.

"MEMSZ" (MB)

For a VM, it is the amount of configured guest physical memory.

For a user world, it includes not only the virtual memory that is backed by the machine memory, but also the reserved backing store size.

Q: How do I break down "MEMSZ" of a VM?
A: A VM's guest physical memory could be mapped to machine memory, reclaimed by balloon driver, or swapped to disk, or never touched. The guest physical memory can be "never touched", because (1) the VM has never used it since power on; or, (2) it was reclaimed by balloon driver before, but has not been used since the balloon driver releases it last time. This part of memory is not measured directly by VMKernel.

"MEMSZ" = "GRANT" + "MCTLSZ" + "SWCUR" + "never touched"

Please refer to "GRANT", "MCTLSZ", "SWCUR".

"GRANT" (MB)

For a VM, it is the amount of guest physical memory granted to the group, i.e., mapped to machine memory. The overhead memory, "OVHD" is not included in GRANT. The shared memory, "SHRD", is part of "GRANT". This statistics is added to esxtop in ESX 4.0.

The consumed machine memory for the VM, not including the overhead memory, can be estimated as "GRANT" - "SHRDSVD". Please refer to "SHRDSVD".

For a user world, it is the amount of virtual memory that is backed by machine memory.

Q: Why is "GRANT" less than "MEMSZ"?
A: Some guest physical memory has never been used, or is reclaimed by balloon driver, or is swapped out to the VM swap file. Note that this kind of swap is host swap, not the guest swap by the guest OS.

"MEMSZ" = "GRANT" + "MCTLSZ" + "SWCUR" + "never touched"

Q: How do I know how much machine memory is consumed by this VM?
A: GRANT accounts the guest physical memory, it may not be the same as the mapped machine memory, due to page sharing.

The consumed machine memory can be estimated as "GRANT" - "SHRDSVD". Please note that this is an estimate. Please refer to "SHRDSVD".

Note that overhead memory, "OVHD", is not part of the above consumed machine memory.

"SZTGT" (MB)

The amount of machine memory to be allocated. (TGT is short for "target".) Note that "SZTGT" includes the overhead memory for a VM.

This is an internal counter, which is computed by ESX memory scheduler. Usually, there is no need to worry about this. Roughly speaking, "SZTGT" of all the VMs is computed based on the resource usage, available memory, and the "limit/reservation/shares" settings. This computed "SZTGT" is compared against the current memory consumption plus overhead memory for a VM to determine the swap and balloon target, so that VMKernel may balloon or swap appropriate amount
of memory to meet its memory demand. Please refer to "Resource Management Guide" for details.

Q: How come my "SZTGT" is larger than "MEMSZ"?
A: "SZTGT" includes the overhead memory, while "MEMSZ" does not. So, it is possible for "SZTGT" be larger than "MEMSZ".

Q: How do I use "SZTGT"?
A: This is an internal counter. You don't need to use it.

This counter is used to determine future swapping and ballooning activities. Check "SWTGT" and "MCTLTGT".

"TCHD" (MB)

The amount of guest physical memory recently used by the VM, which is estimated by VMKernel statical sampling.

VMKernel estimates active memory usage for a VM by sampling a random subset of the VM's memory resident in machine memory to detect the number of memory reads and writes. VMKernel then scales this number by the size of VM's configured memory and averages it with previous samples. Over time, this average will approximate the amount of active memory for the VM.

Note that ballooned memory is considered inactive, so, it is excluded from "TCHD".

Because sampling and averaging takes time, "TCHD" won't be exact, but becomes more accurate over time.

VMKernel memory scheduler charges the VM by the sum of (1) the "TCHD" memory and (2) idle memory tax. This charged memory is one of the factors that memory scheduler uses for computing the "SZTGT".

Q: What is the difference between "TCHD" and working set estimate by guest OS?
A: "TCHD" is the working set estimated by VMKernel. This number may be different from guest working set estimate. Sometimes the difference may be big, because (1) guest OS uses a different working set estimate algorithm, (2) guest OS has a different view of active guest physical memory, due to ballooning and host swapping,

Q: How is "TCHD" used?
A: "TCHD" is a working set estimate, which indicates how actively the VM is using its memory. See above for the internal use of this counter.

"%ACTV"

Percentage of active guest physical memory, current value.

"TCHD" is actually computed based on a few parameters, coming from statistical sampling. The exact equation is out of scope of this document. Esxtop shows some of those parameters, %ACTV, %ACTVS, %ACTVF, %ACTVN. Here, this document provides simple descriptions without further discussion.

%ACTV reflects the current sample.
%ACTVS is an EWMA of %ACTV for long term estimate.
%ACTVF is an EWMA of %ACTV for short term estimate.
%ACTVN is a predict of what %ACTVF will be at next sample.

Since they are very internal to VMKernel memory scheduler, we do not discuss their usage here.

"%ACTVS"

Percentage of active guest physical memory, slow moving average. See above.

"%ACTVF"

Percentage of active guest physical memory, fast moving average. See above.

"%ACTVN"

Percentage of active guest physical memory in the near future. This is an estimated value. See above.

"MCTL?"

Memory balloon driver is installed or not.

If not, install VMware tools which contains the balloon driver.

"MCTLSZ" (MB)

The amount of guest physical memory reclaimed by balloon driver.

This can be called "balloon size". A large "MCTLSZ" means lots of this VM's guest physical memory is "stolen" to decrease host memory pressure. This usually is not a problem, because balloon driver tends to smartly steal guest physical memory that cause little performance problems.

Q: How do I know the VM is ballooning?
A: If "MCTLSZ" is changing, balloon driver is actively reclaiming or releasing memory. I.e., the VM is ballooning. Please note that the ballooning rate for a short term can be estimated by the change of "MCTLSZ", assuming it is either increasing or decreasing. But, for a long term, we cannot do it this way, because that monotonically increase/decrease assumption may not hold.

Q: Does ballooning hurt VM performance?
A: If guest working set is smaller than guest physical memory after ballooning, guest applications won't observe any performance degradation. Otherwise, it may cause guest swapping and hurt guest application performance.

Please check what causes ballooning and take appropriate actions to reduce memory pressure. There are two possible reasons: (1) The host does not have enough machine memory for use. (2) Memory used by the VM reaches the "limit" setting of itself or "limit" of the resource pools that contain this VM. In either case, ballooning is necessary and preferred over swapping.

"MCTLTGT" (MB)

The amount of guest physical memory to be kept in balloon driver. (TGT is short for "target".)

This is an internal counter, which is computed by ESX memory scheduler. Usually, there is no need to worry about this.

Roughly speaking, "MCTLTGT" is computed based on "SZTGT" and current memory usage, so that the VM can balloon appropriate amount of memory. If "MCTLTGT" is greater than "MCTLSZ", VMKernel initiates inflating the balloon immediately, causing more VM memory to be reclaimed. If "MCTLTGT" is less than "MCTLSZ", VMKernel will deflate the balloon when the guest is requesting memory, allowing the VM to map/consume additional memory if it needs it. Please refer to "Resource Management Guide" for details.

Q: Why is it possible for "MCTLTGT" to be less than "MCTLSZ" for a long time?
A: If "MCTLTGT" is less than "MCTLSZ", VMKernel allows the balloon to deflate. But, balloon deflation happens lazily until the VM requests new memory. So, it is possible for "MCTLTGT" to be less than "MCTLSZ" for a long time, when the VM is not requesting new memory.

"MCTLMAX" (MB)

The maximum amount of guest physical memory reclaimable by balloon driver.

This value can be set via vmx option "sched.mem.maxmemctl". If not set, it is determined by the guest operating system type. "MCTLTGT" will never be larger than "MCTLMAX".

If the VM suffers from ballooning, "sched.mem.maxmemctl" can be set to a smaller value to reduce this possibility. Remember that doing so may result in host swapping during resource contention.

"SWCUR" (MB)

Current swap usage.

For a VM, it is the current amount of guest physical memory swapped out to the backing store. Note that it is the VMKernel swapping not the guest OS swapping.

It is the sum of swap slots used in the vswp file or system swap, and migration swap. Migration swap is used for a VMotioned VM to hold swapped out memory on the destination host, in case the destination host is under memory pressure.

Q: What does it mean if "SWCUR" of my VM is high?
A: It means the VM's guest physical memory is not resident in machine memory, but on disk. If those memory will not be used in the near future, it is not an issue. Otherwise, those memory will be swapped in for guest's use. In that case, you will see some swap-in activities via "SWR/s", which may hurt the VM's performance.

"SWTGT" (MB)

The expected swap usage. (TGT is short for "target".)

This is an internal counter, which is computed by ESX memory scheduler. Usually, there is no need to worry about this.

Roughly speaking, "SWTGT" is computed based on "SZTGT" and current memory usage, so that the VM can swap appropriate amount of memory. Again, note that it is the VMKernel swapping not the guest swapping. If "SWTGT" is greater than "SWCUR", VMKernel starts swapping immediately, causing more VM memory to be swapped out. If "SWTGT" is less than "SWCUR", VMKernel will stop swapping. Please refer to "Resource Management Guide" for details.

Q: Why is it possible for "SWTGT" to be less than "SWCUR" for a long time?
A: Since swapped memory stays swapped until the VM accesses it, it is possible for "SWTGT" be less than "SWCUR" for a long time.

"SWR/s" (MB)

Rate at which memory is being swapped in from disk. Note that this stats refers to the VMKernel swapping not the guest swapping.

When a VM is requesting machine memory to back its guest physical memory that was swapped out to disk, VMKernel reads in the page. Note that the swap-in operation is synchronous.

Q: What does it mean if SWR/s is high?
A: It is very bad for VM's performance. Because swap-in is synchronous, the VM needs to wait until the requested pages are read into machine memory. This happens when VMKernel swapped out the VM's memory before and the VM needs them now. Please refer to "SWW/s".

"SWW/s" (MB)

Rate at which memory is being swapped out to disk. Note that this stats refers to the VMKernel swapping not the guest swapping.

As discussed in "SWTGT", if "SWTGT" is greater than "SWCUR", VMKernel will swap out memory to disk. It happens usually in two situations. (1) The host does not have enough machine memory for use. (2) Memory used by the VM reaches the "limit" setting of itself or "limit" of the resource pools that contain this VM.

Q: What does it mean if SWW/s is high?
A: It is very bad for VM performance. Please check the above two reasons and fix your problem accordingly.

If this VM is swapping out memory due to resource contention, it usually means VMKernel does not have enough machine memory to meet memory demands from all the VMs. So, it will swap out mapped guest physical memory pages to make room for the recent requests.

"SHRD" (MB)

Amount of guest physical memory that are shared.

VMKernel page sharing module scans and finds guest physical pages with the same content and backs them with the same machine page. "SHRD" accounts the total guest physical pages that are shared by the page sharing module.

"ZERO" (MB)

Amount of guest physical zero memory that are shared. Thisis an internal counter.

A zero page is simply the memory page that is all zeros. If a zero guest physical page is detected by VMKernel page sharing module, this page will be backed by the same machine page on each NUMA node. Note that "ZERO" is included in "SHRD".

"SHRDSVD" (MB)

Estimated amount of machine memory that are saved due to page sharing.

Because a machine page is shared by multiple guest physical pages, we only charge "1/ref" page as the consumed machine memory for each of the guest physical pages, where "ref" is the number of references. So, the saved machine memory will be "1 - 1/ref" page."SHRDSVD" estimates the total saved machine memory for the VM.

The consumed machine memory by the VM can be estimated as "GRANT" - "SHRDSVD".

"COWH" (MB)

Amount of guest physical hint pages for page sharing. This is an internal counter.

"OVHDUW" (MB)

Amount of overhead memory reserved for the vmx user world of a VM group. This is an internal counter.

"OVHDUW" is part of "OVHDMAX".

"OVHD" (MB)

Amount of overhead memory currently consumed by a VM.

"OVHD" includes the overhead memory consumed by the monitor, the VMkernel and the vmx user world.

"OVHDMAX" (MB)

Amount of reserved overhead memory for the entire VM.

"OVHDMAX" is the overhead memory a VM wants to consume in the future. This amount of reserved overhead memory includes the overhead memory reserved by the monitor, the VMkernel, and the vmx user world. Note that the actual overhead memory consumption is less than "OVHDMAX". "OVHD" < "OVHDMAX".

"OVHDMAX" can be used as a conservative estimate of the total overhead memory.

Section 4 Disk

Section 4.1 Adapter, Device, VM screens

The ESX storage stack adds a few layers of code between a virtual machine and bare hardware. All virtual disks in virtual machines are seen as virtual SCSI disks. The ESX storage stack allows these virtual disks to be located on any of the multiple storage options available.

For performance analysis purposes, an IO request from an application in a virtual machine traverses through multiple levels of queues, each associated with a resource, in the guest OS, the VMkernel and the physical storage. (Note that physical storage could be an FC- or IP- SAN or disk array.) Each queue has an associated latency, dictated by its size and whether the IO load is low or high, which affects the throughput and latency seen by applications inside VMs.

Esxtop shows the storage statistics in three different screens: adapter screen, device screen, and vm screen. Interactive command 'd' can be used to switch to the adapter screen, 'u' for the device screen, and 'v' for the vm screen.

The main difference in the data seen in these three screens is the level at which it is aggregated, even though these screens have similar counters. By default, data is rolled up to the highest level possible for each screen. (1) On the adapter screen, by default, the statistics are aggregated per storage adapter but they can also be expanded to display data per storage channel, target, path or world using a LUN. See interacitive commands, 'e', 'E', 'P', 'a', 't', 'l', for the expand operations. (2) On the device screen, by default, statistics are aggregated per storage device. Statistics can also be viewed per path, world, or partition. See interactive commands, 'e', 'p', 't', for the expand operations. (3) On the VM screen, statistics are aggregated on a per-group basis by default. One VM has one corresponding group, so they are equivalent to per-VM statistics. You can use interactive command 'V' to show only statistics related to VMs. Statistics can also be expanded so that a row is displayed for each world or a per-world-per-device basis. See interactive commands, 'e' and 'l'.

Please refer to esxtop man page for the details of the interactive commands.

Section 4.2 Disk Statistics

Due to the similarities in the counters of the three disk screens, this section discusses the counters without distinguishing the screens. Similar to other esxtop screens, the storage counters are also organized in different sets, each of which contains related counters. The counters can be selected as a set by selecting the appropriate field option in esxtop. If esxtop is used in batch mode, make sure that the esxtop configuration file includes all counters of interest.

Each group of counters in the following subsections corresponds to a particular field option.

Section 4.2.1 I/O Throughput Statistics

CMDS/s

Number of commands issued per second.

READS/s

Number of read commands issued per second.

WRITES/s

Number of write commands issued per second.

MBREAD/s

Megabytes read per second.

MBWRTN/s

Megabytes written per second.

Section 4.2.2 Latency Statistics

This group of counters report latency values measured at three different points in the ESX storage stack. In the context of the figure below, the latency counters in esxtop report the Guest, ESX Kernel and Device latencies. These are under the labels GAVG, KAVG and DAVG, respectively. Note that GAVG is the sum of DAVG and KAVG counters.

Note that esxtop shows the latency statistics for different objects, such as adapters, devices, paths, and worlds. They may not perfectly match with each other, since their latencies are measured at the different layers of the ESX storage stack. To do the correlation, you need to be very familiar with the storage layers in ESX Kernel, which is out of our scope.

Latency values are reported for all IOs, read IOs and all write IOs. All values are averages over the measurement interval.

All IOs: KAVG/cmd, DAVG/cmd, GAVG/cmd, QAVG/cmd
Read IOs: KAVG/rd, DAVG/rd, GAVG/rd, QAVG/rd
Write IOs: KAVG/wr, DAVG/wr, GAVG/wr, QAVG/wr

GAVG

This is the round-trip latency that the guest sees for all IO requests sent to the virtual storage device.

GAVG should be close to the R metric in the figure.

Q: What is the relationship between GAVG, KAVG and DAVG?
A: GAVG = KAVG + DAVG

KAVG

These counters track the latencies due to the ESX Kernel's command.

The KAVG value should be very small in comparison to the DAVG value and should be close to zero. When there is a lot of queuing in ESX, KAVG can be as high, or even higher than DAVG. If this happens, please check the queue statistics, which will be discussed next.

DAVG

This is the latency seen at the device driver level. It includes the roundtrip time between the HBA and the storage.

DAVG is a good indicator of performance of the backend storage. If IO latencies are suspected to be causing performance problems, DAVG should be examined. Compare IO latencies with corresponding data from the storage array. If they are close, check the array for misconfiguration or faults. If not, compare DAVG with corresponding data from points in between the array and the ESX Server, e.g., FC switches. If this intermediate data also matches DAVG values, it is likely that the storage is under-configured for the application. Adding disk spindles or changing the RAID level may help in such cases.

QAVG

The average queue latency. QAVG is part of KAVG.

Response time is the sum of the time spent in queues in the storage stack and the service time spent by each resource in servicing the request. The largest component of the service time is the time spent in retrieving data from physical storage. If QAVG is high, another line of investigation is to examine the queue depths at each level in the storage stack.

Section 4.2.3 Queue Statistics

AQLEN

The storage adapter queue depth. This is the maximum number of ESX Server VMKernel active commands that the adapter driver is configured to support.

LQLEN

The LUN queue depth. This is the maximum number of ESX Server VMKernel active commands that the LUN is allowed to have. (Note that, in this document, the terminologies of LUN and Storage device can be used interchangeably.)

WQLEN

The World queue depth. This is the maximum number of ESX Server VMKernel active commands that the World is allowed to have. Note that this is a per LUN maximum for the World.

ACTV

The number of commands in the ESX Server VMKernel that are currently active. This statistic is only applicable to worlds and LUNs.

Please refer to %USD.

QUED

The number of commands in the VMKernel that are currently queued. This statistic is only applicable to worlds and LUNs.

Queued commands are commands waiting for an open slot in the queue. A large number of queued commands may be an indication that the storage system is overloaded. A sustained high value for the QUED counter signals a storage bottleneck which may be alleviated by increasing the queue depth. Check that LOAD < 1 after increasing the queue depth. This should also be accompanied by improved performance in terms of increased cmd/s.

Note that there are queues in different storage layers. You might want to check the QUED stats for devices, and worlds.

%USD

The percentage of queue depth used by ESX Server VMKernel active commands. This statistic is only applicable to worlds and LUNs.

%USD = ACTV / QLEN * 100%

For world stats, WQLEN is used as the denominator. For LUN (aka device) stats, LQLEN is used as the denominator.

%USD is a measure of how many of the available command queue "slots" are in use. Sustained high values indicate the potential for queueing; you may need to adjust the queue depths for system’s HBAs if QUED is also found to be consistently > 1 at the same time. Queue sizes can be adjusted in a few places in the IO path and can be used to alleviate performance problems related to latency. For detailed information on this topic please refer to the VMware whitepaper entitled "Scalable Storage Performance".

LOAD

The ratio of the sum of VMKernel active commands and VMKernel queued commands to the queue depth. This statistic is only applicable to worlds and LUNs.

The sum of the active and queued commands gives the total number of outstanding commands issued by that virtual machine. The LOAD counter values is the ratio of this value with respect to the queue depth. If LOAD > 1, check the value of the QUED counter.

Section 4.2.4 Error Statistics

ABRTS/s

The number of commands aborted per second.

It can indicate that the storage system is unable to meet the demands of the guest operating system. Abort commands are issued by the guest when the storage system has not responded within an acceptable amount of time, e.g. 60 seconds on some windows OS’s. Also, resets issued by a guest OS on its virtual SCSI adapter will be translated to aborts of all the commands outstanding on that virtual SCSI adapter.

RESETS/s

The number of commands reset per second.

Section 4.2.5 PAE Statistics

PAECMD/s

The number of PAE commands per second.

It may point to hardware misconfiguration. When the guest allocates a buffer, the vmkernel assigns some machine memory, which might come from a “highmem” region. If you have a driver that is not PAE-aware, then this counter is updated if accesses to this memory region result in copies by the vmkernel into a lower memory location before issuing the request to the adapter. This might happen if you do not populate the DIMMs with low memory first, then you may artificially cause “highmem” memory accesses.

PAECP/s

The number of PAE copies per second.

Section 4.2.6 Split Statistics

SPLTCMD/s

The number of split commands per second.

Commands can be split when they reach the vmkernel. This might impact perceived latency to the guest. The guest may be issuing commands of large block sizes which have to be broken down by the vmkernel. For ESX3.0.x, guest requests greater than 128KB are split into 128KB chunks. Since few applications do larger than 128KB ops, this is unlikely to be an issue. Splitting can also occur when IOs fall across partition boundaries but these are easily differentiated from the splitting as a result of the IO size.

SPLTCP/s

The number of split copies per second.

Section 4.3 Batch Mode Output

Esxtop batch mode output can be loaded in perfmon directly. It uses a csv (comma separated values) format. The instance type can be ideitified via its name. Because there are quite a number of instances related to disk statistics, let's list a few examples below. You may easily match the format in your own environment.

LUN (aka device): "\\<host>\Physical Disk(DEV-vmhba0:0:0)\<counter>"
Partition: "\\<host>\Physical Disk(PN-vmhba0:0:0-1)\<counter>"
Path: "\\<host>\Physical Disk(PH-vmhba0:C0:T0:L0)\<counter>"
Per-World-Per-Device: "\\<host>\Physical Disk(WD-vmhba0:0:0-1024)\<counter>"
Adapter: "\\<host>\Physical Disk(vmhba0)\<counter>"

Section 5 Network

Section 5.1 Port

We arrange the network stats per port of a virtual switch. "PORT-ID" identifies the port and "DNAME" shows the virtual switch name. A port can be linked to a physical NIC as an uplink, or can be connected by a virtual NIC. "UPLINK" indicates whether the port is an uplink.

If the port is an uplink, i.e., "UPLINK" is 'Y', "USED-BY" shows the physical NIC name.

If the port is connected by a virtual NIC, i.e., "UPLINK" is 'N', "USED-BY" shows the port client name. (a) If the port is used by a virtual machine, the client name contains a world id and the VM name. The world id identifies the leader world of the VM group. Note that "vswif" is used by COS (on classic ESX). (b) If the port is used by VMKernel system, there is no world id. The client name can be used to identify the use of the port. To give two examples.

"vmk" is a port used by vmkernel. Users can create vmk NICs for their uses, such as VMotion. On ESXi, there will be at least one vmk NIC to communicate with outside of the host.
"Management" is a management port for a portset. This is internal. Usually no need to worry about it.

For each non-uplink port, the NIC teaming policy determines which physical NIC is in charge of the port. "TEAM-PNIC" shows the physical NIC name, if valid. Please refer to NIC teaming documentation for details.

Section 5.2 Port Statistics

"SPEED" (Mbps)

The link speed in Megabits per second. This information is only valid for a physical NIC.

"FDUPLX"

'Y' implies the corresponding link is operating at full duplex. 'N' implies it is not. This information is only valid for a physical NIC.

"UP"

'Y' implies the corresponding link is up. 'N' implies it is not. This information is only valid for a physical NIC.

"PKTTX/s"

The number of packets transmitted per second.

"PKTRX/s"

The number of packets received per second.

"MbTX/s" (Mbps)

The MegaBits transmitted per second.

"MbRX/s" (Mbps)

The MegaBits received per second.

Q: Why does MbRX/s not match PKTRX/s for different workloads?
A: This is because the packet size may not be the same. The average packet size can be computed as follows: average_packet_size = MbRX/s / PKTRX/s . A large packet size may improve CPU efficiency of processing the packet. However, it may potentially increase latency.

"%DRPTX"

The percentage of transmit packets dropped.

"%DRPTX" = "dropped Tx packets" / ("success Tx packets" + "dropped Tx packets")

Q: What does it mean if %DRPTX is high?
A: This usually means the network transmit performance is bad. Please check whether the phsycial NICs are fully utilizing their capacity. You probably need physical NICs with better performance. Or, you may add more physical NICs and use a good NIC teaming load balancing policy.

"%DRPRX"

The percentage of receive packets dropped.

"%DRPRX" = "dropped Rx packets" / ("success Rx packets" + "dropped Rx packets")

Q: What does it mean if %DRPRX is high?
A: This usally means the network recieve performance is bad. Try to give more CPU resource to the impacted VM, or increase the ring buffer size.

"ACTN/s"

Number of actions per second. The actions here are VMkernel actions. It is an internal counter. We won't discuss it further here.

Section 6. Interrupt

Interrupt screens are under development for our next release.

Section 7. Batch Mode

Esxtop batch mode output uses a csv (comma separated values) format. The first line contains the names of the performance counters and their instances. Each of the following lines contains the performance data for those counter instances in one snapshot.

One way to read the batch mode output file is to load it in Windows perfmon. (1) Run perfmon; (2) Type "Ctrl + L" to view log data; (3) Add the file to the "Log files" and click OK; (4) Choose the counters to show the performance data. Each batch mode counter has a category name (listed as a performance object in perfmon) and a counter name (listed in the counter list in perfmon).

The counter names in esxtop batch mode are different from the ones in interactive mode listed in the sections above. The tables below describe their relationships. The first column is the interactive mode counter name; the second column is the batch mode counter category; the last column is the batch mode counter name.

Table 7-1 CPU Batch Mode Counters

Counter Name	Batch Mode Category	Batch Mode Counter Name
CPU load average	Physical Cpu Load	Cpu Load (1 Minute Avg)
		Cpu Load (5 Minute Avg)
		Cpu Load (15 Minute Avg)
PCPU(%)	Physical Cpu	% Processor Time
LCPU(%)	Logical Cpu	% Processor Time
CCPU(%) us	Console Physical Cpu	% User Time
CCPU(%) sy	Console Physical Cpu	% System Time
CCPU(%) id	Console Physical Cpu	% Idle Time
CCPU(%) wa	Console Physical Cpu	% I/O Wait Time
CCPU(%) cs/sec	Console Physical Cpu	% Context Switches/sec

%USED	Group Cpu (or Vcpu)	% Used
%SYS	Group Cpu (or Vcpu)	% System
%OVRLP	Group Cpu (or Vcpu)	% Overlap
%RUN	Group Cpu (or Vcpu)	% Run
%RDY	Group Cpu (or Vcpu)	% Ready
%MLMTD	Group Cpu (or Vcpu)	% Max Limited
%CSTP	Group Cpu (or Vcpu)	% CoStop
%WAIT	Group Cpu (or Vcpu)	% Wait
%IDLE	Group Cpu (or Vcpu)	% Idle
%SWPWT	Group Cpu (or Vcpu)	% Swap Wait

Table 7-2 Memory Batch Mode Counters

Counter Name	Batch Mode Category	Batch Mode Counter Name
MEM overcommit avg	Memory	Memory Overcommit (1 Minute Avg)
		Memory Overcommit (5 Minute Avg)
		Memory Overcommit (15 Minute Avg)
PMEM total	Memory	Machine MBytes
PMEM cos	Memory	Console MBytes
PMEM vmk	Memory	Kernel MBytes
PMEM other	Memory	NonKernel MBytes
PMEM free	Memory	Free MBytes
VMKMEM managed	Memory	Kernel Managed MBytes
VMKMEM minfree	Memory	Kernel MinFree MBytes
VMKMEM rsvd	Memory	Kernel Reserved MBytes
VMKMEM ursvd	Memory	Kernel Unreserved MBytes
VMKMEM state	Memory	Kernel State (0: high, 1: soft, 2:hard, 3: low)
COSMEM free	Console Memory	Free MBytes
COSMEM swap_t	Console Memory	Swap Total MBytes
COSMEM swap_f	Console Memory	Swap Free MBytes
COSMEM r/s	Console Memory	Swap MBytes Read/sec
COSMEM w/s	Console Memory	Swap MBytes Write/sec
NUMA	Numa Node	Total MBytes
		Free MBytes
PSHARE shared	Memory	PShare Shared MBytes
PSHARE common	Memory	PShare Common MBytes
PSHARE saving	Memory	PShare Savings MBytes
SWAP curr	Memory	Swap Used MBytes
SWAP target	Memory	Swap Target MBytes
SWAP r/s	Memory	Swap MBytes Read/sec
SWAP w/s	Memory	Swap MBytes Write/sec
MEMCTL curr	Memory	Memctl Current MBytes
MEMCTL target	Memory	Memctl Target MBytes
MEMCTL max	Memory	Memctl Max MBytes

MEMSZ	Group Memory	Memory Size MBytes
GRANT	Group Memory	Memory Granted Size MBytes
SZTGT	Group Memory	Target Size MBytes
TCHD	Group Memory	Touched MBytes
%ACTV	Group Memory	% Active Estimate
%ACTVS	Group Memory	% Active Slow Estimate
%ACTVF	Group Memory	% Active Fast Estimate
%ACTVN	Group Memory	% Active Next Estimate
MCTL?	Group Memory	Memctl?
MCTLSZ	Group Memory	Memctl MBytes
MCTLTGT	Group Memory	Memctl Target MBytes
MCTLMAX	Group Memory	Memctl Max MBytes
SWCUR	Group Memory	Swapped MBytes
SWTGT	Group Memory	Swap Target MBytes
SWR/s	Group Memory	Swap Read MBytes/sec
SWW/s	Group Memory	Swap Written MBytes/sec
SHRD	Group Memory	Shared MBytes
ZERO	Group Memory	Zero MBytes
SHRDSVD	Group Memory	Shared Saved MBytes
COWH	Group Memory	Copy On Write Hint MBytes
OVHDUW	Group Memory	Overhead UW MBytes
OVHD	Group Memory	Overhead MBytes
OVHDMAX	Group Memory	Overhead Max MBytes

Table 7-3 Disk Batch Mode Counters

Counter Name	Batch Mode Category	Batch Mode Counter Name
CMDS/s	Physical Disk	Commands/sec
READS/s	Physical Disk	Reads/sec
WRITES/s	Physical Disk	Writes/sec
MBREAD/s	Physical Disk	MBytes Read/sec
MBWRTN/s	Physical Disk	MBytes Written/sec
KAVG/cmd	Physical Disk	Average Kernel MilliSec/Command
DAVG/cmd	Physical Disk	Average Driver MilliSec/Command
GAVG/cmd	Physical Disk	Average Guest MilliSec/Command
QAVG/cmd	Physical Disk	Average Queue MilliSec/Command
KAVG/rd	Physical Disk	Average Kernel MilliSec/Read
DAVG/rd	Physical Disk	Average Driver MilliSec/Read
GAVG/rd	Physical Disk	Average Guest MilliSec/Read
QAVG/rd	Physical Disk	Average Queue MilliSec/Read
KAVG/wr	Physical Disk	Average Kernel MilliSec/Write
DAVG/wr	Physical Disk	Average Driver MilliSec/Write
GAVG/wr	Physical Disk	Average Guest MilliSec/Write
QAVG/wr	Physical Disk	Average Queue MilliSec/Write
AQLEN	Physical Disk	Adapter Q Depth
LQLEN	Physical Disk	Lun Q Depth
DQLEN	Physical Disk	Device Q Depth
WQLEN	Physical Disk	World Q Depth
ACTV	Physical Disk	Active Commands
QUED	Physical Disk	Queued Commands
%USD	Physical Disk	% Used
LOAD	Physical Disk	Load
LOAD	Physical Disk	Load
ABRTS/s	Physical Disk	Aborts/sec
RESETS/s	Physical Disk	Resets/sec
PAECMD/s	Physical Disk	PAE Commands/sec
PAECP/s	Physical Disk	PAE Copies/sec
SPLTCMD/s	Physical Disk	Split Commands/sec
SPLTCP/s	Physical Disk	Split Copies/sec

Table 7-4 Network Batch Mode Counters

Counter Name	Batch Mode Category	Batch Mode Counter Name
SPEED	Network Port	Link Speed (Mb/s)
FDUPLX	Network Port	Full Duplex?
UP	Network Port	Link Up?
PKTTX/s	Network Port	Packets Transmitted/sec
PKTRX/s	Network Port	Packets Received/sec
MbTX/s	Network Port	MBits Transmitted/sec
MbRX/s	Network Port	MBits Received/sec
%DRPTX	Network Port	% Outbound Packets Dropped
%DRPRX	Network Port	% Received Packets Dropped
ACTN/s	Network Port	Actions Posted/sec

Best Practices for SQL Server

Mon, 01 Dec 2008 20:45:05 GMT

Introduction

At VMworld 2008 in Las Vegas several of us in our virtual performance team met with a variety of customers to talk about Microsoft SQL Server. We already had a large base of customers running very many SQL Server DBs on our products and we wanted to collect information on the challenges posed in the process of virtualizing this critical workload. We were pleased to see that ESX Server handled SQL VMs with excellent performance. But, for many customers, the first efforts at virtualizing SQL didn't yield high-performing SQL VM. After careful investigation and many, many discussions we've started to put together the puzzle as to where SQL Server performance problems come from. This page will document these common problems, borrowing slides from our presentations on the subject.

Virtualizing SQL: The Checklist

We've talked with dozens of customers in the past months to document the issues that resulted in poor SQL performance. Happily, none of the issues were due to underlying technologies. Here is a list of issues and an explanation of the impacts. These items are roughly listed in the order of decreasing likelihood of occurrence.

Item 1: Configure Storage Correctly

Storage configuration problems are the number one cause of SQL performance issues. Usually these problems arise because the DBA requests a virtual disk of the VI admin, the VI admin places the VMDK on a LUN that may or may not meet the DBA's performance needs. For instance:

VMs' VMDK files placed on VMFS volumes without enough spindles.
Many VMDK files placed on a single VMFS volume which could use more spindles.
Database and log files placed on the same LUN which, you guessed it, could use more spindles.

This may be obvious to some, but this problem occurs again and again. The VI administrator should be aware of a few technical items that can help understand and avoid this problem:

Based on the IO demands of the DB files, a certain number of spindles should be guaranteed to this file. This means that its VMDK must be placed on a VMFS volume to accout for the SQL Server's demands and all of the other demands on that volume.
Mixing sequential activity (such as log file update) and random activity (such as database access) results in random behavior. This means that the LUN configuration in the pre-virtual physical environment may not be sufficient for the consolidated environment. This is discussed some in Storage Performance: VMFS and Protocols.
When storage isn't meeting the SQL Server's demands, the device latency or kernel latency (queueing time) will increase. Read up on these counters in Storage Performance Analysis and Monitoring.

Item 2: Use Recent Hardware

Often companies that are dipping their metaphorical toes into
virtualization want to run proof-of-concept (POC) experiments to verify
that the virtual platform can meet their performance expectations. But
its surprising how many times these experiments are run on older,
poorly-performing hardware. Presumably the shiny, new systems were in
use for production applications so only the mothballed, cobweb-covered
servers from a previous generation were available for the POC. This
causes many problems. Check out this slide from a talk on SQL Server at VMworld Europe 2009:

The slide points out a couple of things. First, the larger caches and shorter pipelines on newer Intel processors results in a considerable drops in performance overheads. Second, the latency of the VMEXIT instruction, which determines the amount of time it takes to transition from the VM to the VMkernel, has shrunk by a large amount with subsequent generations of hardware. And don't forget the other additions from Intel and AMD such as hardware assisted memory management and IO virtualization.

Item 3: Follow SQL Server Best Practices

Microsoft has kindly provided a web page of best practices for SQL Storage configuration. These be practices should still be followed when configuring your virtual SQL deployments!

Item 4: Configure VM Identically to Native and Run The Right Test

For many SQL Server POCs the goal is to measure the VM's ability to perform, with respect to the virtual platform. If this comparison is to be performed, its critical that the VM be configured identically to the physical hardware. Obviously this means that the VM should be run on the same hardware using identically configured LUNs. Its also important to ensure that the VM has the same number of vCPUs and amount of memory as the physical baseline. This means restricting the number of pCPUs and amount of memory with NUMPROC and MAXMEM, respectively, in boot.ini.

It also means that the test being applied should be understood. If a benchmark is chosen that uses a very small database, the content will be cached and the storage system won't be used. This can skew the results and produce recommendations not consistent with production deployments. Here is another slide from the same VMworld Europe 2009 presentation detailing some of what we know about the SQL Server benchmarking alternatives:

We at VMware prefer DVD Store.

Item 5: Use VMware's ESX Server

VMware's hosting products, VMware Server, VMware Workstation, and even VMware Fusion, are all capable of running SQL Server. But if the database is going to be run in production on enterprise-class hardware, use VMware's enterprise-class hypervisor: ESX Server. These products are not often confused by the initiated but rogue members of large companies often run off-the-books proof-of-concept experiments on VMware's hosted products. When they produce results they don't like, the results get spread throughout the company which can slow the virtual deployment.

Consider the following data, again from the VMworld Europe 2009 SQL Server presentation:

This information is getting a bit dated now, as it was performed years ago on ESX Server 3.0. But the point stands: before believing results claiming that "VMware cannot run SQL Server" its worth investigating the platform used to generate the results.

Item 6: Understand Memory Management and Configure Correctly

Database performance is heavily dependent on the amount of memory available. Almost without exception, providing more memory to SQL Server will improve performance. However, if that memory is coming from a host that is already over-committed or is being provided through workarounds to 32-bit limitations, performance may suffer. Here are a few keys for SQL Server memory management:

If more than 3 GB is desired, use 64-bit versions of the OS and application.
If memory is over-committed on the box, set reservations for performance-critical SQL Server VMs to guarantee that those VMs' memory isn't ballooned or swapped out.
If SQL Server's "lock pages in memory" parameter has been set, provide set the VM's reservations to the amount of memory in the VM. This setting can adversely interfere with ESX Server's balloon driver. Setting reservations will stop the balloon driver from inflating into the VM's memory space.

Item 7: Align Disk Partitions

This item is really a special but very important case of item two, follow best practices. Partition alignment can impact storage performance which can be critical to some SQL Server VMs' performance. See VMware's paper on partition alignment for more information on this.

Whitepapers

Ready Time

Wed, 27 Aug 2008 17:50:36 GMT

Introduction

Ready time is as important as it is confusing. I'm going to collect a few thoughts on ready time in this collection point with the hopes that some of the confusion around this important part of virtual system performance can be eliminated.

Details

Stated simply, ready time is the amount of time a VM wants to run but has not be provided CPU resources on which to execute. Somewhat confusingly, ready time is reported in two different values between esxtop and VirtualCenter. In esxtop is reported in an easily-consumed percentage format. A number of 5% means the VM spent 5% of its last sample period waiting for available CPU resources. In VirtualCenter ready time is reported as a time measurement. In VC's real-time data, which produces sample values every 20,000 ms, a number of 1,000 ms is reported for a 5% ready time.

There is so much more to know about ready time that I'm not going to reproduce here. Read the whitepaper on the subject for more details. There have been no changes in the details on ready time since ESX 3.0 that make that paper out-of-date.

Interpreting Ready Time Values

The most common question we get on ready time is, "what ready time numbers constitute a problem?" While there is no easy answer to this, we can offer some guidance on the acceptable values. But before I lay that out, let me say that ready time should not be the ultimate measurement of system performance. As always, user experience and latency should be. There are some situations where user experience is horrible on a system with no load and virtually zero ready time. This could happen with a mis-configured array, as an example. And occasionally we see aggressively-consolidated hosts showing very high ready times that are meeting user needs. There are no absolutes with ready time.

But, there are a few general regions into which ready time values can be binned. Note that these ready time values are per vCPU. esxtop reports ready time for a VM once its been summed up across all vCPUs. That means that 5% ready on each of four vCPUs will be reported as 20% ready at the VM level. This is the high end of a very light amount of ready time.

Value, per vCPU	Description
r == 0%	This doesn't happen. The very presence of a hypervisor between the operating system and the hardware means that there is a non-zero ready time on all operations. But on healthy systems this number is so small that end-users don't know their workload has been virtualized. See the next section.
0% < r <= 5%	This is the "normal" region for ready time. Very small single digit numbers result in a minimal impact to user experience. If performance problems exist on the system and ready time falls into this region, your problems lie elsewhere.
5% < r <= 10%	In this region ready time is starting to be worth watching. Most systems function healthily with ready time in this region but highly sensitive measurements may be suffering.
10% < r	While some systems continue to meet expectations, double-digit ready time percentages often mean some action is required to address performance issues. See the last section for guidance.

Again, remember that VirtualCenter performance numbers must be re-calculated to percentages to find the category on the above table. But since VC reports ready time per vCPU, no special arithmetic is needed to account for the number of vCPUs in the VM (as is needed with esxtop.)

Causes and Correction

There are two general areas that can cause unnecessarily high ready times:

Overloaded hosts.
Excessive use of SMP.

Host Overloading

The most common cause of high ready time is trying to get too much work out of too little hardware. Consider the following simple case: on a hypothetical system with only one physical CPU, if two 1-way VMs are fully loaded by their users then each wants to have an entire CPU. Because only one is available, ESX will time share that resource and give each of them only 50% of the CPU. As a result, each VM will spend 50% of its time waiting for the processor. This would be reported as 50% ready time.

Often this condition is observable when ready time is high and total host CPU utilization is also very high. The only fix for this is to back off the load on the system. VMs should be migrated off or processor resources should be increased.

Excessive SMP

In ESX Server 2.5, SMP guests had to be co-scheduled to start at the exact same moment. If a 2-way VM was ready to run but only one physical core was available, the VM would not be scheduled until a second core was freed up. This would increase its ready time. In ESX Server 3.0 and later versions, relaxed co-scheduling was introduced which meant that a subset of a VM's vCPUs could be scheduled ahead of others. However, guest operating systems still require some degree of co-scheduling which means that the relaxation isn't absolute. In short, increasing vCPUs still puts some burden on the scheduler to try and co-schedule the vCPUs that can increase ready time. This is one ready why VMware advises only allocating vCPUs to VMs that are using them. Read Co-scheduling SMP VMs in VMware ESX Server for more information on co-scheduling.

This condition is manifested by hosts that have sub-optimal CPU utilization and lots of SMP VMs. A host may have a dozen 4-way VMs with each showing high ready time but only be at an aggregate 40% CPU utilization. This is a clear sign that the scheduler is spending a great deal of time managing unneeded vCPUs.

Large Memory Pages

Fri, 08 Aug 2008 16:02:31 GMT

In ESX Server 3.5 VMware introduced support for large memory pages in the guest. Large memory pages, an architecteral feature available in x86 microprocessors for decades, can be used to improve performance on workloads that make use of them. With CPU, hypervisor, OS, and application support, throughputs can go up and CPU utilization can go down. Since applications such as Oracle databases and Java have been using large pages on Linux and Windows for years, the introduction of this support on ESX Server allows for increased gains in performance over previous virtual installs. VMware is currently the only virtualization vendor to support large pages.

VMware's support for large memory pages is detailed in the Large Page Performance performance study. That paper includes data on throughput gains in SPECjbb. The results are duplicated here:

At LinuxWorld 2008 VMware presented further data on the value of large memory pages with Oracle databases. Here is a chart showing those gains with VMware binary translation (BT):

Data also shared at LinuxWorld on the value of large memory pages with AMD Rapid Virtualization Indexing (RVI; formerly called NPT):

The gains derived from the presence of large pages in these Oracle/Swingbench results are atypical. Large pages have been documented in numerous locations to provide benefits between 5-20% on most database applications. The increases shown here (of up to 350%) are due to the specialized configuration which is less about demonstrating real-world application performance and more about stressing the underlying configuration to uncover strengths and weaknesses.

Storage Queues and Performance

Thu, 24 Jul 2008 00:51:16 GMT

Introduction

VMware recently published a paper titled Scalable Storage Performance that delivered a wealth of information on storage with respect to the ESX Server architecture. This paper contains details about the storage queues that are a mystery to many of VMware's customers and partners. I wanted to start a wiki article on some aspects of this paper that may be interesting to storage enthusiasts and performance freaks.

Two Important Queues

Let's use the following figure as a starting point for this discussion.

For the purposes of this paper, I'm going to call the two different queue types the "kernel queue" and the "device driver queue". The device driver queue is specified in the device itself and has historically been configured through Linux-like module commands in the console operating system. More on that in "Changing Queue Depth" below. The kernel queue should be thought of as infinitely long, for all practical purposes. Any time the device driver queue gets full, commands to the storage will queue up in the kernel.

Note that each LUN gets its own queue. This means that when you change the queue depth in the device driver, you're changing the queue depths for many queues. The underlying device (HBA) is going to have a hard limit on the number of active commands it will allow at one time. This should be considered when setting queue depth. If your HBA can support only 2,000 active commands but it is addressing 40 LUNs, a specified queue depth of 64 won't allow that many commands to all LUNs. This being due to the fact that 64*40 = 2,560--which is more than the 2,000 maximum commands. In practice this is rarely a concern, though, as rarely are so many LUNs being simultaneously addressed through a single HBA and so many outstanding commands being issued to these LUNs.

Device Driver Queue Function

The device driver queue is used for a low-level interaction with the storage device. It controls how many active, or "in flight", commands there can be at any one time. This is effectively the concurrency of the storage stack. Set the device queue to 1 and each storage command becomes sequential: each one must complete before the next starts.

But if the device queue is left at its default of 32, as an example, 32 commands will be concurrently processed by the storage system. All 32 will be shipped off to the storage device by the kernel and new commands are queued when completions arrive.

Kernel Queue Function

The kernel queue can be thought of as kind of an overflow queue for the device driver queues. But it's not just an overflow queue. ESX Server contains all kinds of cool optimizations to get the most out of your storage. And these features apply to commands in the kernel queue only. Here are some examples of features provided to commands queued at the kernel queues:

Multi-pathing for failover and load balancing.
Prioritization of storage activities based on VM and cluster shares.
Optimizations to improve efficiency for long sequential operations.

There are others, as well.

Impacts of Queue Depths

So, increasing queue depths in the device driver can greatly improve the performance of the storage at the device level. Decreasing the device driver queue will result in increases in usage of the kernel queues. This decreases the device efficiency, but introduces opportunities for optimizations across multiple VMs and devices. So, what's the right ratio of these two depths? We think that the sweet spot lies with a depth 32 device driver queue. That's why we've set 32 as the default device driver queue length.

But your configuration and workloads may benefit from a change to this default queue depth. I'll refer you to the aforementioned storage paper for information on when you might want to change the driver queue depth. I'll just point out a couple of broad observations here:

With fewer, very high IO VMs on a host, larger queues at the device driver will improve performance.
As the VM count grows and storage performance features--like shares, load balancing, failover, etc.--become more important, the default queue depth is best.
With too many servers each having too large of device queues, your storage array could easily be overloaded and see its performance suffer.

Improving Storage Performance

Now that we've covered how storage queuing works, you may be wondering how you can monkey around with these queue sizes for optimal performance. I can tell you as someone that has been involved with many, many performance analysis projects that changing queue size is rarely a fix to an acute storage performance problem. You should first go through the analysis techniques in Storage Performance Analysis and Monitoring. That may or may not lead to changing queue depths.

But, in the event that you do end up changing queue depths...

Changing Queue Depth

We have a helpful knowledge base article that describes the process of changing the device driver queue. Unfortunately, as of today (7/24/08) this document only describes how to change queues through the console operating system. No information is provided for ESXi. I've contacted the KB owner and will have that document updated ASAP.

Guest-based Performance Measurement

Tue, 03 Jun 2008 16:50:47 GMT

Because VMware products provide a virtual interface to the hardware, traditional performance instrumentation that is based on measuring hardware resources may not be accurate. As a result, Perfmon (in Windows) and top (in UNIX variants) will not provide accurate measurements of CPU utilization. The problems seen as a result of usage of traditional in-guest performance measurements come from three areas:

They are unaware of work being performance by the virtualization software, they will not have complete information on the resources being used by the virtualization software. This includes memory management, scheduling, and other support processes like the service console in ESX.
The way in which guest OSes account time is different and ineffective in a virtual machine.
Their visibility into available CPU resources is based on the fraction of the CPU that they have been provided by the virtualization software.

Items two and three are covered in more detail in a KB on the subject. Performance analysis on virtual deployments should always use host-based tools. On ESX Server, this means esxtop or VirtualCenter.

vCenter Performance Counters

Fri, 30 May 2008 00:15:21 GMT

Introduction

The following table of vCenter (VC) performance counters lists the counters with a description of their purpose. This page has been updated for vSphere 4, so the counter levels will differ slightly on older versions of VC.

Remember, with the exception of ready time, statistic levels one and two are the only ones needed for 99% of the performance monitoring and analysis out there. Don't spend many of your own cycles worrying about levels three and four!

For information on enabling VC to display and archive these counters see the Understanding vCenter Performance Statistics article.

Understanding vCenter Measurement Windows

Before you continue, you should know that all total count metrics reported by VC are reported over the sample window. When you're looking at live stats, this sample window is 20 seconds. When you're looking at archive stats, it will depend on the interval duration. That duration could be five minutes, 30 minutes, two hours, or one day.

This causes a lot of confusion when comparing esxtop results to live VC results to archived VC results. As an example, ready time might be reported as 10% in esxtop. In live VC results this amount of ready time would be reported as 2000 ms (10% of the 20s window.) In one day archive results, the same number would be reported as 30,000 ms (10% of the five minute interval duration.) All of these numbes reflect the same amount of ready time.

CPU Statistics

Level	Counter name in API	Description	Units
1	cpu.ready.summation	Ready time is the time spend waiting for CPU(s) to become available in the past update interval.	millisecond
1	cpu.usagemhz.average	The CPU utilization. The maximum possible value here is the frequency of the processors times the number of cores. As an example, a VM using 4000 MHz on a system with four 2 GHz processors is using 50% of the CPU (4000 / (4 * 2000) = 0.5)	megaHertz
1	cpu.usage.average	The CPU utilization. This value is reported with 100% representing all processor cores on the system. As an example, a 2-way VM using 50% of a four-core system is completely using two cores.	percent
2	cpu.reservedCapacity.average	CPU Reserved Capacity	megaHertz
2	cpu.idle.summation	CPU Idle	millisecond
2	cpu.swapwait.summation	Swap wait time is time that the world spent waiting for memory to be swapped in. When the VM is waiting for memory, it is not doing work.	millisecond
3	cpu.system.summation	System time is the time spent in VMkernel during the last update interval. This does not include guest code execution.	millisecond
3	cpu.wait.summation	Wait time is the time spent waiting for hardware or VMkernel lock thread locks during the last update interval.	millisecond
3	cpu.extra.summation	CPU extra is the time above the statically calculated entitlement. Entitlement is the share of processing time that a VM should get as a result of its vCPU count and assigned shares. You should not use or care about this counter in any of your own analysis.	millisecond
3	cpu.used.summation	CPU Used	millisecond
3	cpu.guaranteed.latest	Guaranteed time is reported as the amount of the reservation time that the VM used in the past update interval. As an example, if 2000 MHz have been reserved for the VM on an four-way, 2 GHz host, that's 25% of the CPU resource. In a 20s update interval, there are 80,000 ms available on this four-way system. That means 20,000 ms of time has been reserved. If a VM used only half of its available cycles, the guaranteed time is 10,000 ms.	millisecond
4	cpu.usage.none	CPU Usage (None)	percent
4	cpu.usage.minimum	CPU Usage (Minimum)	percent
4	cpu.usage.maximum	CPU Usage (Maximum)	percent
4	cpu.usagemhz.none	CPU Usage in MHz (None)	megaHertz
4	cpu.usagemhz.minimum	CPU Usage in MHz (Minimum)	megaHertz
4	cpu.usagemhz.maximum	CPU Usage in MHz (Maximum)	megaHertz

Memory Statistics

Level	Counter name in API	Description	units
1	mem.consumed.average	The amount of machine memory that is in use by the VM. While a VM may have been configured to use 4 GB of RAM, as an example, it might have only touched half of that. Of the 2 GB left, half of that might be saved from memory sharing. That would result in 1 GB of consumed memory.	kiloBytes
1	mem.overhead.average	The memory used by the VMkernel to maintain and execute the VM.	kiloBytes
1	mem.swapinrate.average	The swap in rate reports the rate at which a VM's memory is being swapped in from disk.	kiloBytesPerSecond
1	mem.swapoutrate.average	The swap out rate reports the rate at which a VM's memory is being swapped out to disk.	kiloBytesPerSecond
1	mem.usage.average	The percentage of memory used as a percent of all available machine memory. Available for host and VM.	percent
1	mem.vmmemctl.average	The amount of memory currently claimed by the balloon driver. This is not a performance problem, per se, but represents the host starting to take memory from less needful VMs for those with large amounts of active memory. But if the host is ballooning, check swap rates (swapin and swapout) which would be indicative of performance problems.	kiloBytes
2	mem.granted.average	The amount of memory that was granted to the VM by the host. Memory is not granted to the host until it is touched one time and granted memory may be swapped out or ballooned away if the VMkernel needs the memory.	kiloBytes
2	mem.active.average	The amount of memory used by the VM in the past small window of time. This is the "true" number of how much memory the VM currently has need of. Additional, unused memory may be swapped out or ballooned with no impact to the guest's performance.	kiloBytes
2	mem.shared.average	The average amount of shared memory. Shared memory represents the entire pool of memory from which sharing savings are possible. The amount of memory that this has been condensed to is reported in shared common memory. So, total saving due to memory sharing equals shared memory minus shared common memory.	kiloBytes
2	mem.zero.average	The amount of zero pages in the guest. Zero pages are not represented in machine memory so this results in 100% savings when mapping from the guest to the machine memory.	kiloBytes
2	mem.unreserved.average	Memory Unreserved (Average)	kiloBytes
2	mem.swapused.average	The amount of swap memory currently in use. A large amount of swap memory is not a performance problem. This could be memory that the guest doesn't need. Check the swap rates (swapin, swapout) to see if the guest is actively in need of more memory than is available.	kiloBytes
2	mem.swapunreserved.average	Memory Swap Unreserved (Average)	kiloBytes
2	mem.sharedcommon.average	The average amount of shared common memory. Shared memory represents the entire pool of memory from which sharing savings are possible. The amount of memory that this has been condensed to is reported in shared common memory. So, total saving due to memory sharing equals shared memory minus shared common memory.	kiloBytes
2	mem.heap.average	Memory Heap (Average)	kiloBytes
2	mem.heapfree.average	Memory Heap Free (Average)	kiloBytes
2	mem.state.latest	Memory State	number
2	mem.swapped.average	Memory Swapped (Average)	kiloBytes
2	mem.swaptarget.average	Memory Swap Target (Average)	kiloBytes
2	mem.swapin.average	The rate at which memory is being swapped in from disk. A large number here represents a problem with lack of memory and a clear indication that performance is suffering as a result.	kiloBytes
2	mem.swapout.average	The rate at which memory is being swapped out to disk. A large number here represents a problem with lack of memory and a clear indication that performance is suffering as a result.	kiloBytes
2	mem.vmmemctltarget.average	Memory Balloon Target (Average)	kiloBytes
2	mem.sysUsage.average	Memory Used by vmkernel	kiloBytes
2	mem.reservedCapacity.average	Memory Reserved Capacity	megaBytes
4	mem.usage.none	Memory Usage (None)	percent
4	mem.usage.minimum	Memory Usage (Minimum)	percent
4	mem.usage.maximum	Memory Usage (Maximum)	percent
4	mem.granted.none	Memory Granted (None)	kiloBytes
4	mem.granted.minimum	Memory Granted (Minimum)	kiloBytes
4	mem.granted.maximum	Memory Granted (Maximum)	kiloBytes
4	mem.active.none	Memory Active (None)	kiloBytes
4	mem.active.minimum	Memory Active (Minimum)	kiloBytes
4	mem.active.maximum	Memory Active (Maximum)	kiloBytes
4	mem.shared.none	Memory Shared (None)	kiloBytes
4	mem.shared.minimum	Memory Shared (Minimum)	kiloBytes
4	mem.shared.maximum	Memory Shared (Maximum)	kiloBytes
4	mem.zero.none	Memory Zero (None)	kiloBytes
4	mem.zero.minimum	Memory Zero (Minimum)	kiloBytes
4	mem.zero.maximum	Memory Zero (Maximum)	kiloBytes
4	mem.unreserved.none	Memory Unreserved (None)	kiloBytes
4	mem.unreserved.minimum	Memory Unreserved (Minimum)	kiloBytes
4	mem.unreserved.maximum	Memory Unreserved (Maximum)	kiloBytes
4	mem.swapused.none	Memory Swap Used (None)	kiloBytes
4	mem.swapused.minimum	Memory Swap Used (Minimum)	kiloBytes
4	mem.swapused.maximum	Memory Swap Used (Maximum)	kiloBytes
4	mem.swapunreserved.none	Memory Swap Unreserved (None)	kiloBytes
4	mem.swapunreserved.minimum	Memory Swap Unreserved (Minimum)	kiloBytes
4	mem.swapunreserved.maximum	Memory Swap Unreserved (Maximum)	kiloBytes
4	mem.sharedcommon.none	Memory Shared Common (None)	kiloBytes
4	mem.sharedcommon.minimum	Memory Shared Common (Minimum)	kiloBytes
4	mem.sharedcommon.maximum	Memory Shared Common (Maximum)	kiloBytes
4	mem.heap.none	Memory Heap (None)	kiloBytes
4	mem.heap.minimum	Memory Heap (Minimum)	kiloBytes
4	mem.heap.maximum	Memory Heap (Maximum)	kiloBytes
4	mem.heapfree.none	Memory Heap Free (None)	kiloBytes
4	mem.heapfree.minimum	Memory Heap Free (Minimum)	kiloBytes
4	mem.heapfree.maximum	Memory Heap Free (Maximum)	kiloBytes
4	mem.swapped.none	Memory Swapped (None)	kiloBytes
4	mem.swapped.minimum	Memory Swapped (Minimum)	kiloBytes
4	mem.swapped.maximum	Memory Swapped (Maximum)	kiloBytes
4	mem.swaptarget.none	Memory Swap Target (None)	kiloBytes
4	mem.swaptarget.minimum	Memory Swap Target (Minimum)	kiloBytes
4	mem.swaptarget.maximum	Memory Swap Target (Maximum)	kiloBytes
4	mem.swapin.none	Memory Swap In (None)	kiloBytes
4	mem.swapin.minimum	Memory Swap In (Minimum)	kiloBytes
4	mem.swapin.maximum	Memory Swap In (Maximum)	kiloBytes
4	mem.swapout.none	Memory Swap Out (None)	kiloBytes
4	mem.swapout.minimum	Memory Swap Out (Minimum)	kiloBytes
4	mem.swapout.maximum	Memory Swap Out (Maximum)	kiloBytes
4	mem.vmmemctl.none	Memory Balloon (None)	kiloBytes
4	mem.vmmemctl.minimum	Memory Balloon (Minimum)	kiloBytes
4	mem.vmmemctl.maximum	Memory Balloon (Maximum)	kiloBytes
4	mem.vmmemctltarget.none	Memory Balloon Target (None)	kiloBytes
4	mem.vmmemctltarget.minimum	Memory Balloon Target (Minimum)	kiloBytes
4	mem.vmmemctltarget.maximum	Memory Balloon Target (Maximum)	kiloBytes
4	mem.overhead.none	Memory Overhead (None)	kiloBytes
4	mem.overhead.minimum	Memory Overhead (Minimum)	kiloBytes
4	mem.overhead.maximum	Memory Overhead (Maximum)	kiloBytes
4	mem.consumed.none	Memory Consumed (None)	kiloBytes
4	mem.consumed.maximum	Memory Consumed (Maximum)	kiloBytes
4	mem.consumed.minimum	Memory Consumed (Minimum)	kiloBytes
4	mem.sysUsage.none	Memory Used by vmkernel	kiloBytes
4	mem.sysUsage.maximum	Memory Used by vmkernel	kiloBytes
4	mem.sysUsage.minimum	Memory Used by vmkernel	kiloBytes

Disk Statistics

Level	Counter name in API	Description	units
1	disk.maxTotalLatency	The highest reported total latency (device and kernel times) in the sample window.	milliseconds
1	disk.usage.average	Average disk throughput over the sample period.	kiloBytesPerSecond
2	disk.read.average	Average disk throughput due to read operaitons over the sample period.	kiloBytesPerSecond
2	disk.write.average	Average disk throughput due to write operations over the sample period.	kiloBytesPerSecond
2	disk.commands.summation	Disk Commands Issued	number
2	disk.commandsAborted.summation	The number of aborts that have occurred in the last window of time. Abort commands are issued by the guest when the storage system has not responded within an acceptable amount of time (as defined by the guest OS or application.)	number
2	disk.busResets.summation	Disk Bus Resets	number
2	disk.deviceReadLatency.average	Device read latency. This is the time the physical device from the HBA to the platter takes to service an IO request.	millisecond
2	disk.kernelReadLatency.average	Kernel read latency. This is the time the VMkernel takes to service an IO. This is the time between the guest OS and the device.	millisecond
2	disk.totalReadLatency.average	Total read latency. The sum of the device and kernel read latencies.	millisecond
2	disk.queueReadLatency.average	Queue Read Latency	millisecond
2	disk.deviceWriteLatency.average	Device write latency. This is the time the physical device from the HBA to the platter takes to service an IO request.	millisecond
2	disk.kernelWriteLatency.average	Kernel write latency. This is the time the VMkernel takes to service an IO. This is the time between the guest OS and the device.	millisecond
2	disk.totalWriteLatency.average	Total write latency. The sum of the device and kernel write latencies.	millisecond
2	disk.queueWriteLatency.average	Queue Write Latency	millisecond
2	disk.deviceLatency.average	Physical Device Command Latency	millisecond
2	disk.kernelLatency.average	Kernel Disk Command Latency	millisecond
2	disk.queueLatency.average	Queue Command Latency	millisecond
3	disk.numberRead.summation	The number of IO read operations in the previous sample period. Note that these operations may be variable sized up to 64 KB.	number
3	disk.numberWrite.summation	The number of IO write operations in the previous sample period. Note that these operations may be variable sized up to 64 KB.	number
3	disk.totalLatency.average	This is the average total latency over the sample window. Total latency is the sum of kernel and device latency for both read and write commands.	millisecond
3	disk.write.average	Disk Write Rate	kiloBytesPerSecond
4	disk.usage.none	Disk Usage (None)	kiloBytesPerSecond
4	disk.usage.minimum	Disk Usage (Minimum)	kiloBytesPerSecond
4	disk.usage.maximum	Disk Usage (Maximum)	kiloBytesPerSecond

Network Statistics

Level	Counter name in API	Description	units
1	net.usage.average	Network Usage (Average)	kiloBytesPerSecond
2	net.droppedRx.summation	The number of received packets that were dropped over the sample period.	number
2	net.droppedTx.summation	The number of transmitted packets that were dropped over the sample period.	number
2	net.received.average	Average network throughput for received traffic.	kiloBytesPerSecond
2	net.transmitted.average	Average network throughput for transmitted traffic.	kiloBytesPerSecond
3	net.packetsRx.summation	Network Packets Received	number
3	net.packetsTx.summation	Network Packets Transmitted	number
4	net.usage.none	Network Usage (None)	kiloBytesPerSecond
4	net.usage.minimum	Network Usage (Minimum)	kiloBytesPerSecond
4	net.usage.maximum	Network Usage (Maximum)	kiloBytesPerSecond

Other Statistics

Level	Counter name in API	Description	units
1	sys.uptime.latest	Uptime	second
1	sys.heartbeat.summation	Heartbeat	number
1	clusterServices.cpufairness.latest	CPU Fairness	number
1	clusterServices.memfairness.latest	Memory Fairness	number
1	clusterServices.effectivecpu.average	Effective CPU Resources	megaHertz
1	clusterServices.effectivemem.average	Effective Memory Resources	megaBytes
1	clusterServices.failover.latest	Current failover level	number
3	sys.resourceCpuUsage.average	Resource CPU Usage (Average)	megaHertz
3	managementAgent.memUsed.average	Memory Used (Average)	kiloBytes
3	managementAgent.swapUsed.average	Memory Swap Used (Average)	kiloBytes
3	managementAgent.swapIn.average	Memory Swap In (Average)	kiloBytesPerSecond
3	managementAgent.swapOut.average	Memory Swap Out (Average)	kiloBytesPerSecond
3	rescpu.actav1.latest	CPU Active (1 min. average)	percent
3	rescpu.actpk1.latest	CPU Active (1 min. peak)	percent
3	rescpu.runav1.latest	CPU Running (1 min. average)	percent
3	rescpu.actav5.latest	CPU Active (5 min. average)	percent
3	rescpu.actpk5.latest	CPU Active (5 min. peak)	percent
3	rescpu.runav5.latest	CPU Running (5 min. average)	percent
3	rescpu.actav15.latest	CPU Active (15 min. average)	percent
3	rescpu.actpk15.latest	CPU Active (15 min. peak)	percent
3	rescpu.runav15.latest	CPU Running (15 min. average)	percent
3	rescpu.runpk1.latest	CPU Running (1 min. peak)	percent
3	rescpu.maxLimited1.latest	CPU Throttled (1 min. average)	percent
3	rescpu.runpk5.latest	CPU Running (5 min. peak)	percent
3	rescpu.maxLimited5.latest	CPU Throttled (5 min. average)	percent
3	rescpu.runpk15.latest	CPU Running (15 min. peak)	percent
3	rescpu.maxLimited15.latest	CPU Throttled (15 min. average)	percent
3	rescpu.sampleCount.latest	Group CPU Sample Count	number
3	rescpu.samplePeriod.latest	Group CPU Sample Period	millisecond
4	sys.resourceCpuUsage.none	Resource CPU Usage (None)	megaHertz
4	sys.resourceCpuUsage.maximum	Resource CPU Usage (Maximum)	megaHertz
4	sys.resourceCpuUsage.minimum	Resource CPU Usage (Minimum)	megaHertz

Time-based Measurements in Virtual Machines

Thu, 29 May 2008 18:05:22 GMT

Introduction

All benchmarking relies on accurate time keeping so production can be measured with respect to the passage of time. Because hosted and hypervisor products virtualize the hardware timer, minute fluctuations in guest time keeping can occur. Details are provided on this in many locations including the following:

During benchmarking these fluctuations in time can cause unexpected results. Performance measurements can appear inflated if time slows down while work occurs, depressed if time accelerates while work is occurring, or somewhere in between. This topic will provide some details on this phenomenon.

If the hypervisor (or host operating system for a hosted product) is busy with other tasks it may stall slightly when delivering timer interrupts to the VM. This means that the guest timer appears to run slow. VMware products will correct for these deviations by pushing time back to its correct position but these "slow downs" and "catch ups" may occur at different points in a benchmark.

Artificially High Results

Consider the case where a great number of IO operations are measured over a long period of time. If a benchmark wants to run 10,000 IOPS and the native system takes 10ms to process each operation, the benchmark would measure 10ms on a native system. However, if the benchmarking is run on a virtual system and the virtualization software is busy servicing the IO operation instead of updating time, time may not progress properly during the operation. Although 10ms or more would have passed, perhaps the VM was only informed of the passing of 9ms. In this case, the operation appeared to run faster on the VM than the native system.

For benchmarks where many operations are measured of a large time window, this isn't a problem. On our 10K operation benchmark, if time were started before operation one and stopped after operation 10,000, a fluctuation of 1ms will make no difference. After all, that's only a 1ms inaccuracy on a sequence that would take 100s to run.

However, if the benchmark measured each operation individually and summarized them all to product a result, each individual 1ms inaccuracy would be summed over the entire run. The benchmark would report that the average IO length was 9ms even though observation of wall time would still show the passage of 100s during the 10,000 IO operation run.

Benchmarking

Wed, 28 May 2008 19:12:13 GMT

Introduction

Many VMware users wish to perform analysis on their own virtual deployments. This page will collect information on setting up and executing your own tests to analyze performance.

General Best Practices

Always measure performance from a native (non-virtual) system. Be aware that time measurements in virtual machines can be subject to minute fluctuations. Many benchmarks produce results by summing times from large number of small operations so these small inaccuracies can be compiled to produce a large error. See Time-based Measurements in Virtual Machines for more information on this subject. The only way to guarantee correct measurement is to run the measurement tool on a native system. This is easy for client-server test architectures but may require clever architecture for in-guest testing.

Always ensure apples-to-apples comparison. Make sure that the benchmark or application under test are both constrained by the same resources. For instance, if the virtual machine was configured with 512M of RAM and two virtual CPUs, restrict the native system to the same resources if a virtual-to-native comparison is desired.

Collect accurate host-based performance statistics. Guest OS performance metrics (such as CPU utilization) are not accurate. Use VirtualCenter or esxtop to collect accurate performance counters during the test. See the Performance Monitoring and Analysis for more information on analysis.

Application Benchmarking

Microsoft Exchange.

Microsoft SQL Server.

Subsystem Benchmarking

Storage

Internally at VMware we've used Iometer for a variety of storage analyses. See the Storage System Performance Analysis with Iometer for more information.

Best Practices for Oracle

Wed, 28 May 2008 00:01:33 GMT

While Oracle makes a lot of software for a bunch of purposes, the only best practice guidance available today is for Oracle databases. We'll build out pages dedicated to other Oracle products if and when other best practices are available.

Real Time Scheduling for Oracle DBs on Linux

The following nugget came from a database expert in performance engineering. Due to the possibility of this change screwing up your DBs it was only with great reluctance that he even committed this to ink. You're taking your own job in your hands if you try this with your production DBs. But, for those of you with a test/dev Oracle DB and an unrepentant need for speed, give this a try. --drummonds

Traditional Unix/Linux timeshare scheduler policies attempt to provide good interactive response times by favoring a process that has just had an I/O request completed over a running process. This can create havoc on a high transaction-rate database server, but can be avoided by manipulating the scheduling policies for the database processes.

The scheduler policies date back to the 1970s when we would be running an editor session simultaneously with nroff (a text formatting program that often ran for minutes to process a document) on a single-processor system. Allowing the long-running process to execute without preemption would mean unacceptably long response times for interactive applications. So, the scheduler decays the priority of a process as it runs and accumulates CPU time. Furthermore, when a timeshare priority process goes to sleep waiting for I/O completion, it sleeps at a stronger priority than any running process with a timeshare priority. This ensures, for example, that a text editor user will get immediate feedback for every character typed on the keyboard even if the system is busy running CPU hungry tasks.

The problem with this scheme comes to the forefront when we run an application such as Oracle on a modern computer system. In an online transaction processing environment, the database management system threads/processes typically run for a short period of time, in the order of milliseconds or tens of milliseconds, then issue a disk I/O or send a network packet. A typical large system may issue many thousands of disk accesses per second. Every time an I/O completes, the scheduler makes the issuing process (or thread) runable, and at a stronger priority than all running timeshare processes. So, we are guaranteed a preemption unless the system can find an idle CPU. And, the preempted process goes to the end of the run queue for its priority level.

Now, if a preemption occurs, and the running process was holding an important database resource, say, a latch, all other processes that may need that latch, possibly including the preemptor process itself, will go into a spin loop waiting for the latch, and will eventually put themselves to sleep until the process holding the latch runs again and releases the latch. This is very expensive in term of CPU usage. The spinning and the context switches will drive up the CPU utilization without an increase in the system throughput.

An even worse phenomenon can occur in a large system with a very high I/O rate, where the problem may actually exhibit itself as excess CPU idle due to processes putting themselves to sleep waiting for latches and not waking up when the latch is released. Another possible symptom is the thundering herd problem: once the process holding the latch runs and releases it, a large number of processes become runable, and chaos and inefficiency follows.

The simplest solution is to maintain a constant priority for the DBMS processes, even when sleeping. This way, we allow the running process to voluntarily give up the CPU, at which time it has supposedly released all latches. Because of the nature of OLTP workloads, processes will voluntarily give up the CPU after a few, or at most tens of, milliseconds. So avoiding involuntary preemptions will fix the problem.

One way of achieving this is using the real-time priority feature. We are not really interested in running the database processes at a strong priority. We only use real time priorities because of an additional feature that this scheduling policy offers: the priority of a real time process does not change when the process sleeps on I/O. As a result when an I/O completion occurs, a process becomes runable at the same priority as the currently running process, and is put at the end of the run queue. We avoid the involuntary preemption and the associated spinning and sleeping costs.

Here is a sample RHEL4.4 script that accomplishes this. This script will run the Oracle processes at a stronger priority than all timeshare priority processes and could result in starvation of other applications. It should be used only when you are certain that running these processes at a strong priority won't deny resources to other applications.

for DAEMON_PID in `ps -u oracle -f|grep -v grep|grep ora_|awk '{print $2}'`
do
sudo chrt --rr -p 82 --pid ${DAEMON_PID}
done
LGWR_PID=`ps -u oracle -f|grep -v grep|grep lgwr|awk '{print $2}'`
sudo chrt --rr -p 83 --pid ${LGWR_PID}
LSNR_PID=`ps -u oracle -f|grep -v grep|grep tnslsnr|awk '{print $2}'`
sudo chrt --rr -p 81 --pid ${LSNR_PID}
for SHADOW_PID in `ps -u oracle -f|grep -v grep|grep -v -E 'ora_|tnslsnr'|awk '{print $2}'`
do
sudo chrt --rr -p 81 --pid ${SHADOW_PID}
done

To avoid starving the Oracle daemons, we run them at a stronger priority than the shadow processes, with particular attention paid to the logwriter.

Best Practices for IIS

Tue, 27 May 2008 23:26:59 GMT

Microsoft Internet Information Services (IIS) web server is a common target for virtualization. This page will collect best practices for installation and configuration of IIS VMs for maximum performance.

No IIS-specific information has yet been provided by the Best Practices for Web Servers page contains hits common to all web servers.

Best Practices for Apache

Tue, 27 May 2008 23:24:29 GMT

Apache web servers are a common target for virtualization in almost every data center. This page will collect best practices for setting up and configuring Apache for best performance.

No Apache-specific information has yet been provided. But tips are available on the Best Practices for Web Servers page.

Best Practices for Web Servers

Tue, 27 May 2008 23:10:36 GMT

Introduction

This page is a collection point for ESX Server configuration options that can maximize web server performance regardless of the choice of web server.

TCP Transmit Coalescing

In a recent whitepaper published on SPECweb performance on VMware ESX Server, the use of TCP transmit coalescing was described to improve web server performance. The general idea behind transmit coalescing is to buffer TCP transmits at the ESX Server for a brief period of time to allow multiple packets to be transmitted at one time. This introduces a very slight increase in latency but can provide a dramatic increase in efficiency.

Transmit coalescing can be turned on with the following steps:

Using the VMware Infrastructure Client, choose the ESX Server host on which the virtual machine is deployed.
Click the Configuration tab.
Click Advanced Settings in the Software panel.
Click the Net tab.
Edit the Net.vmxnetThroughputWeight value to 128, then click OK.
Reboot the virtual machine.

Details and performance results are provided in the SPECweb paper .

Resources

Apache Best Practices

IIS Best Practices

VMkernel Scheduler

Tue, 27 May 2008 22:56:02 GMT

Introduction

Details on the ESX Server scheduler are commonly requested when I engage customers and partners. People want to know more about how the scheduler works, when SMP should be used, and what the deal is with SMP co-scheduling. This page will answer these questions and others as they arise in the forum or the discussion portion of this page.

Terminology and Architecture

In VMware parlance, the monitor is the part of our products that provides a virtual interface to the guest operating systems. The VMkernel is the part of our products that manages interactions with the devices, handles memory allocation, and schedules access to the CPU resources, among other things. This is shown in the following figure.

This document will provide information on one part of the VMkernel: the scheduler.

Performance Scaling and the Scheduler

It is a critical requirement for enterprise deployments that an operating system provide fast and fair access to the underlying resources. As a critical part of this design, the scheduler has undergone countless engineer-years of development to guarantee that this requirement is met. We've now released dozens of papers showing linear scaling of workloads as vCPU count is scaled up within a single VM and VM count is scaled up within a single host. Here are a few such papers that contain supporting data.

VM scaling as demonstrated by Oracle databases.
VM and vCPU scaling under IBM DB2 load.
VM and vCPU scaling with SQL Server running in the VM.

The scheduler's ability to fairly scale up to and beyond totally committed CPU resources is no accident. In fact, in a conversation I had with a QA manager I was assured that the VMkernel's scheduler would fairly distribute CPU resources to all VMs at least up to 4x CPU overcommitment. Of course, on a system with the CPU over-committed by 4x each VM will only run at 1/4 native speed but the scheduler keeps the VMs running at that performance. Not one at 1/8 speed, one at 1/10 speed, and another at 1/4 speed.

SMP and the Scheduler

As ESX Server supports uniprocessor (UP) and symmetric multiprocessor (SMP) VMs, the fair-and-fast requirement for the scheduler must be upheld in the presence of concurrently executing UP and SMP VMs. Internal testing of this requirement shows fair scheduling even in the presence of concurrently executing 1-way, 2-way, and 4-way VMs.

In fact, the ability to fairly execute under such environments is a very tricky problem for a scheduler. We've run analysis on competitors' products and found that the ability to fairly balance differently-sized VMs is something of which ESX Server alone is capable. Stay tuned in the coming months as we back this claim up with performance data.

Cell Size

One construct that assists the scheduler in optimally placing VMs on a heavily utilized system is a cell. A cell is a logical grouping of a subset of CPU cores in the system. In ESX 3 versions the cell size is equal to four. Since the cell is statically assigned to physical cores, this means that each four-core processor is in exactly one cell. When only dual-core processors are present, a cell is comprised of two sockets. The most important thing to know about cells is the following:

A VM cannot span more than one cell.

This means that four-way VMs run on only one socket at a time in systems with quad-core CPUs. For this case, the number of options presented to the scheduler is equal to the number of sockets. In future versions of ESX we plan to increase the cell size to eight. In some cases (such as systems with hexa-core CPUs) a modification of the cell size can improve performance. See KB article 1007361 for more information.

UP or SMP?

When and if to use SMP is a common question from VMware users. The simple answer to this is to only use SMP when needed. Why only use SMP when needed? There are two reasons:

SMP schedulers are less efficient than UP schedulers. This is a simple experiment that can be confirmed with trivial benchmarks like Netperf or Passmark. On UP systems (either virtual or native) the UP hardware abstraction layer (HAL) will provide marginally better results than the SMP HAL.
Even when unused, virtualization of idle vCPUs requires resources by the kernel. Memory is needed to maintain data structures and CPU resources are needed to virtualize the idle system. The amount of work needed to support an idle CPU varies greatly but usually is in the realm of 1-2% of a single CPU core.
The work required to deliver timer interrupts increases quadratically with the number of vCPUs, like RHEL5, the number of timing interrupts delivered by the VMkernel can be quite high. See Red Hat Enterprise Linux for more information on this issue.

What About Co-scheduling?

Back in the days of ESX Server 2.5, SMP VMs had to have their vCPUs co-scheduled at the same instant to begin running. Because only 2-way VMs were supported at this time, that meant that two CPU cores had to be available simultaneously to launch a 2-way VM. On a server with a total of only two cores, this meant that the VM could not be launched concurrently with any other process on the server. This would include the service console, the web interface, or any other process.

This requirement was reduced in ESX Server 3.0 through a process called relaxed co-scheduling. Effectively SMP VMs can have their vCPUs scheduled at slightly different times and idle vCPUs didn't necessarily have to be scheduled concurrently with running vCPUs. More details on this are available in the Co-scheduling SMP VMs in VMware ESX Server page.

NUMA Considerations

Support for non-uniform memory access (NUMA) architectures was introduced in ESX Server 2. This meant that the scheduler became aware that memory was not uniform across each CPU. Each CPU node had access to its own local memory and a larger pool of remote memory (which was divided as local memory for the other CPU nodes.) Memory access to local memory is much faster than remote memory so the scheduler should favor the placement of processes on nodes that held the processes' memory.

Subsequent generations of ESX Server continued to optimize for the use of NUMA memory. This included placement of vCPUs next to needed memory and startup of VMs at NUMA nodes with resources available for execution. All of this is transparently handled by the scheduler but it should be noted that the newer your version of ESX Server, the better its NUMA scheduling is.

Network Performance Analysis and Monitoring

Tue, 27 May 2008 21:05:20 GMT

Introduction

This page is a living, up-to-date version of the performance analysis methods whitepaper.

Check Utilization

esxtop will provide network information on the network screen which is displayed with the ‘n' key.

The following properties of this screen are worth particular attention:

Each row represents one of several relevant network items on the server: a physical NIC (vmnicX), a virtual switch interface (vswifX), a VM (contains the VM name), the VMkernel network stack (vmk-tcpip-A.B.C.D), and others.
The network items are organized by the virtual switch to which they are attached. The virtual switch name is listed under the DNAME column.
Network traffic on the hypervisor's iSCSI initiator will show up on the VMkernel network row which will contain the name "vmk-tcpip-A.B.C.D", where A.B.C.D is the VMkernel IP address.
Network traffic on an iSCSI initiators that were configured in the guest will show up on the vNIC displayed using the VM's name on the network panel.
Total throughput for each item can be observed by summing the total transmitted data (MbTX/s) and received data (MbRX/s) for each item. As the physical hardware becomes saturated transmitted and received packets will start to be dropped (%DRPTX and %DRPRX, respectively) which, depending on protocol, may result in a retransmission at a later time.

Evaluate the Data

Does the physical NIC's reported speed and duplex setting match the expectation of the hardware? Hardware connectivity issues may result in a NIC autonegotiating to a lower speed or half duplex mode.
Is there a significant load on the appropriate network items? For instance, is a network-intensive load in a guest actually generating the network activity on its vNIC that is expected? Are storage-intensive loads generating traffic on the vNIC or vmkNIC when the hypervisor or guest initiators are used?
Verify that the network traffic is flowing on appropriate NICs. A typical ESX host may have network traffic generated by VMs, network traffic from iSCSI protocol, VMotion related network traffic and service console associated network activity. It is recommended to have to separate NICs to handle these different network packets.
During periods of saturation, is the total throughput (MbTX/s summed with MbRX/s) matching expectations? Either the guest or the other end of the communication link may be throttling the performance.
Are packets being dropped? When overworked the hardware will refuse packets which get reported as dropped transmitted (%DRPTX) and received (%DRPRX) packets.

Correct the System

Make sure that the hardware is configured to run at its maximum capability. This means verifying that 1 Gb NICs are not autonegotiating down to 100 Mb/s for having been connected to an older switch. Similarly, ensure that NICs are running in full duplex mode.
When network throughput seems lower than expected, apply traditional network diagnosis techniques to investigate every link in the connection. Low throughput at the ESX Server is not necessarily due to server configuration.
Verify that VMware Tools is installed on the guests and TSO, Jumbo Frames, and 10 Gb Ethernet are enabled, where possible.
Bond multiple physical NICs to virtual switches with high utilization.
Provide separate virtual switches their own physical NICs and separate network-intensive VMs on their own vSwitches.
If VMs running on the same ESX Server communicate with each other, connect them to a dedicated virtual switch so that all network transfers occur in memory and not packets are shipped over the wire.

VMFS and RDM Considerations

ESX Server supports the mapping of physical LUNs to virtual machines via a method called raw device mapping (RDM). RDM eliminates VMFS from the stack which is incorrectly believed to be a source of performance problems. Removing VMFS reduces the total number of addressable LUNs, eliminates the ability to perform storage migrations (storage VMotion), and greatly increases the effort required for simplified maintenance activities provided by site recovery manager. And the performance benefits derived from the removal of VMFS are negligible.

See the performance characteristics of VMFS and RDM whitepaper for more information on this subject.

Resources

The top-level performance analysis page: Performance Monitoring and Analysis

VirtualCenter performance counters: Understanding VirtualCenter Performance Statistics

esxtop performance counters: esxtop Performance Counters

Storage Performance Analysis and Monitoring

Tue, 27 May 2008 19:02:23 GMT

This document is a living, wiki version of the performance analysis methods whitepaper . That document will ultimately be replaced with this one.

Introduction

Storage often bounds the performance of enterprise workloads. More so than CPU or memory performance investigation, traditional means of analysis continue to be sound for storage performance in virtual deployments. This section will introduce the tools for identifying heavily-used resources and VMs that have high demands of their storage system. Traditional correction methods will then apply.

iSCSI storage using software initiators is not covered in this section. When accessed through the hypervisor's iSCSI initiator or an in-guest initiator traffic will show up on the VMkernel network or the VM's network stack. Check the Network section for more information.

Navigating esxtop

As before, esxtop is the best place to start when investigating potential performance issues. To view the disk adapter information in esxtop, hit the ‘d' key once it is running.

On ESX Server 3.5, the storage system can be displayed per VM (using ‘v') or per storage device (using ‘u'). But the same counters are displayed on each. Look at the following items:

For each of the three storage views:
- On the adapter view (‘d'), each physical HBA is displayed on a row of its own with the appropriate adapter name. This short name may be checked against the more descriptive data provided through the Virtual Infrastructure Client to identify the hardware type.
- On ESX Server 3.5's VM disk view (‘v'), each row represents a group of worlds on the ESX Server. Each VM will have its own row and rows will be displayed for the console, system, and other less-important (from a storage perspective) worlds. The groups' IDs (GID) match those on the CPU screen and can be expanded by pressing ‘e'.
- On ESX Server 3.5's disk device view (‘u'), each device is displayed on its own row.
As with the other system screens, the disk displays can have groups expanded for more detailed information:
- The HBAs listed on the adapter display can be expanded with the ‘E' key to show worlds that are using those HBAs. By finding a VM's world ID the activity due to that world can be seen on the expanded line with the matching world ID (WID) column.
- The worlds for each VM can be displayed by expanding the VM row on the VM disk view with the ‘e' key.
- The disk devices on the device display can be expanded to show usage by each world on the host.

Relevant Counters

Type	VirtualCenter	esxtop	Details
Queued Disk Commands	disk.queueLatency.average	QUED	Queued commands are queued in the kernel queue. They are awaiting an open slot in the device driver queue. A large number of queued commands means a heavily loaded storage system. See Storage Queues and Performance for information on queues.
Queue Usage	Not available	%USD	This counter tracks the percentage of the device driver queue that is in use. See Storage Queues and Performance for info on this queue.
Command Rate	disk.commands.summation	ACTV	VirtualCenter reports the number of commands that have been issued in the previous sample period. esxtop provides a live look at the number of commands that are being processed at any one time. Consider these counters a snapshot of activity. But don't consider any number here "too much" until large queues start developing.
HBA Load	Not available	LOAD	In esxtop the LOAD counter tracks how full the device queues are. Once LOAD exceeds one, commands will start to queue in the kernel. See Storage Queues and Performance for information on these queues.
Storage Device Latency	disk.deviceReadLatency disk.deviceWriteLatency	DAVG/cmd	These counters track the latencies of the physical storage hardware. This includes everything from the HBA to the platter.
Kernel Latency	disk.kernelReadLatency disk.kernelWriteLatency	KAVG/cmd	These counters track the latencies due to the kernel's command processing.
Total Storage Latency	Not available	GAVG/cmd	This is the latency that the guest sees to the storage. It is the um of the DAVG and KAVG stats.
Aborts	disk.commandsAborted.summation	ABRTS/s	These counters track SCSI aborts. Aborts generally occur because the array is taking far too long to respond to commands.

Evaluate the Data

It is important to have a solid understanding of the storage architecture and equipment before attempting to analyze performance data. Consider the following questions:

Is the host or any of the guests swapping? The guest's swap activity must be checked with traditional OS tools and the host can be checked with SWR/s and SWW/s counters detailed in Memory Performance Analysis and Monitoring.
Are commands being aborted? This is a certain sign that the storage hardware is overloaded and unable to handle the requests in a manner in line with the host's expectations. Corrective action could include hardware upgrades, storage redesign (increasing spindles on the RAID), or guest redesign.
Is there a large queue? While less dangerous than abortions, queued commands are similarly a sign that hardware upgrades or storage system redesign is necessary.
Is the array responding at expected rates? Storage vendors will provide latency statistics for their hardware that can be checked against the latency statistics in esxtop. When the latency numbers are high, the hardware could be overworked by too many servers. As examples, 2-5 ms latencies are usually a sign of a healthy storage system reading data on the array cache, 5-12 ms latencies reflecting a healthy storage architecture were data is being randomly read across the disk, and 15 ms latencies or greater possibly representing an over-utilized or misbehaving array.

Identifying a Slow Array

Its worth pausing at this moment to point out that 95% of all storage performance problems are not fixed in ESX. Believe me, I (Scott) have been called into a dozen performance escalations where poor storage performance was blamed on the hypervisor and not a single one was being caused by ESX. If you're seeing high latencies in VirtualCenter or esxtop to the storage device, its worth treating this problem as an array configuration issue. Check ESX's logs for obvious storage errors, check array stats, and make sure that there are no fabric configuration problems.

At the point of high storage latencies you shouldn't be using complex benchmarks to reproduce and solve this problem. Go with Iometer and make certain you're doing an apples-to-apples comparison against a physical system (ideally dual-booted from the ESX server under test) to make sure of what your expected, non-virtual results are. Check Storage System Performance Analysis with Iometer for information on using Iometer for problems like this.

Correct the System

Corrections for these problems can include the following:

Reduce the guests and host's need for storage.
1. Some applications such as databases can utilize system memory to cache data and avoid disk access. Check in the VMs to see if they may benefit from increased caches and provide more memory to the VM if resources permit. This may reduce the burden on the storage system.
2. Eliminate all possible swapping to reduce the burden on the storage system. First verify that the VMs have the memory they need by checking swap statistics in the guest. Provide memory if resources permit. Next, as described in the "Memory" section of this paper, eliminate host swapping.
Configure the HBAs and RAID controllers for optimal use. It may be worth reading Storage Queues and Performance for information on how disk queueing works.
1. Increase the number of outstanding disk requests for the VM by adjusting the "Disk.SchedNumReqOutstanding" parameter. For detailed instructions, check the "Equalizing Disk Access Between Virtual Machines " section in the "Fibre Channel SAN Configuration Guide". This step and the following one must both be applied for either to work.
2. Increase the queue depths for HBAs. Check the section "Setting Maximum Queue Depth for HBAs " in the "Fibre Channel SAN Configuration Guide" for detailed instructions. Note that you have to set two variables to correctly change queue depths. This step and the previous one must both be applied for either to work.
3. Make sure the appropriate caching is enabled for the disk controllers. You will need to the vendor provided tools to verify this.
If latencies are high, inspect array performance using the vendor's array tools. When too many servers simultaneously access common elements on an array the disks may have trouble keeping up. Consider array-side improvements to increase throughput.
Balance load across the physical resources that are available.
1. Spread heavily used storage across LUNs being accessed by different adapters. The presence of separate queues for each adapter can yield some efficiency improvements.
2. Use multi-pathing or multiple links in case the combined disk I/O is higher than a single HBA capacity.
3. Using VMotion, migrate IO-intensive VMs across different ESX Servers, if possible.
Upgrade hardware, if possible. Storage system performance often bottlenecks storage-intensive applications but for the very highest storage workloads (many tens of thousands of IOs per second) CPU upgrades at the ESX Server will increase the host's ability to handle IO.

Resources

Top-level performance analysis page: Performance Monitoring and Analysis

VirtualCenter performance counters: Understanding VirtualCenter Performance Statistics

esxtop performance counters: esxtop Performance Counters

Fibre Channel SAN Configuration Guide

Storage Queues and Performance

Memory Performance Analysis and Monitoring

Fri, 23 May 2008 23:37:14 GMT

Introduction

This document is a living, up-to-date version of the performance analysis methods whitepaper.

Host memory utilization represents the entirety of memory usage due to the VM and all tasks required by ESX Server to manage and provide control of the VMs. Using ESX Server's monitoring capabilities there is no visibility into improper usage of configuration of memory within the guest. Continue to use traditional monitoring tools in the guest to identify memory-hungry applications or shortages that lead to in-guest swapping.

Navigating esxtop

As before, bring up esxtop to inspect system specifics. Hitting the ‘m' key will display the memory counters.

Once running, the following can be observed from the esxtop report:

The header data contains host data that impacts all VMs running on the host. The physical memory row (PMEM) contains the total RAM installed on the system, the amount used by the console operating system (COS), the memory used by the kernel (VMK), and other statistics.
The next few rows contains host-level memory statistics for various ESX subsystems:
- VMKMEM: shows memory statistics for the ESX Server VMkernel
- COSMEM: displays the memory statistics as reported by the ESX Server service console.
- PSHARE: displays the ESX Server page-sharing statistics.
- SWAP: displays the ESX Server swap usage statistics.
- MEMCTL: displays the memory balloon driver statistics.

Relevant Counters

Type	VirtualCenter	esxtop	Details
Total memory size		MEMSZ	The is the amount of memory that the VM has been sized to. The VM will never get more than this but most of the time will be using far less than this amount due to sharing, ballooning, and swapping.
Memory target		SZTGT	The amount of memory that the kernel would like to provide to the VM. This number is calculated by on the guest's memory usage. When memory is over-committed, it may not equal the amount of memory that is actually provided due to ballooning and swapping.
Granted memory	mem.granted.average		The amount of memory that has been provided to the VM. Memory is not granted to the VM until it has been touched once. In the case of Linux, which does not zero out pages upon boot, a 4G VM will only be granted the small portion (100M or so) needed to run the OS until the OS or applications start to access more.
Touched memory		TCHD	The amount of memory (in MB) that has been "touched" (read from or written to) in the past X minutes.
Consumed memory	mem.consumed.average		The amount of machine memory allocated to the VM. For instance, a Linux VM might have been sized to 4G. Half of the pages may not yet have been used by the OS. Perhaps 1G of this remaining 2G can be shared. That leaves a consumed memory of only 1G.
Shared memory	mem.shared.average		Shared memory represents the entire pool of shareable memory. For instance, if two VMs each have 500M of identical memory, the shared memory is 1G.
Shared common memory	mem.sharedcommon.average		Shared common memory represents the footprint in machine memory as a result of memory sharing. For instance, if two VMs each have 500M of identical memory, the shared common memory is 500M.
Active memory	mem.active.average	%ACTV, %ACTVS, %ACTVF	The amount of memory (as a percentage of the entire host's memory) that has been used by the VM in the past sample period. %ACTVS and %ACTVF are slow and fast counters showing recent and long-term averages.
Ballon driver usage	mem.vmmemctl.average	MCTLSZ	The amount of memory claimed by the balloon driver for us in other VMs.
Swap rate		SWW/s SWR/s	The rates at which memory is swapped out (written) or in (read).
Swap Totals	mem.swapout.average, mem.swapin.average		These are cumulative amounts of swapping that has occurred since the VM was powered on. It's important to check if swapin and swapout are increasing, rather than just seeing if they are nonzero. Because if they are non-zero, it could be the result of swapping in the past, and not swapping at the present time.
NUMA migrations		NMIG	The number of NUMA migrations that have occurred since the VM's creation.
NUMA memory		NLMEM, NRMEM	The amount of the VM's memory that is on the local and remote NUMA nodes.
Overhead	mem.overhead.average	OVHD	The amount of memory required by the VMkernel to maintain and execute the VM.

Evaluate the Data

Memory analysis on an ESX Server means not just investigation of server-side statistics but also a solid understanding of the application that is running in the VM. When memory is short on the host, ballooning and swapping may be visible in esxtop, with swapping having a great impact on performance. When memory is short within the VM the guest will swap.

How much memory are the VMs actually using? While they may have been allocated large amounts of memory, its likely that the OS and applications are only using a small percentage of what the VM was assigned. Check the active and touched memory counters for accurate numbers on guest memory usage.
Is memory short in the host? Swapping (SWW/s and SWR/s) is a certain sign of this problem. Heavy use of the balloon driver may also suggest this but ballooning has a very slight impact to guest performance.
Can memory deficiencies be addressed through VM resizing? Checking memory usage of critical apps within the VMs can help inform decisions to decrease the amount of RAM provided to those VMs. Some operating systems will expand to utilize all available memory at little or no value to the application. Reducing the memory space and correcting over-sized caches frees up memory for other VMs.
Is the collection of all VMs' active memory (TCHD or %ACTV) sustaining at an amount that exceeds the total available memory? If so, then either more memory must be added to the host or VMs must be migrated to another DRS cluster.
Are the guests swapping? If the VM has been sized with too little memory then the guest OS will swap inside the VM. This will appear to ESX Server as any other disk activity but should be investigated and solved with traditional OS analysis tools.
Can NUMA migrations (NMIG) be seen on the system? NMIG reports total migrations since the VM has been powered on. If this number continues to climb then the VM is being migrated from node to node which most certainly degrades performance.
Does the amount of memory located on a remote NUMA node (NRMEM) remain at a non-zero number? This may be a sign that the VM has been sized to exceed the memory of a single NUMA node. If the VM is using more memory than fits on a single node, some of its memory is certain to be located on a remote node. Remote memory access is quite slow relative to local memory access.

Correct the System

The prescriptive advice for memory shortages is fairly simple: use less memory or buy more. The following recommendations are variations on this theme:

Verify that VMware Tools has been installed on every VM on the system and that the memory balloon driver has not been disabled. (The balloon driver is always on by default and disabled manually through text-based advanced configuration in extremely rare cases.) When provide the ability to balloon memory within the guests, ESX Server is able to take memory from VMs that are not using it and make it available to those that do need it.
Provide more memory to the DRS cluster. As total resources go up, VirtualCenter will balance VMs across the cluster so VMs that need the memory are able to get it.
Set memory reservations to minimally provide the amount of memory required of the OS and critical applications. This will allow for sustained, fast access for critical code and provide hints to VirtualCenter for optimal VM positioning across the DRS cluster.
Make sure the amount of memory used by the VMkernel to maintain the VMs is acceptable. This value, reported for each VM with the overhead counter (OVHD), is dependent on the memory size of the VM, the number of vCPUs provided to it, and whether or not it is executing a 64-bit OS. Fewer VMs on the host, fewer aggregate vCPUs, and lower precision OSes (32-bit as opposed to 64-bit) will lower this number. Reducing any of these in the cluster will free up resources for every VM in the cluster.
Size VMs on NUMA systems to guarantee that each VM's memory will fit on a single node. This means either decreasing the memory allocated to a VM or increasing the node memory size.
Size guests appropriately according to their needs. For example:
1. Depending on the access pattern of the data, databases may not benefit from the last doubling of cache size. Experiment with smaller cache sizes and see if performance drops. If not, decrease the VM's available memory so it can be used by other VMs.
2. Check the guest OS's statistics for in-guest swapping. Provide memory as its needed and pay attention to esxtop statistics to see if the additional memory provided generates a new bottleneck in the host.

Understanding Page Sharing

One cannot fully optimize an ESX Server's memory without understanding the performance implications of page sharing. VMware's page sharing algorithm was presented at EMC World 2008 as resulting in a 2% increase in CPU load. But the benefits of page sharing have been demonstrated to provide overcommitment of memory safely to 2X and beyond.

The value of page sharing can be seen int the following counters:

esxtop	VirtualCenter	Description
SHRD	memory.shared	The amount of memory in the VM that is sharable.
SHRDSVD	No equivalent.	The amount of memory saved due to page sharing.
No equivalent.	memory.sharedcommon	The size of the memory after redundant pages have been removed.

Note that missing counters can be calculated using the other two. Shared memory minus shared common memory equals shared savings.

References

The top-level Performance Monitoring and Analysis paper.

The esxtop Performance Counters index.

The Understanding VirtualCenter Performance Statistics page.

CPU Performance Analysis and Monitoring

Fri, 23 May 2008 19:05:48 GMT

Introduction

CPU load is generated by the guest and its applications as well ESX Server as it provides a virtual interface to the hardware. While the work performed by the host does result in some increase in load, the great majority of processing is due to the applications in the VM. A solid understanding of the workload profile regardless of the virtual environment can assist CPU analysis.

Check Utilization

Invoke esxtop. By default, it should show CPU utilization but pressing ‘c' will ensure this data is being displayed. The following figure shows example data produced on a test system.

Observe the following:

The PCPU(%) line in the header shows utilization for the processor(s) by core and in total. The comma-delimited data first displayed shows core utilization followed by "used total" which averages utilization of all cores.
The LCPU(%) line shows the percentage of CPU utilization per logical CPU. The percentages for the logical CPUs belonging to a package add up to 100 percent. This line appears only if hyperthreading is present and enabled.
The CCPU(%) line shows the percentages of total CPU time as reported by the ESX Server service console. Use of any third party software, such as management agents and backup agents, inside the service console, may result in high CCPU(%) number.
There is an idle world running whose %USED entry displays the amount of CPU cycles that remain unused. If the idle world is reported at less than 100% utilization then only a fraction of one physical core remains for additional work. As this number can max out at many hundreds of percentages (100% for each core) small numbers here represent heavily loaded systems.
Check the utilization (%USED) of the interesting VMs. The VMs are reported here with the names specified at their time of creation. Like the idle row, utilization for each VM can exceed 100%. A VM that was provided two vCPUs, as an example, can max out at 200% CPU utilization.
Expand the group data for the VM that is most interesting. This is done by hitting ‘e' and then entering the group ID number (GID) for the VM. The figure below contains a CPU-expanded version for GID "30" in the previous figure. Once expanded, esxtop will expand rows and provide counter data for every world in the group. This includes:
- vmmX: For each vCPU provided to the VM, a virtual machine monitor (VMM) world is displayed. This world will perform the majority of the work required to execute and virtualize the guest code (OS, application, and hypervisor).
- vcpu-X: A vcpu-X world is created to assist the VMM world for each vCPU. Primarily this work revolves around the virtualization of the IO devices.
- mks: Mouse, keyboard, and screen interrupt servicing.
- vmware-vmx: The VMX worlds assist in maintenance and communications with other worlds and should not represent a material portion of the group utilization.

Evaluate the Data and Correct the System

The general flow for evaluation starts by considering the system's load. Is the system overloaded with too many VMs? Is the guest using all of its vCPUs and simply requires more or faster processors? Are all guests waiting for IO? For example:

Check the PCPU(%) line to see if all cores' utilization is near 100%. In this case the system is saturated. If multiple VMs are competing for the CPUs, try to reduce the VMs on the system or find other means of decreasing the load on the system. See "CPU Saturation of Host" below.
See if the PCPU(%) line shows an unequal load across processor cores with some at saturation and some remaining near idle. This would indicate applications within the VM utilizing all of the cores provided to them. Increase its vCPU count, if possible, and verify that the guest is making use of the additional cores. If the application supports horizontal scalability, you may run multiple VMs to use the additional cores. See "CPU Saturation of VM" below.
If all CPUs remain underutilized, either the application in the VM is misconfigured or the VM is waiting for IO operations to complete. See "Low CPU Utilization" below.

CPU Saturation of Host

As stated above, both the PCPU(%) and %USED counters can be used to identify systems hosts that are using all physical CPUs. It is possible, however, for the VMs on the system to be utilized nearly all of the processor cycles without actually requesting more that is available. This near-saturation case is the sign of a heavily loaded system.

A better sign of over-utilization on a host is ready time (%RDY). When any world's ready time starts to climb, that world is spending the reported percentage of its time waiting for some CPU to become available for work. Ready time above 10% is worth investigation and may be a sign of an over-utilized host. For a more detailed discussion on ready time, see Ready Time.

Host saturation is a clear sign that too much work has been loaded onto a single server. This is usually due to overly aggressive consolidation ratios. Overcommiting CPU resources in this case will only worsen the performance. Consider the following remedies:

Verify that VMware Tools has been installed on every VM on the system. In addition to many other benefits, VMware Tools provides a network driver (vmxnet) without which guest networking will be unnecessarily inefficient.
Verify that the all systems in the DRS cluster are carrying load when the server of interest is overloaded. If they aren't, increasing aggression of DRS algorithm and check VM reservations against other hosts in the cluster to ensure migrations will happen. Lastly, increase the number of servers in the DRS cluster so VMs from this server can be migrated to servers with available resources.
Increase the CPU resources available to the VMs by increasing or improving CPUs or cores on some of the systems in the DRS cluster.
Set CPU reservations for the VMs that most need the processing power to guarantee that they get the CPU cycles they need.
Ensure the newest version of ESX Server is being used. The newer versions of ESX Server provide better efficiency and CPU-saving features such as TCP segmentation offload (TSO), large memory pages, and jumbo frames.
Reduce the CPU resource footprint of running VMs. As examples:
1. Decrease disk and or network activity for applications that cache data by increasing the amount of memory provided to the VM. This may lower IO and reduce ESX Server's responsibility to virtualize the hardware.
2. Assist CPU by replacing software I/O with dedicated hardware (such as iSCSI HBAs or TCP segmentation offload NICs).
3. Reduce vCPU count for guests to only the number required to execute the workload. For instance, a single-threaded application in a 4-way guest will only benefit from a single vCPU. But the hypervisor's maintenance of the three idle vCPUs takes CPU cycles that could be used for other work.
4. For VMs created using P2V conversion, analyze the VM resources as well as the applications running inside the VM. Stop the unnecessary services that may running inside the P2V'ed VM. Also reduce the number of vCPUs and memory count to only the number required to execute the workload.

The easiest general comment for addressing CPU bottlenecks given correctly-configured VMs is to address processing power at the cluster level. If VirtualCenter reports fully utilized CPUs for all hosts in the cluster, there is little possibility of avoiding a need to increase cluster resources or decrease VM count.

One last nuance of virtual system tuning, mentioned in item 6c above, is the correct balancing of virtual CPU count. Few applications fully utilize two or more vCPUs and many VMs are often committed to a special purpose with a single application. The guest OS and the hypervisor must expend CPU cycles managing multiple vCPUs. If the applications are not using them, the system efficiency as a whole will improve by reducing vCPU count for VMs.

CPU Saturation of VM

Like host CPU saturation, VM CPU saturation can be seen when the %USED for a VM is high. Unlike host CPU saturation, the idle world may report a large amount of free computational resources and the VM's ready time (%RDY) may remain low. This behavior can be seen when a single VM utilizes all of the processors allocated to it but additional CPUs remain unused on the host. The VM's utilization of all of its vCPUs can be confirmed by expanding the VM's world on the CPU screen. Once this has been confirmed, the following options are available:

Verify that VMware Tools has been installed on every VM on the system. In addition to many other benefits, VMware Tools provides a network driver (vmxnet) without which guest networking will be unnecessarily inefficient.
If possible, increase the number of vCPUs provided to the VM. As the application in the guest is successfully using all of its vCPUs, it may continue to scale as the vCPU count is increased. Pay attention to the vmmX world for each vCPU after increasing vCPU count to verify that the VM is making use out of its newly provided resources. As detailed in item 6c in the "CPU Saturation of Host" section, the addition of vCPUs imposes an overhead on the host whether they are being used or not. So carefully assess the guest's needs to avoid unneeded vCPU count increases.
If possible, you can power on multiple VMs running the same application. This will depend upon whether how well an application supports horizontal scalable configuration. It is possible that an application may perform better when running as multiple single vCPU Vms, rather a single SMP VM.
Utilize faster processors. As processor performance is continually increasing the option of upgrading processors or migrating the VM to systems with newer processors can provide more total throughput to the VM.
Set CPU reservations for the VMs that most need the processing power to guarantee that they get the CPU cycles they need.
Decrease the work as a result of running the VM. As examples:
1. Decrease disk and or network activity for applications that cache data by increasing the amount of memory provided to the VM. This may lower IO and reduce ESX Server's responsibility to virtualize the hardware.
2. Assist CPU by replacing software I/O with dedicated hardware (such as iSCSI HBAs or TCP segmentation offload NICs).

Low CPU Utilization

Assuming performance problems have been confirmed, low CPU utilization is usually a sign of inefficiently designed datacenter architecture. The design could be flawed in an individual VM or in the connectivity between various components. The Performance Monitoring and Analysis will walk through investigation of system-level components such as memory and then system-wide components such as network and storage.

References

esxtop Performance Counters

Understanding VirtualCenter Performance Statistics

Ubuntu

Fri, 16 May 2008 22:13:21 GMT

Ubuntu supports paravirtualization!

I'll have more to say about Ubuntu later.

Windows Server 2003

Fri, 16 May 2008 22:04:43 GMT

Please be patient while we build out this content. If you have ideas for additions, mail me!

Short list:

Always install Service Pack 2. Microsoft changed the interaction with the APIC to improve efficiency on virtual platforms.
Read up on idle loop behavior for Service Pack 1 .

Red Hat Enterprise Linux

Fri, 16 May 2008 21:55:13 GMT

Page under construction.

Quick notes:

RHEL5 always uses a Linux Timer Rate timer rate of 1000 Hz. This can be decreased in RHEL5.1 for greater efficiency of RHEL SMP guests.
RedHat has not enabled VMI in their kernels, so out-of-the-box paravirtualization is not possible. But custom kernels can easily be built to take advantage of this feature. See VMware KB article #1003644.

Best Practices for Performance

Fri, 16 May 2008 21:35:33 GMT

VMware and our partners have published a wide variety of white papers on best practices for installation and configuration of enterprise applications on VI3. We'll collect additional material on this page as we build this content out. Keep checking back!

Operating Systems

Windows Server 2003
SUSE Linux Enterprise Server
Red Hat Enterprise Linux
Ubuntu

Applications

Understanding Performance

Fri, 16 May 2008 21:30:03 GMT

The following documents will explain some of the principles for virtual system performance. Please check back as we grow the number of articles here with time.

ESX and Guest Operating Systems

CPU and Scheduling

Memory

Large Memory Pages

Network

Advanced Networking Performance Options

Storage

esxtop Performance Counters

Fri, 16 May 2008 18:59:39 GMT

Introduction

This article contains a list of some of the performance counters provided by esxtop. This is far from exhaustive, as this list was created to answer the question: "which are the most important esxtop counters?" Recently VMware has published an exhaustive list of esxtop information on Interpreting esxtop Statistics. Check that out for more information.

CPU Counters

Counter	Description
%RDY	The percentage of time that the world or group is waiting a processor to be available to execute its workload.
%USED	The percentage of CPU that is used by that world or group.
GID	Group ID
NWLD	The number of worlds in the group. When this number is greater than one, the row can be expanded to get information on each world.

Memory Counters

Counter	Description
%ACTV	Instantaneous view of the percentage of memory pages that have been used by the VM in the previous seconds. Unlike TCHD which counts pages by following working sets, %ACTV is a more frequently updated number that is based on a sample of the entire memory pool.
%ACTVS	Slow moving average of the %ACTV counter.
%ACTVF	Fast moving average of the %ACTV counter.
MCTL?	Set to "Y" when the balloon driver is active in the guest and "N" when not.
MCTLSZ	This counter reports the amount of memory that the balloon driver is currently holding for use by other VMs.
MEMSZ	The amount of memory (in MB) allocated to the VM at the time of its creation.
NHN	The NUMA home node. This is the node on which the VM is booted. Migrations that have occurred since the VM started running would result in this VM running on another node(s).
NMIG	The number of NUMA node migrations since the VM was booted. ESX Server's scheduler should avoid NUMA migrations so if this number continues to climb during normal operations some tuning of the VMs may be required.
NRMEM	The amount of memory that exists on a remote NUMA node.
NLMEM	The amount of memory that exists on the local NUMA node.
N%L	The percentage of the VM's memory that exists on the local NUMA node. N%L = NLMEM / (NRMEM+NLMEM)
OVHD	The amount of memory used by the VMkernel to maintain and execute the VM.
SHRD	The amount of the VM's memory that is shared with other VMs.
SHRDSVD	The amount of memory that was saved due to page sharing. This number may be less than or equal to SHRD. As one VM must always claim the single copy of a shared page, one VM with a shared page will not be able to claim savings.
SWR/s	The rate at which memory is being swapped in from disk. High swap rates indicate a need for more memory in the cluster.
SWW/s	The rate at which memory is being swapped out to disk. High swap rates indicate a need for more memory in the cluster.
TCHD	The amount of memory (in MB) that has been touched (recently used) by the VM. In this case "recently" means within a minute or two.

Storage Counters

Counter	Description
ABRTS/s	The rate at which disk operations are being aborted. Abort commands are issued by the guest when the storage system has not responded within an acceptable amount of time (as defined by the guest OS or application.)
ACTV	The number of IO operations that are currently active. This represents operations for which the host is processing and can serve as a snapshot view of storage activity. When this number hovers near zero, the storage system isn't being used. If is sustains non-zero numbers, the a constant interaction with the strorage is occurring.
DAVG/cmd	The average amount of time it takes a device (HBA, array, and everything in between) to service a single request.
GAVG/cmd	The total latency seen from the VM when performing an IO operation. GAVG = DAVG+KAVG.
KAVG/cmd	The average amount of time it takes ESX Server's VMkernel to service a disk operation. Since this number represents time spent by the CPU to manage IO and processors are orders of magnitude faster than disks, it should be much, much less DAVG.
QUED	The number of IO operations that require processing but have not yet be addressed. Commands are queued and awaiting management by the kernel when the driver's active buffer is full (see ACTV). Occasionally a queue will form and result in a small, non-zero QUED number but any significant (double-digit) average of queued commands means the storage hardware is unable to keep up with the host's needs.
READS/s	The number of disk reads per second. READS/s + WRITES/s = IOPS.
WRITES/s	The number of disk writes per second. READS/s + WRITES/s = IOPS.

Network Counters

Counter	Description
%DRPRX	The percentage of packets that were dropped that was supposed to be received.
%DRPTX	The percentage of packets that were dropped for which transmission was attempted.
MbRX/s	The megabits per second that are received at the network item.
MbTX/s	The megabits per second that are transmitted from the network item.

Understanding VirtualCenter Performance Statistics

Thu, 15 May 2008 19:15:58 GMT

Introduction

VirtualCenter (VC) is the entry point for virtual platform management but is less frequently used for performance analysis than esxtop. On the surface, VC is insufficient for performance analysis. But this is not necessarily the case. The VirtualCenter performance counter collection is reduced by default to minimize the data maintained by VC's database. The performance counters maintained by VC can be modified and detailed analysis can be performed based on those counters. This document will provide details necessary for understanding and enabling VC's performance monitoring capabilities.

Refer to the Performance Monitoring and Analysis for information on using these counters.

VirtualCenter Statistic Archival

Our stats infrastructure has a lot of counters but our documentation has traditionally been quite thin in terms of descriptions. I got so sick of asking what stats are available at what stats level that I decided to start this page. Obviously it needs to be made more readable, but hopefully it is a start.

Remember that stats in VC are generally organized into 2 archival categories:

Not archived: these are the "real-time" (past-hour) stats, which are refreshed every 20 seconds, and are displayed for the past hour in the VI client. These stats are not stored in the database.
Archived stats. These stats are aggregations (rollups) of the real-time stats. They are aggregated at different sampling intervals and stored in the database. We follow the MRTG standard.
- Past day: past day stats take the real-time stats and roll them up so that there is 1 data point for every 5 minutes. Thus, there are 12 data points per hour and 288 per day.
- Past week: past week stats take the past day stats and roll them up so that there is 1 data point for every 30 minutes. Thus, there are 48 data points per day and 336 per week.
- Past month: past month stats take the past week stats and roll them up so there is 1 data point per 2 hours. Thus, there are 12 data points per day and 360 per month (30-day month).
- Past year: past year stats take the past month stats and roll them up so there is 1 data point per day. Thus, there are 365 data points per year.

The basic flow is this: an ESX host stores statistics at 20s granularity for a period of 1 hour. Therefore, using the Host Client one can view the stats for a host/VM for the past-hour, or one can view those stats using the VI client attached to VirtualCenter. ESX will also aggregate the statistics into past-day statistics and store them for up to 1 day. These past-day statistics are sent to VC periodically and then stored in the database. The database is responsible for periodically taking these past-day stats and rolling them up into 30-minute weekly stats, and then doing the same for converting the weekly stats to monthly stats, etc. Because past-day, past-week, and past-month stats are stored in the database, I call them "archived" stats.

VirtualCenter Statistics Level

Statistics level is a means of organizing statistics for archiving purposes. Its worth noting that only stats levels one and two are useful for deployment performance monitoring and analysis. Levels three and four provide granularity and visibility that is useful only for developers.

The concept of "stats level" applies only to the archived stats: we only store a stat in the database if we are at the appropriate stats level for that particular statistic. Non-archived stats are unaffected by stats level. In other words, every metric listed below is collected at 20s granularity and stored on the ESX host for 1 hour. However, unless VC is set to the stats level appropriate to that statistic, we will not store the data in the database or rollup the stat into a past-day stat on the ESX host. You can specify the stats level independently for each of the archiving interval. In other words, you might want to store level 4 stats for up to 1 day, but level 3 stats for 1 week.

In practice, we use stats level to vary the level of detail for statistics that are archived. At stats level 1, we have pretty coarse-grained stats, while stats level 4 contains very detailed statistics, and also includes statistics for various instances (e.g., for each NIC of a VM).

There are 3 important calls that I often use for stats (please refer to the SDK documentation for more information):

QueryStatsByLevel: this tells you what stats are available at what stats level. This is what I used to generate the tables below.
QueryAvailableMetrics: this tells you what stats are available for a given entity during a specified time period.
QueryPerf: this call takes a QuerySpec as an input and collects the stats for the specified entity over the specified time interval.

Let me give a concrete example of stats level.

Suppose I want to know the value of mem.consumed.maximum for a given VM. This is the maximum amount of machine memory allocated to a VM (including overhead memory) over a specified interval. As shown below, this is a "level 4" statistic. This means that if I've set the stats level to 4 for past-day stats and then formulate a QuerySpec that asks for the value of this data 20 minutes ago at "past-day" granularity (i.e., at 5-minute granularity), then I will get a value. If the stats level is 2 for past-day (5-minute granularity) statistics, however, then such a query will not return a value, because it is level-4 stat and only level-2/level-1 stats are being stored at 5-minute granularity. In contrast, even if the stats level is 1, then if I formulate a QuerySpec with 20s (i.e., "real-time" or "past hour") as the interval of collection, I will get this value, because this data is stored for up to one hour at 20s granularity no matter what the stats level.

Update Interval

Understanding the update interval is a key component to understanding the performance statistics. The Virtual Infrastructure Client (VIC) displays live stats at a 20s update frequency. Archived stats are archived at their archive frequency. This is key to understanding the relative amounts of data presented by VC.

For instance, a ready time of 1,000ms in the VIC's live stats graph translates into 5% ready time (1,000 / 20,000.) The same amount of ready time in a five minute archival frequency would be 15,000 ms.

Counter Index

For a list of all counters, see the vCenter Performance Counters page.

Hyper-Threading on ESX Server

Fri, 09 May 2008 21:31:11 GMT

Introduction

Those of us at VMware that regularly engaged with our field or directly with customers are often asked how Hyper-Threading impacts the performance of a system. I've now been asked this question enough times to have my canned "It Depends" response on the tip of my tongue for every conference I present at. I'm going to use this document to elaborate on that point a bit and provide a little more detail.

What Is Hyper-Threading?

Hyper-Threading is a technology included by Intel first in their Netburst line of parts. Hyper-Threaded processors present their individual processing cores to the system as if they are two processing cores. To use Intel's parlance, that means that each physical core appears in the operating system as two logical cores. While the OS can distinguish between a system that has two logical cores (i.e. a single physical core with Hyper-Threading enabled) and two physical cores, applications cannot. It is up the the OS's scheduler to choose if it wishes to use logical cores in the same manner as physical cores.

Is It Supported In ESX Server?

Hyper-Threading (HT) has been supported in ESX Server since version 2. ESX Server's scheduler is aware of the presence of HT and treats logical cores differently from physical cores. Virtual CPUs (vCPUs) requesting resources are assigned first to physical cores until all physical cores are loaded. If there are additional vCPUs requesting CPU resources they will then be assigned to the additional logical cores. By this method HT has no impact on performance until more vCPUs are concurrently executing than there exist physical cores.

How Does It Perform on ESX Server?

Understanding HT performance on native systems is tricky enough. Try Googling "hyperthreading performance" and you'll discover a world of information on this feature. By faking the presence of another processing core, Hyper-Threading removes some SMP scheduling from the guest operating system which is then handled by the processor's thread scheduler. Since the processor can manage context switches between its threads much faster than the OS can, this means that often heavy parallelism in applications results in improved performance due to Hyper-Threading.

The exact gains due to HT even on native systems is dependent on the workload. The industry has cited numbers that range from 0% to 40% gain when HT is enabled on supported processors. For the most part, HT improves performance nominally. There are a few cases where performance can slow down, but these are exceptions rather than the norm.

The best generalizations we can provide about HT on ESX are:

Until you have more vCPUs requesting processing power than there are physical cores, HT cannot hurt and provides no value.
Once you have more vCPUs requesting CPU than physical cores on the system, HT usually provides small gains.
While very early versions ESX may have sub-optimally handled HT, since ESX Server 2.5.3 robust support of HT in the scheduler means that it should not hurt performance.

For More Information

Hyper-Threading configuration options for performance optimization were provided in ESX Server 3 and beyond. See the Resource Management Guide (page 123) for more information.

Using Perfmon for esxtop-based Performance Analysis

Fri, 09 May 2008 16:48:20 GMT

Introduction

By now everyone knows that guest performance metrics are not reliably in virtual machines (Guest-based Performance Measurement). Slight fluctuations in very small time measurements and a lack of knowledge of the hypervisor's activities produces misleading numbers. Windows' performance monitoring tool, Perfmon, suffers from the same problems. However, Perfmon remains highly valuable for virtual machine performance analysis.

The performance analysis methods document that was published early in 2008 provided tips on esxtop-based performance analysis. To record esxtop data, page 159 of the resource management guide describes running esxtop in batch mode. But on ESX Server 3.5, running esxtop in batch mode with all counters enabled results in an incredibly large CSV file that cannot easily be parsed. But Perfmon can help with this process.

esxtop was constructed so that its CSV-formatted batch output file can be readily consumed by Perfmon. This means that Perfrmon can be used for two key activities:

Quickly analyzing results.
Generating smaller CSV files of a subset of the data that can be more easily consumed by other analysis tools (such as Microsoft Excel.)

Quick Results Analysis

esxtop's batch output CSV file can be opened and viewed in Perfmon with the following steps:

Transfer the CSV file to a Windows sytem.
Launch Perfmon (Run: "perfmon".)
Right click on the graph and select "Properties..." from the drop-down menu.
Select the "Source" tab.
Select the "Log files:" radio button from the "Data source" section.
Click the "Add..." button.
Browse to and select the CSV file created by esxtop and click "OK".
Click the "Apply" button.
Optionally: reduce the range of time over which the data will be displayed by using the sliders under the "Time Range" button.
Click "OK".

Once the data has been loaded into Perfmon you may select ESX performance counters and display them using Perfmon's graphing system just like normal Windows performance counters. Refer to the Perfmon documentation for instructions on doing this.

Pruning CSV File Data

Because so many people use Microsoft Excel to analyze performance data but its row and column limitations are quickly exceeded when generating large esxtop batch files, a means of removing unneeded data will assist analysis. Once Perfrmon has been loaded with the esxtop data, it can be used to generate smaller CSV files that can be easily consumed by Excel. Follow these steps:

Start Perfmon and load the esxtop batch data as described above.
Have the Perfmon graph display the data of interest (that you'd like to import into Excel.)
On the graph, right click "Save Data As..."
In the popup box, select "CSV" as type and save the file.

This will save only the counters that were displayed in the graph. By iteratively selecting subsets of counters and saving off individual CSV files it becomes possible to quickly build performance graphs in Excel using esxtop batch output data.

Co-scheduling SMP VMs in VMware ESX Server

Fri, 02 May 2008 15:12:56 GMT

Background

VMware ESX Server efficiently manages a mix of uniprocessor and multiprocessor VMs, providing a rich set of controls for specifing both absolute and relative VM execution rates. For general information on cpu scheduling controls and other resource management topics, please see the official VMware Resource Management Guide.

For a multiprocessor VM (also known as an "SMP VM"), it is important to present the guest OS and applications executing within the VM with the illusion that they are running on a dedicated physical multiprocessor. ESX Server faithfully implements this illusion by supporting near-synchronous coscheduling of the virtual CPUs within a single multiprocessor VM.

The term "coscheduling" refers to a technique used in concurrent systems for scheduling related processes to run on different processors at the same time. This approach, alternately referred to as "gang scheduling", had historically been applied to running high-performance parallel applications, such as scientific computations. VMware ESX Server pioneered a form of coscheduling that is optimized for running SMP VMs efficiently.

Motivation

An operating system generally assumes that all of the processors it manages run at approximately the same rate. This is certainly true in non-virtualized environments, where the OS manages physical processor hardware. However, in a virtualized environment, the processors managed by a guest OS are actually virtual cpu abstractions scheduled by the hypervisor, which time-slices physical processors across multiple VMs.

At any particular point in time, each virtual cpu (VCPU) may be scheduled, descheduled, preempted, or blocked waiting for some event. Without coscheduling, the VCPUs associated with an SMP VM would be scheduled independently, breaking the guest's assumptions regarding uniform progress. We use the term "skew" to refer to the difference in execution rates between two or more VCPUs associated with an SMP VM.

Inter-VCPU skew violates the assumptions of guest software. Non-trivial skew can result in severe performance problems, and may even induce failures when the guest expects inter-VCPU operations to complete quickly. Let's first consider the performance implications of skew. Guest OS kernels typically use spin locks for interprocessor synchronization. If the VCPU currently holding a lock is descheduled, then the other VCPUs in the same VM will waste time busy-waiting until the lock is released. Similar performance problems can also occur in multi-threaded
user-mode applications, which may also synchronize using locks or barriers. Unequal VCPU progress will also confuse the guest OS cpu scheduler, which attempts to balance load across VCPUs.

An extreme form of this performance problem may also lead to correctness issues. For example, a guest kernel may perform inter-processor operations, such as TLB shootdowns, that are expected to complete quickly on physical hardware (e.g. several microseconds). The guest OS may timeout if it finds that such operations have not completed after an unreasonably long period of time (e.g. several milliseconds). Without coscheduling, we have observed this behavior in practice for several different guest operating systems, including Windows BSODs, and Linux kernel panics.

Strict coscheduling in ESX Server 2.x

VMware introduced support for running SMP VMs with the release of ESX Server 2 in 2003. ESX Server 2.x implemented coscheduling using an approach based on skew detection and enforcement.

The ESX scheduler maintains a fine-grained cumulative skew value for each VCPU within an SMP VM. A VCPU is considered to be making progress when it is running or idling. A VCPU's skew increases when it is not making progress while at least one of its sibling VCPUs is making progress. A VCPU is considered to be "skewed" if its cumulative skew value exceeds a configurable threshold, typically a few milliseconds.

Once any VCPU is skewed, all of its sibling VCPUs within the same SMP VM are forcibly descheduled ("co-stopped") to prevent additional skew. After a VM has been co-stopped, the next time any VCPU is scheduled, all of its sibling VCPUs must also be scheduled ("co-started"). This approach is called "strict" coscheduling, since all VCPUs must be scheduled simultaneously after skew has been detected.

In some situations, such as when the physical machine has few cores, and is running a mix of UP and SMP VMs, coscheduling may incur "fragmentation" overhead. For example, consider an ESX Server with two physical cores running one dual-VCPU VM and one single-VCPU VM. When the UP VM is running, the scheduler cannot use the remaining physical core to run just one of the SMP VM's two VCPUs. This effect is typically negligible in systems with larger numbers of cores (or with hyperthreading enabled), due to the increased flexibility available when mapping VCPUs to hardware execution contexts.

Note that a VCPU executing in the guest OS idle loop can be descheduled without affecting coscheduling, since the guest OS can't tell the difference. In other words, an idle VCPU does not accumulate skew, and is treated as if it were running for coscheduling purposes. This optimization ensures that idle guest VCPUs don't waste physical processor resources, which can instead be allocated to other VMs. For example, an ESX Server with two physical cores may be running one VCPU each from two different VMs, if their sibling VCPUs are idling, without incurring any coscheduling overhead. Similarly, in the fragmentation example above, if one of the SMP VM's VCPU is idling, then there will be no coscheduling fragmentation, since its sibling VCPU can be scheduled concurrently with the UP VM.

Relaxed coscheduling in ESX Server 3.x

The coscheduling algorithm employed by the ESX scheduler was significantly enhanced with the release of ESX Server 3 in 2006. The basic coscheduling approach is still based on skew detection and enforcement.

However, instead of requiring all VCPUs to be co-started, only those VCPUs that are skewed must be co-started. This ensures that when any VCPU is scheduled, all other VCPUs that are "behind" will also be scheduled, reducing skew. This approach is called "relaxed" coscheduling, since only a subset of a VM's VCPUs must be scheduled simultaneously after skew has been detected.

To be more precise, suppose an SMP VM consists of multiple VCPUs, including VCPUs A, B, and C. Suppose VCPU A is skewed, but VCPUs B and C are not skewed. Since VCPU A is skewed, VCPU B can be scheduled to run only if VCPU A is also co-started. This ensures that the skew between A and B will be reduced. But note that VCPU C need not be co-started to run VCPU B. As an optimization, the ESX scheduler will still try to co-start VCPU C opportunistically, but will not require this as a precondition for running VCPU B.

Relaxed coscheduling significantly reduces the possibility of coscheduling fragmentation, improving overall processor utilization.

Conclusions

ESX Server employs sophisticated cpu scheduling algorithms that enforce rate-based quality-of-service for both uniprocessor and multiprocessor VMs. For multiprocessor VMs, coscheduling techniques ensure that virtual CPUs make uniform progress, faithfully implementing the illusion that the VM is running on dedicated multiprocessor hardware, ensuring efficient execution of guest software. Optimizations such as relaxed coscheduling and descheduling idle VCPUs provide a high-performance execution environment that efficiently utilizes physical host resources.

Appendix: ESX Server coscheduling statistics

ESX Server 3.x exports statistics related to the coscheduling behavior of multiprocessor VMs. The "esxtop" utility can be used to examine these statistics on a live ESX system.

The %CSTP column in the CPU statistics panel shows the fraction of time the VCPUs of a VM spent in the "co-stopped" state, waiting to be "co-started". This gives an indication of the coscheduling overhead incurred by the VM. If this value is low, then any performance problems should be attributed to other issues, and not to the coscheduling of the VM's virtual cpus.

Storage System Performance Analysis with Iometer

Thu, 27 Mar 2008 18:02:53 GMT

Introduction

Iometer is an open source tool originally developed by Intel that remains the simplest and best means of generating load on a system for performance analysis. Because of minute fluctuations in the in-guest timers many in-guest benchmarks are unable to produce accurate results. We at VMware have used Iometer for years and find its results to be accurate in most situations. But see the disclaimer below for a word of caution.

Figure 1. Iometer with disk targets tab selected.

Figure 1 shows the topology and disk target tab for a particular VM. As the manager only has one host visible beneath it and only one worker is present on another host, you can tell that this is a single server with only one processor. Had Iometer been started on a 4-way box, four workers would be visible. As the UI remains operational, dynamo workers can be connected from other hosts to drive load across multiple VMs that may be on multiple servers.

Configuring the Test

For all Iometer tests, under "Disk Targets" always increase the "# of Outstanding I/Os" per target. When left at the default value of ‘1', a relative low load will be placed on the array. By increasing this number some the OS will queue up multiple requests and really saturate the storage. The ideal number of outstanding IOs can be determined by running the test multiple times and increasing this number all the while. At some point IOPS will stop increasing. Generally an increase in return diminishes around 16 IOs/target but certainly more than 32 IOs/target will have no value due to the default queue depth in ESX. See Storage Queues and Performance for more information on queues. In Figure 1 you can see that "# of Outstanding I/Os defaults to 1."

When choosing the system to test in the topology frame, the "Disk Targets" tab will provide options as to the storage target. The options here include formatted disks (yellow) or unformatted disks (blue). In the former case Iometer address the storage through the OS's file system (FS). In the latter, direct calls are made to the hardware without using a FS. Storage specialists are usually more interested in just the hardware so evaluation of unformatted LUNs (blue) is preferable. There is some cost of virtualizing the OS's interface to the disk through the FS so formatting the disk with the correct FS and testing the yellow target can be helpful. Figure 1 shows two testable drives: the yellow-iconed C drive on which the OS was installed and the blue-iconed unformatted drive that is preferable for benchmarking.

Always make certain that the "maximum disk size" in the "Disk Targets" tab is larger then the available memory! For instance, when testing a formatted disk, setting the maximum size to 200,000 sectors (or 100 MB) could be cached by the guest OSes in a VM provided 1 GB of RAM. In this case all Iometer calls to storage will be intercepted and cached by the guest, host, or storage cache. Setting the disk maximum disk size to a number at least four times greater than the memory available in the largest cache will avoid caching.

Under the "Access Specifications" tab, choose a workload that matches the most interesting profile. Real workloads that are dominated by database performance often randomly read and write small, fixed-size block IO. SQL Server on Windows, for instance, uses 16K blocks, 66% read (which implies 34% write), and 100% random (thus 0% sequential). Exchange 2007 uses a similar profile but with an 8K block size. Oracle on Linux has flexibility to use the block size set when the file system was created. Depending on the DB specialists needs, this can range from 2K to 64K but will again be random with a 2:1 read-to-write ratio. Note: you can approximate this Linux performance on a Windows guest but do not run Iometer on Linux (see "Iometer On Linux" below.)

Application	Block Size	Randomness	Read/write Ratio
Exchange 2003	4K	80%	60% read (40% write)
Exchange 2007	8K	80%	55% read (45% write)
SQL Server	16K, 64K	100%	66% read (34% write)

Table 1. Example Iometer profiles.

Note the number of workers that has been specified in under the manager. This will default to one worker (thread) for each physical or virtual processor on the system. In the event that Iometer is being used to compare native to virtual performance, make sure that the worker numbers match! For instance, the work count will be one on a UP VM but four for the same native measurements if the system is quad-core. Correct the native worker count by detaching workers.

Before invoking the test, be aware of the potential impacts of data alignment. VMware has demonstrated substantial differences in performance based on alignment of data on storage arrays in our partition alignment paper. Make sure that partitions and virtual disks have been created by Virtual Center to guarantee that the partitions and files are properly aligned.

Disclaimer (or: When Not To Trust Iometer)

As discussed in Time-based Measurements in Virtual Machines, the hypervisor may introduce some inaccuracy to in-guest time measurement. The likelihood of measurement error increases as the server's load increases. Generally servers that are using less than 30% of their available CPU resources can be trusted. In the event that a large VM on a small server is driving all CPUs to high utilization, Iometer results may suffer from some inaccuracy. This is rarely the case with Iometer runs.

Analyzing the Results

As mentioned above, the results provided by Iometer tend to be trustworthy. But for unimpeachable results use the analysis techniques provide by VMware. See the Performance Monitoring and Analysis that's already been dedicated to this subject.

Performance Monitoring and Analysis

Wed, 26 Mar 2008 20:12:44 GMT

Introduction

A common question that crosses my desk is, "how do I analyze and correct performance?" There are a variety of techniques for doing this and many sources of information spread around the interweb, so I'm going to collect a few thoughts here.

Monitoring

Guest-based performance monitoring is an inaccurate and unhelpful means of evaluating performance in virtual deployments. See Guest-based Performance Measurement for more inforamtion. Monitoring and analysis of VMware ESX Server should be performed with esxtop and VirtualCenter.

esxtop

esxtop is the tried-and-true means of collecting every performance stat needed and making it available in a way that is conducive to analysis. The best source of information on launching esxtop can be found in the Resource Management Guide (page 159). It's worth nothing that the directions in that guide came from ESX Server 3.0.1. Since then, the developers have been kind enough to simplify the process of including all performance counters. This is done with the "-a" switch. So,

esxtop -a -b > analysis.csv

runs esxtop in batch mode and prints all performance counters. Let me give you the quick "do-s" and "don't-s" of esxtop:

With esxtop on ESX Server 3.5 and newer, always include the "-a" option to display all counters.
With esxtop on versions of ESX Server prior to 3.5 always follow the resource management guide to enable all counters. The storage latency statistics, for instance, are not displayed by default!
Always start your VMs before running esxtop in batch mode. If you start them after starting "esxtop -b", then esxtop will only produce VM data based on the VMs that were running at the time of its start.

VirtualCenter

VirtualCenter doesn't provide all of the performance counters you might need to analyze performance. But, it provides more than you might think! The default setup for VirtualCenter performance counter collection is fairly minimal. But this can be expanded by reconfiguring VC's performance counter collection. This is done as follows:

From the VI client, Administration->VirtualCenter Management Server Configuration...
On the left, click "Statistics".
Now increase the stats level as you see fit.

More information on VC performance counters and archival can be found in the Understanding VirtualCenter Performance Statistics wiki article. A few things are worth considering before going crazy with this:

You probably will never need level four stats. Those are mainly for debugging.
Use the DB size estimator after you monkey around with these levels. The DBs can get quite large and you want to know this before it happens and your VC system performance suffers.

Analysis

Performance analysis techniques were previously detailed in the performance analysis whitepaper. That has been migrated to this wiki and new material will appear here.

The following pages will provide guidance for identifying and correcting performance problems on ESX-based systems. We recommend following each of these pages in the order they're presented here.

Check and correct CPU utilization: CPU Performance Analysis and Monitoring
Identify memory bottlenecks and remove: Memory Performance Analysis and Monitoring
Characterize storage performance and correct: Storage Performance Analysis and Monitoring
Understand and improve the network utilization profile: Network Performance Analysis and Monitoring

Within each of these pages are techniques for using counters from VirtualCenter and esxtop. Information on those counters is provided in vCenter Performance Counters and esxtop Performance Counters, respectively.

Also, note that, while useless in collecting performance data, Perfmon can help with analysis of large esxtop output files. See Using Perfmon for esxtop-based Performance Analysis for more information.

Linux Timer Rate

Fri, 14 Mar 2008 16:30:54 GMT

Introduction

A hardware timer is used by modern systems for a variety of fine-grained operations at the operating system level. VMware's virtualization platforms virtualize this timer in the kernel. Because the virtual timer provided to the VM is actually software, it is subject to the same resource restrictions as other processes. The busier the system the more the timer execution must contend with other hypervisor activities. There are two implications of this:

When the system is very busy, the software timer may not execute as regularly and virtual time may fall behind.
Depending on how frequently the OS wishes to be interrupted by the timer, the hypervisor must do different amount of work.

From VMware KB article #1420.

Linux guest operating systems keep time by counting timer interrupts. Unpatched 2.4 and earlier kernels program the virtual system timer to request clock interrupts at 100Hz (100 interrupts per second). 2.6 kernels, on the other hand, request interrupts at 1000Hz - ten times as often. Some 2.4 kernels modified by distribution vendors to contain 2.6 features also request 1000Hz interrupts, or in some cases, interrupts at other rates, such as 512Hz.
Furthermore, an SMP-capable Linux kernel requests additional timer interrupts from the virtual local APIC timer. An SMP-capable kernel running on a one-CPU system generates twice as many total timer interrupts as the corresponding UP kernel, while such a kernel running on a two-CPU system requests three times as many. In general, an SMP-capable kernel running on <n> CPUs requests <n+1> times as many interrupts per second as a UP kernel. For example, an unmodified 2.6 Linux kernel running on a two-CPU virtual machine requests a total of 3000 clock interrupts per second.
When a guest asks for more than 1000 clock interrupts per second, it can be difficult for the virtual machine to keep up, especially if other applications are running on the host at the same time. This can cause the clock in the guest operating system to fall so far behind real time that it is unable to catch up. The overhead of delivering so many virtual clock interrupts can also hurt guest performance and increase host CPU consumption.

The amount of work required to manage the virtual timer is greatest with Red Hat Enterprise Linux 5 (RHEL5) SMP systems, which use a clock frequency of 1000 Hz and suffer from a multiplicative amount of work due to SMP support. For instance, the following table of timer interrupts was created on a 1000 Hz RHEL VM:

vCPU Count	Interrupts/sec
2	6,000
4	20,000
8	72,000

So, the amount of work that needs to be done by the hypervisor increases dramatically with the addition of vCPUs. In addition, decreasing the timer interrupt rate greatly decreases the work that needs to be done by the VMkernel to virtualize the timer. In RHEL 5.1, a Linux kernel that enables reducing the timer rate was included. By adding the parameter "divider=10" to the boot parameters, the amount of work required of the VMkernel to virtualize the timer goes down by an order of magnitude.

VMware Communities : Document List - Performance & VMmark

IOmega ix4-200d IOmeter results (100MB/Full Network)

Advanced Networking Performance Options

rename_vm.sh

clone_vm2.sh

VAM Doc

Memory Performance Chart Metrics in the vSphere Client

Performance Troubleshooting for VMware vSphere 4 and ESX 4.0

Storage Workload Characterization and Consolidation in Virtualized Enviornments

vscsiStats: Fast and Easy Disk Workload Characterization on VMware ESX Server

Using vscsiStats for Storage Performance Analysis

Introduction

Running vscsiStats

Starting and Stopping vscsiStats Collection

Viewing Statistics

Using vscsiStats Results

Use Case 1: Identifying Sequential IO

Use Case 2: Optimizing for IO Sizes

Use Case 3: Storage Latency Analysis (Including NFS!)

vscsiStats on ESXi

Additional Resources

Meet the Engineer Series: VMware Performance Advancements

ESX Monitor Modes

Monitor Defaults with Intel Processors

Monitor Defaults with AMD Processors

Legend

Storage Performance: VMFS and Protocols

Introduction

Storage Protocols

VMFS Scalability

Best Practices for IBM Lotus Domino

Introduction

General Recommendations

VM configuration

Storage

Virtual CPUs

Memory

Networking

Resource Management

Deployment

Interpreting esxtop Statistics

Table of Contents

Section 1. Introduction

Section 2. CPU

Section 2.1 Worlds and Groups

Section 2.2 Global Statistics

Section 2.3 World Statistics

Section 3. Memory

Section 3.1 Machine Memory and Guest Physical Memory

Section 3.2 Global Statistics

Section 3.3 Group Statistics

Section 4 Disk

Section 4.1 Adapter, Device, VM screens

Section 4.2 Disk Statistics

Section 4.2.1 I/O Throughput Statistics

Section 4.2.2 Latency Statistics

Section 4.2.3 Queue Statistics

Section 4.2.4 Error Statistics

Section 4.2.5 PAE Statistics

Section 4.2.6 Split Statistics

Section 4.3 Batch Mode Output

Section 5 Network

Section 5.1 Port

Section 5.2 Port Statistics

Section 6. Interrupt

Section 7. Batch Mode

Best Practices for SQL Server

Introduction

Virtualizing SQL: The Checklist

Item 1: Configure Storage Correctly

Item 2: Use Recent Hardware

Item 3: Follow SQL Server Best Practices

Item 4: Configure VM Identically to Native and Run The Right Test

Item 5: Use VMware's ESX Server

Item 6: Understand Memory Management and Configure Correctly

Item 7: Align Disk Partitions

Whitepapers

Ready Time

Introduction

Details