VMware Communities : All Content - All Communities

Advanced Networking Performance Options

drummonds — Wed, 14 Oct 2009 16:58:33 GMT

Some of the advanced networking options available in vSphere 4.0 are reviewed in this paper. Many of these options control trade-offs between latency, throughput, CPU utilization, and reliability (e.g., dropped packets). It is not possible to optimize all of these at the same time, so option defaults are chosen to be suitable for the vast majority of applications. These options are provided to meet the stricter requirements of other applications. Advanced options often have subtle side effects, or merely move an issue from one area to another. Therefore it is recommended that VMware Support be engaged before changing such options, especially for production machines.

There are over 100 options that can be set under Configuration → Advanced Settings → Net. Of these, the ones listed below are most likely to be useful for tuning networking performance. Many of the others are for internal testing or enable unreliable features.

All of the options listed here take integer values. For the “Boolean” ones only the default value is shown: 0 for “false”, and 1 for “true”. Other parameters are shown with their default, minimum, and maximum values.

Parameter Name	(Default, Minimum, Maximum)	Description
MaxPortRxQueueLen	(80, 1, 500)	Maximum length of the Rx queue for virtual ports whose clients support queueing. Possibly should be increased if Rx packet drops are seen in the port connected to a VM. Relevant only for e1000 vNICs used with Fault Tolerance (FT) and VLANs.
MaxNetifTxQueueLen	(500, 1, 1000)	Maximum length of the Tx queue for the physical NICs. Increase if Tx packet drops are seen in uplink port to the pNIC.
GuestTxCopyBreak	(64, 60, 4294967295)	Packet header transmits smaller than this in bytes will be copied rather than mapped. More security and functionality than performance implications.
VmxnetTxCopySize	(256, 0, 4294967295)	Transmits smaller than this in bytes will be copied rather than mapped. Copying costs CPU but puts lets pressure on the Tx queue and doesn’t require completion.
VmxnetWinUDPTxFullCopy	(1)	Enable full copy of Windows vmxnet UDP Tx packets. Might disable to save CPU, especially for jumbo frames, at the cost of risking more packet drops.
NetTxDontClusterSize	(0, 0, 8192)	Tx packet size (in bytes) smaller than this are transmitted immediately (coalescing options are over-ruled for these packets). Used to ensure good latency for small packets.
CoalesceTxTimeout	(4000, 1, 4294967295)	The coalesce timeout in micro-seconds, or effectively the maximum latency without transmitting. Smaller values can reduce the packet latency at the cost of CPU. Risky to go below 1000.
CoalesceDefaultOn	(1)	Enable dynamic coalescing. Disable to test if issues are related to coalescing.
CoalesceHandlerPcpu	(1, 0, 128)	pCPU that coalesce timeout handler runs on. May be important to set this if VM CPU pinning is used.
CoalesceTxQDepthCap	(40, 0, 80)	Maximum number of “normalized” Tx packets to coalesce. Reduce if Tx coalescing appears to be too aggressive.
CoalesceRxQDepthCap	(40, 0, 80)	Maximum number of “normalized” Rx packets to coalesce. Reduce if Rx coalescing appears to be too aggressive.
vmxnetThroughputWeight	(0, 0, 255)	How far to favor Tx throughput for vmxnet 2 & 3. “0” is dynamic, otherwise this is a weight where a lower value favors latency and a higher value favors throughput.
TcpipHeapSize	(24, 24, 120)	Initial size of the TCP/IP module heap in megabytes. May need to increase if there are many vmkernel connections (NFS, iSCSI, etc.).
TcpipDefLROMaxLength	(16000, 1, 65535)	Maximum length for the LRO aggregated packet for vmkernel connections. Increasing this reduces the number of acknowledgments, which improves efficiency but may increase latency.
E1000TxZeroCopy	(0)	If disabled copy UDP or non-TSO Tx packets for e1000.
E1000TxTsoZeroCopy	(1)	If enabled do not copy TSO Tx packets for e1000.
E1000IntrCoalesce	(1)	Enable interrupt coalescing for e1000. Disabling can improve latency at the expense of CPU.
MaxPktRxListQueue	(3500, 0, 200000)	Maximum number of packets queued in vmkernel. Increasing this can reduce the number of dropped packets but at the cost of increased vmkernel memory and queuing latency.
Vmxnet3RSSHashCache	(1)	Enable RSS hash cache for vmxnet3 in Windows guests.
VmklnxLROEnabled	(0)	Enable large packets for recent Linux guests with vmxnet 2 & 3. Most likely to benefit hosts with small number of VMs with few sessions each, where each session has a heavy Rx load (more than 1 MB/sec). This is an experimental feature and has not been tested extensively.
VmklnxLROMaxAggr	(6, 0, 24)	Maximum aggregation count in number of packets for vmklinux LRO.

Using vscsiStats for Storage Performance Analysis

drummonds — Fri, 29 May 2009 22:28:27 GMT

Introduction

esxtop is a great tool for performance analysis of all types. However, with only latency and throughput statistics, esxtop will not provide the full picture of the storage profile. Furthermore, esxtop only provides latency numbers for Fibre Channel and iSCSI storage. Latency analysis of NFS traffic is not possible with esxtop.

Since ESX 3.5, VMware has provided a tool specifically for profiling storage: vscsiStats. vscsiStats collects and reports counters on storage activity. Its data is collected at the virtual SCSI device level in the kernel. This means that results are reported per VMDK (or RDM) irrespective of the underlying storage protocol. The following data are reported in histogram form:

IO size
Seek distance
Outstanding IOs
Latency (in microseconds)
More!

Running vscsiStats

vscsiStats collection and analysis requires two steps:

Start statistics collection.
View accrued statistics.

Documentation on command-line parameters are available when running '/usr/lib/vmware/bin/vscsiStats -h'.

Starting and Stopping vscsiStats Collection

The tool is started with the following command:

/usr/lib/vmware/bin/vscsiStats -s -w <world_group_id>

This command starts the process that will accrue statistics. The world group ID must be set to a running virtual machine. The running VMs' IDs can be obtained by running '/usr/lib/vmware/bin/vscsiStats -l'.

After about 30 minutes vscsiStats will stop running. If the analysis is needed for a longer period, the start command should be repeated above in this window. That will defer the timeout and termination by another 30 minutes.

Since results are accrued and reported out in summary, the histograms will include data since collection was started. To reset all counters to zero, run '/usr/lib/vmware/bin/vscsiStats -r'.

Viewing Statistics

Counters are displayed by using the following command:

/usr/lib/vmware/bin/vscsiStats -p <histo_type> [-c]

The histogram type is used to specify either all of the statistics or one group of them. Options include all, ioLength, seekDistance, outstandingIOs, latency, interarrival.

Results can be produced in a more compact comma-delimited list by adding the optional "-c" above.

Using vscsiStats Results

Use Case 1: Identifying Sequential IO

Storage arrays can process sequential IO much faster than random IO. You can therefore improve the performance of a sequential workload by placing it on a dedicated LUN to allow the array to optimize access. vscsiStats can help you identify your sequential workloads even if you don't understand anything about the application in the VM.

Take the following graph as example, which I generated by running '/usr/lib/vmware/bin/vscsiStats -p seekDistance':

This graph shows that most of the commands are being issued a great distance from the previous command. It looks like all of the commands were 50,000 or more logical blocks away from the previous command. When I looked at the raw data, I saw that over 99% of the commands were more than 128 blocks away from the previous command. That's random access if I've ever seen it. Here's the opposite example:

In this case the logical block number (LBN) of each command is most frequently exactly one larger than the previous command. That's the signature of a heavily sequential workload. It shouldn't surprise you to learn that both of these profiles were generated by Iometer using random and sequential writes, respectively.

Use Case 2: Optimizing for IO Sizes

The IO size is an important characteristic of storage profiles. A variety of best practices have been provided by storage vendors to enable customers to tune their storage to a particular IO size. As an example, it may make sense to optimize an array's stripe size to its average IO size. vscsiStats can provide a histogram of IO sizes to help this process. The following graph was generated by '/usr/lib/vmware/bin/vscsiStats -p ioLength':

From these results I can see that about a quarter of the commands came in IOs smaller than 4k. About half of the commands were sized to 4k commands. The minute number of remaining IOs were larger than 4k. This signature is common of a VMDK formatted to 4k blocks and supporting OS and application execution. The storage array should be optimized for 4k blocks if this disk's performance is a priority.

Use Case 3: Storage Latency Analysis (Including NFS!)

esxtop is a terrific tool for latency-based storage analysis. Fibre Channel and iSCSI HBAs have device and kernel latencies in esxtop's storage panel. Software iSCSI initiators will show up as vmhba32 (ESX 3.5 and earlier) and vmhba33 (ESX 4.0 and later.) But esxtop does not provide latency statistics for NFS stores.

Because vscsiStats collects its results where the guest interacts with the hypervisor, it is unaware of the storage implementation. Latency statistics can be collected for all storage configurations with this tool.

The above graph shows that the server in my office with a single direct-attached SCSI disk is performing as I would expect. About half of all the operations are completing in under 5 ms. The other half take 5-15 ms to complete. A few commands took longer than 15 ms, but the number is so small that it doesn't concern me. Similar results can be seen with NFS arrays.

vscsiStats on ESXi

vscsiStats can be installed on ESXi hosts after putting the host into tech support mode. More information on this process is availalble on Scott's blog on the subject on vPivot.

Additional Resources

My colleagues Ajay Gulati, Chethan Kumar, and Irfan Ahmad presented at VPACT 09 Storage Workload Characterization and Consolidation in Virtualized Enviornments. This paper serves as an excellent example of vscsiStats in action.

I learned vscsiStats by reviewing Irfan's VMworld 2007 presentation (vscsiStats: Fast and Easy Disk Workload Characterization on VMware ESX Server) and playing with the tool. Check out his presentation if you'd like more detail.

Understanding Performance

drummonds — Fri, 16 May 2008 21:30:03 GMT

The following documents will explain some of the principles for virtual system performance. Please check back as we grow the number of articles here with time.

ESX and Guest Operating Systems

CPU and Scheduling

Memory

Large Memory Pages

Network

Advanced Networking Performance Options

Storage

vCenter Performance Counters

drummonds — Fri, 30 May 2008 00:15:21 GMT

Introduction

The following table of vCenter (VC) performance counters lists the counters with a description of their purpose. This page has been updated for vSphere 4, so the counter levels will differ slightly on older versions of VC.

Remember, with the exception of ready time, statistic levels one and two are the only ones needed for 99% of the performance monitoring and analysis out there. Don't spend many of your own cycles worrying about levels three and four!

For information on enabling VC to display and archive these counters see the Understanding vCenter Performance Statistics article.

Understanding vCenter Measurement Windows

Before you continue, you should know that all total count metrics reported by VC are reported over the sample window. When you're looking at live stats, this sample window is 20 seconds. When you're looking at archive stats, it will depend on the interval duration. That duration could be five minutes, 30 minutes, two hours, or one day.

This causes a lot of confusion when comparing esxtop results to live VC results to archived VC results. As an example, ready time might be reported as 10% in esxtop. In live VC results this amount of ready time would be reported as 2000 ms (10% of the 20s window.) In one day archive results, the same number would be reported as 30,000 ms (10% of the five minute interval duration.) All of these numbes reflect the same amount of ready time.

CPU Statistics

Level	Counter name in API	Description	Units
1	cpu.ready.summation	Ready time is the time spend waiting for CPU(s) to become available in the past update interval.	millisecond
1	cpu.usagemhz.average	The CPU utilization. The maximum possible value here is the frequency of the processors times the number of cores. As an example, a VM using 4000 MHz on a system with four 2 GHz processors is using 50% of the CPU (4000 / (4 * 2000) = 0.5)	megaHertz
1	cpu.usage.average	The CPU utilization. This value is reported with 100% representing all processor cores on the system. As an example, a 2-way VM using 50% of a four-core system is completely using two cores.	percent
2	cpu.reservedCapacity.average	CPU Reserved Capacity	megaHertz
2	cpu.idle.summation	CPU Idle	millisecond
2	cpu.swapwait.summation	Swap wait time is time that the world spent waiting for memory to be swapped in. When the VM is waiting for memory, it is not doing work.	millisecond
3	cpu.system.summation	System time is the time spent in VMkernel during the last update interval. This does not include guest code execution.	millisecond
3	cpu.wait.summation	Wait time is the time spent waiting for hardware or VMkernel lock thread locks during the last update interval.	millisecond
3	cpu.extra.summation	CPU extra is the time above the statically calculated entitlement. Entitlement is the share of processing time that a VM should get as a result of its vCPU count and assigned shares. You should not use or care about this counter in any of your own analysis.	millisecond
3	cpu.used.summation	CPU Used	millisecond
3	cpu.guaranteed.latest	Guaranteed time is reported as the amount of the reservation time that the VM used in the past update interval. As an example, if 2000 MHz have been reserved for the VM on an four-way, 2 GHz host, that's 25% of the CPU resource. In a 20s update interval, there are 80,000 ms available on this four-way system. That means 20,000 ms of time has been reserved. If a VM used only half of its available cycles, the guaranteed time is 10,000 ms.	millisecond
4	cpu.usage.none	CPU Usage (None)	percent
4	cpu.usage.minimum	CPU Usage (Minimum)	percent
4	cpu.usage.maximum	CPU Usage (Maximum)	percent
4	cpu.usagemhz.none	CPU Usage in MHz (None)	megaHertz
4	cpu.usagemhz.minimum	CPU Usage in MHz (Minimum)	megaHertz
4	cpu.usagemhz.maximum	CPU Usage in MHz (Maximum)	megaHertz

Memory Statistics

Level	Counter name in API	Description	units
1	mem.consumed.average	The amount of machine memory that is in use by the VM. While a VM may have been configured to use 4 GB of RAM, as an example, it might have only touched half of that. Of the 2 GB left, half of that might be saved from memory sharing. That would result in 1 GB of consumed memory.	kiloBytes
1	mem.overhead.average	The memory used by the VMkernel to maintain and execute the VM.	kiloBytes
1	mem.swapinrate.average	The swap in rate reports the rate at which a VM's memory is being swapped in from disk.	kiloBytesPerSecond
1	mem.swapoutrate.average	The swap out rate reports the rate at which a VM's memory is being swapped out to disk.	kiloBytesPerSecond
1	mem.usage.average	The percentage of memory used as a percent of all available machine memory. Available for host and VM.	percent
1	mem.vmmemctl.average	The amount of memory currently claimed by the balloon driver. This is not a performance problem, per se, but represents the host starting to take memory from less needful VMs for those with large amounts of active memory. But if the host is ballooning, check swap rates (swapin and swapout) which would be indicative of performance problems.	kiloBytes
2	mem.granted.average	The amount of memory that was granted to the VM by the host. Memory is not granted to the host until it is touched one time and granted memory may be swapped out or ballooned away if the VMkernel needs the memory.	kiloBytes
2	mem.active.average	The amount of memory used by the VM in the past small window of time. This is the "true" number of how much memory the VM currently has need of. Additional, unused memory may be swapped out or ballooned with no impact to the guest's performance.	kiloBytes
2	mem.shared.average	The average amount of shared memory. Shared memory represents the entire pool of memory from which sharing savings are possible. The amount of memory that this has been condensed to is reported in shared common memory. So, total saving due to memory sharing equals shared memory minus shared common memory.	kiloBytes
2	mem.zero.average	The amount of zero pages in the guest. Zero pages are not represented in machine memory so this results in 100% savings when mapping from the guest to the machine memory.	kiloBytes
2	mem.unreserved.average	Memory Unreserved (Average)	kiloBytes
2	mem.swapused.average	The amount of swap memory currently in use. A large amount of swap memory is not a performance problem. This could be memory that the guest doesn't need. Check the swap rates (swapin, swapout) to see if the guest is actively in need of more memory than is available.	kiloBytes
2	mem.swapunreserved.average	Memory Swap Unreserved (Average)	kiloBytes
2	mem.sharedcommon.average	The average amount of shared common memory. Shared memory represents the entire pool of memory from which sharing savings are possible. The amount of memory that this has been condensed to is reported in shared common memory. So, total saving due to memory sharing equals shared memory minus shared common memory.	kiloBytes
2	mem.heap.average	Memory Heap (Average)	kiloBytes
2	mem.heapfree.average	Memory Heap Free (Average)	kiloBytes
2	mem.state.latest	Memory State	number
2	mem.swapped.average	Memory Swapped (Average)	kiloBytes
2	mem.swaptarget.average	Memory Swap Target (Average)	kiloBytes
2	mem.swapin.average	The rate at which memory is being swapped in from disk. A large number here represents a problem with lack of memory and a clear indication that performance is suffering as a result.	kiloBytes
2	mem.swapout.average	The rate at which memory is being swapped out to disk. A large number here represents a problem with lack of memory and a clear indication that performance is suffering as a result.	kiloBytes
2	mem.vmmemctltarget.average	Memory Balloon Target (Average)	kiloBytes
2	mem.sysUsage.average	Memory Used by vmkernel	kiloBytes
2	mem.reservedCapacity.average	Memory Reserved Capacity	megaBytes
4	mem.usage.none	Memory Usage (None)	percent
4	mem.usage.minimum	Memory Usage (Minimum)	percent
4	mem.usage.maximum	Memory Usage (Maximum)	percent
4	mem.granted.none	Memory Granted (None)	kiloBytes
4	mem.granted.minimum	Memory Granted (Minimum)	kiloBytes
4	mem.granted.maximum	Memory Granted (Maximum)	kiloBytes
4	mem.active.none	Memory Active (None)	kiloBytes
4	mem.active.minimum	Memory Active (Minimum)	kiloBytes
4	mem.active.maximum	Memory Active (Maximum)	kiloBytes
4	mem.shared.none	Memory Shared (None)	kiloBytes
4	mem.shared.minimum	Memory Shared (Minimum)	kiloBytes
4	mem.shared.maximum	Memory Shared (Maximum)	kiloBytes
4	mem.zero.none	Memory Zero (None)	kiloBytes
4	mem.zero.minimum	Memory Zero (Minimum)	kiloBytes
4	mem.zero.maximum	Memory Zero (Maximum)	kiloBytes
4	mem.unreserved.none	Memory Unreserved (None)	kiloBytes
4	mem.unreserved.minimum	Memory Unreserved (Minimum)	kiloBytes
4	mem.unreserved.maximum	Memory Unreserved (Maximum)	kiloBytes
4	mem.swapused.none	Memory Swap Used (None)	kiloBytes
4	mem.swapused.minimum	Memory Swap Used (Minimum)	kiloBytes
4	mem.swapused.maximum	Memory Swap Used (Maximum)	kiloBytes
4	mem.swapunreserved.none	Memory Swap Unreserved (None)	kiloBytes
4	mem.swapunreserved.minimum	Memory Swap Unreserved (Minimum)	kiloBytes
4	mem.swapunreserved.maximum	Memory Swap Unreserved (Maximum)	kiloBytes
4	mem.sharedcommon.none	Memory Shared Common (None)	kiloBytes
4	mem.sharedcommon.minimum	Memory Shared Common (Minimum)	kiloBytes
4	mem.sharedcommon.maximum	Memory Shared Common (Maximum)	kiloBytes
4	mem.heap.none	Memory Heap (None)	kiloBytes
4	mem.heap.minimum	Memory Heap (Minimum)	kiloBytes
4	mem.heap.maximum	Memory Heap (Maximum)	kiloBytes
4	mem.heapfree.none	Memory Heap Free (None)	kiloBytes
4	mem.heapfree.minimum	Memory Heap Free (Minimum)	kiloBytes
4	mem.heapfree.maximum	Memory Heap Free (Maximum)	kiloBytes
4	mem.swapped.none	Memory Swapped (None)	kiloBytes
4	mem.swapped.minimum	Memory Swapped (Minimum)	kiloBytes
4	mem.swapped.maximum	Memory Swapped (Maximum)	kiloBytes
4	mem.swaptarget.none	Memory Swap Target (None)	kiloBytes
4	mem.swaptarget.minimum	Memory Swap Target (Minimum)	kiloBytes
4	mem.swaptarget.maximum	Memory Swap Target (Maximum)	kiloBytes
4	mem.swapin.none	Memory Swap In (None)	kiloBytes
4	mem.swapin.minimum	Memory Swap In (Minimum)	kiloBytes
4	mem.swapin.maximum	Memory Swap In (Maximum)	kiloBytes
4	mem.swapout.none	Memory Swap Out (None)	kiloBytes
4	mem.swapout.minimum	Memory Swap Out (Minimum)	kiloBytes
4	mem.swapout.maximum	Memory Swap Out (Maximum)	kiloBytes
4	mem.vmmemctl.none	Memory Balloon (None)	kiloBytes
4	mem.vmmemctl.minimum	Memory Balloon (Minimum)	kiloBytes
4	mem.vmmemctl.maximum	Memory Balloon (Maximum)	kiloBytes
4	mem.vmmemctltarget.none	Memory Balloon Target (None)	kiloBytes
4	mem.vmmemctltarget.minimum	Memory Balloon Target (Minimum)	kiloBytes
4	mem.vmmemctltarget.maximum	Memory Balloon Target (Maximum)	kiloBytes
4	mem.overhead.none	Memory Overhead (None)	kiloBytes
4	mem.overhead.minimum	Memory Overhead (Minimum)	kiloBytes
4	mem.overhead.maximum	Memory Overhead (Maximum)	kiloBytes
4	mem.consumed.none	Memory Consumed (None)	kiloBytes
4	mem.consumed.maximum	Memory Consumed (Maximum)	kiloBytes
4	mem.consumed.minimum	Memory Consumed (Minimum)	kiloBytes
4	mem.sysUsage.none	Memory Used by vmkernel	kiloBytes
4	mem.sysUsage.maximum	Memory Used by vmkernel	kiloBytes
4	mem.sysUsage.minimum	Memory Used by vmkernel	kiloBytes

Disk Statistics

Level	Counter name in API	Description	units
1	disk.maxTotalLatency	The highest reported total latency (device and kernel times) in the sample window.	milliseconds
1	disk.usage.average	Average disk throughput over the sample period.	kiloBytesPerSecond
2	disk.read.average	Average disk throughput due to read operaitons over the sample period.	kiloBytesPerSecond
2	disk.write.average	Average disk throughput due to write operations over the sample period.	kiloBytesPerSecond
2	disk.commands.summation	Disk Commands Issued	number
2	disk.commandsAborted.summation	The number of aborts that have occurred in the last window of time. Abort commands are issued by the guest when the storage system has not responded within an acceptable amount of time (as defined by the guest OS or application.)	number
2	disk.busResets.summation	Disk Bus Resets	number
2	disk.deviceReadLatency.average	Device read latency. This is the time the physical device from the HBA to the platter takes to service an IO request.	millisecond
2	disk.kernelReadLatency.average	Kernel read latency. This is the time the VMkernel takes to service an IO. This is the time between the guest OS and the device.	millisecond
2	disk.totalReadLatency.average	Total read latency. The sum of the device and kernel read latencies.	millisecond
2	disk.queueReadLatency.average	Queue Read Latency	millisecond
2	disk.deviceWriteLatency.average	Device write latency. This is the time the physical device from the HBA to the platter takes to service an IO request.	millisecond
2	disk.kernelWriteLatency.average	Kernel write latency. This is the time the VMkernel takes to service an IO. This is the time between the guest OS and the device.	millisecond
2	disk.totalWriteLatency.average	Total write latency. The sum of the device and kernel write latencies.	millisecond
2	disk.queueWriteLatency.average	Queue Write Latency	millisecond
2	disk.deviceLatency.average	Physical Device Command Latency	millisecond
2	disk.kernelLatency.average	Kernel Disk Command Latency	millisecond
2	disk.queueLatency.average	Queue Command Latency	millisecond
3	disk.numberRead.summation	The number of IO read operations in the previous sample period. Note that these operations may be variable sized up to 64 KB.	number
3	disk.numberWrite.summation	The number of IO write operations in the previous sample period. Note that these operations may be variable sized up to 64 KB.	number
3	disk.totalLatency.average	This is the average total latency over the sample window. Total latency is the sum of kernel and device latency for both read and write commands.	millisecond
3	disk.write.average	Disk Write Rate	kiloBytesPerSecond
4	disk.usage.none	Disk Usage (None)	kiloBytesPerSecond
4	disk.usage.minimum	Disk Usage (Minimum)	kiloBytesPerSecond
4	disk.usage.maximum	Disk Usage (Maximum)	kiloBytesPerSecond

Network Statistics

Level	Counter name in API	Description	units
1	net.usage.average	Network Usage (Average)	kiloBytesPerSecond
2	net.droppedRx.summation	The number of received packets that were dropped over the sample period.	number
2	net.droppedTx.summation	The number of transmitted packets that were dropped over the sample period.	number
2	net.received.average	Average network throughput for received traffic.	kiloBytesPerSecond
2	net.transmitted.average	Average network throughput for transmitted traffic.	kiloBytesPerSecond
3	net.packetsRx.summation	Network Packets Received	number
3	net.packetsTx.summation	Network Packets Transmitted	number
4	net.usage.none	Network Usage (None)	kiloBytesPerSecond
4	net.usage.minimum	Network Usage (Minimum)	kiloBytesPerSecond
4	net.usage.maximum	Network Usage (Maximum)	kiloBytesPerSecond

Other Statistics

Level	Counter name in API	Description	units
1	sys.uptime.latest	Uptime	second
1	sys.heartbeat.summation	Heartbeat	number
1	clusterServices.cpufairness.latest	CPU Fairness	number
1	clusterServices.memfairness.latest	Memory Fairness	number
1	clusterServices.effectivecpu.average	Effective CPU Resources	megaHertz
1	clusterServices.effectivemem.average	Effective Memory Resources	megaBytes
1	clusterServices.failover.latest	Current failover level	number
3	sys.resourceCpuUsage.average	Resource CPU Usage (Average)	megaHertz
3	managementAgent.memUsed.average	Memory Used (Average)	kiloBytes
3	managementAgent.swapUsed.average	Memory Swap Used (Average)	kiloBytes
3	managementAgent.swapIn.average	Memory Swap In (Average)	kiloBytesPerSecond
3	managementAgent.swapOut.average	Memory Swap Out (Average)	kiloBytesPerSecond
3	rescpu.actav1.latest	CPU Active (1 min. average)	percent
3	rescpu.actpk1.latest	CPU Active (1 min. peak)	percent
3	rescpu.runav1.latest	CPU Running (1 min. average)	percent
3	rescpu.actav5.latest	CPU Active (5 min. average)	percent
3	rescpu.actpk5.latest	CPU Active (5 min. peak)	percent
3	rescpu.runav5.latest	CPU Running (5 min. average)	percent
3	rescpu.actav15.latest	CPU Active (15 min. average)	percent
3	rescpu.actpk15.latest	CPU Active (15 min. peak)	percent
3	rescpu.runav15.latest	CPU Running (15 min. average)	percent
3	rescpu.runpk1.latest	CPU Running (1 min. peak)	percent
3	rescpu.maxLimited1.latest	CPU Throttled (1 min. average)	percent
3	rescpu.runpk5.latest	CPU Running (5 min. peak)	percent
3	rescpu.maxLimited5.latest	CPU Throttled (5 min. average)	percent
3	rescpu.runpk15.latest	CPU Running (15 min. peak)	percent
3	rescpu.maxLimited15.latest	CPU Throttled (15 min. average)	percent
3	rescpu.sampleCount.latest	Group CPU Sample Count	number
3	rescpu.samplePeriod.latest	Group CPU Sample Period	millisecond
4	sys.resourceCpuUsage.none	Resource CPU Usage (None)	megaHertz
4	sys.resourceCpuUsage.maximum	Resource CPU Usage (Maximum)	megaHertz
4	sys.resourceCpuUsage.minimum	Resource CPU Usage (Minimum)	megaHertz

New Blog Home

drummonds — Wed, 16 Sep 2009 18:19:47 GMT

I have moved my blog home to a new location. Come visit and read at vPivot.com.

Scott

Love Your Balloon Driver

drummonds — Wed, 09 Sep 2009 17:26:33 GMT

A couple of days ago we finally got out one of my favorite papers from our ongoing vSphere launch activities. This paper on ESX memory management, written by Fei Guo in performance engineering, has three graphs that are absolute gems. These graphs show balloon driver memory savings next to throughput numbers for three common benchmarks. The conclusion is inescapable: the balloon driver reclaims memory from over-provisioned VMs with virtually no impact to performance. This is true on every workload save one: Java.

Example 1: Kernel Compile

Linux kernel compilation models a common developer environment involving a large number of code compiles. This process is CPU and IO intensive but uses very little memory.

Results of two experiments are shown on this graph: in one memory is reclaimed only through ballooning and in the other memory is reclaimed only through host swapping. The bars show the amount of memory reclaimed by ESX and the line shows the workload performance. The steadily falling green line reveals a predictable deterioration of performance due to host swapping. The red line demonstrates that as the balloon driver inflates, kernel compile performance is unchanged.

Kernel compilation performance remains high with ballooning because this workload needs very little memory and the guest OS can easily take unused pages from the application. Performance falls with swapping because ESX randomly selects virtual machine pages for swapping, whether those pages are in use by the application or not. The guest OS is better at selecting pages for reclamation than ESX is.

Example 2: Oracle/Swingbench

Oracle's database is best tested against Swingbench, the OLTP load generation tool provided by Oracle. Database workloads utilize all system resources but show a non-linear dependence on memory. Memory can be safely reclaimed from OSes running databases until the cache becomes smaller than needed by the workload. The following figure shows this.

As before, the virtual machine using only ballooning maintains higher performance under memory pressure than the virtual machine whose memory is being swapped away by the host. Performance is constant and shows no negative impact due to ballooning until the balloon encroaches on the SGA. Again, ESX's host swapping randomly selects pages to send to disk which degrades performance even at small swap amounts.

As with kernel compile, the balloon driver safely reclaims memory from over-provisioned VMs with little impact to application performance.

Example 3: Java/SPECjbb

Java provides a special challenge in virtual environments due to the JVM's introduction of a third level of memory management. The balloon driver draws memory from the virtual machine without impacting throughput because the guest OS efficiently claims pages that its processes are not using. But in the case of Java, the guest OS is unaware of how the JVM is using memory and is forced to select memory pages an arbitrarily and inefficiently as ESX's swap routine.

Neither ESX nor the guest OS can efficiently take memory from the JVM without significantly degrading performance. Memory in Java is managed internal to the JVM and efforts by the host or guest to remove pages will equally negatively impact Java applications performance. In these environments it is wise to manually set the JVM's heap size and specify memory reservations for the virtual machine in ESX to account for the JVM, OS, and heap.

Conclusions and Scott's Special Recommendation

Love your balloon driver. Your application owners are always asking for more memory than they need. With great comfort you can over-provision memory some and rely on ESX and the balloon driver to reclaim what is not in use. Without the balloon driver, ESX will be forced to use its last technology for managing memory over-commit: host swapping. And host swapping always decreases performance.

So here is my special recommendation for you: never, ever disable the balloon driver. This forces the host to swap that virtual machine's memory, should that resource become scarce. And where ballooning usually will not hurt performance, swapping always will. If you must protect an application from memory reclamation due to memory over-commitment, use reservations. They make admission control more effective, they self-document the needs of the VM, and they are easily configured.

Four Things You Should Know About ESX 4's Scheduler

drummonds — Wed, 19 Aug 2009 16:59:14 GMT

I spent a great deal of time answering customers' questions about the scheduler. Never have so many questions been asked about such an abstruse component for which so little user influence is possible. But CPU scheduling is central to system performance, so VMware strives to provide as much information on the subject as possible. In this blog entry, I want to point out a few nuggets of information on the CPU scheduler. These four bullets answer 95% of the questions I get asked.

Item 1: ESX 4's Scheduler Better Uses Caches Across Sockets

On UMA systems with low load levels, virtual machine performance improves when each virtual CPU (vCPU) is placed on its own socket. This is because providing each vCPU its own socket also give it the entire cache on that CPU. On page 18 of a recent paper on the scheduler written by Seongbeom Kim, a graph highlights the case where vCPU spreading improves performance.

The X-axis represents different combinations of VM and vCPU counts. SPECjbb is memory intensive and shows great gains with increases in CPU cache. The few cases that show dramatic benefit due to the ESX 4.0 scheduler are benefiting from the distribution of vCPUs across sockets. Very large gains are possible in this somewhat uncommon case.

Item 2: Overuse of SMP Only Slows Consolidated Environments At Saturation

For years customers have asked me how many vCPUs they should give to their VMs. The best guidance, "as few as possible", seems too vague to satisfy. It remains the only correct answer, unfortunately. But a recent experiment performed by Bruce Herndon's team sheds some light on this VM sizing question.

In this experiment we ran VMmark against VMs that were configured outside of VMmark specifications. In one case some of the virtual machines were given too few vCPUs and in another they were given too many. Because VMmark's workload is fixed, changing VM sizes does not alter the amount of work performed by the VMs. In other words, the system's score does not depend on the VMs' vCPU count. Until CPU saturation, that is.

Notice that the scores are similar between the undersized, right-sized, and over-sized VMs. Up until tile 10 (60 VMs) they are nearly identical. There is a slight difference in processor utilization that begins to impact throughput (score) as the system runs out of CPU. At that point wasted cycles dedicated to unneeded vCPUs negatively impact the system performance. Two points I will call out from this work:

Sloppy VI admins that provide too many vCPUs need not worry about performance when their servers are under low load. But performance will suffer when CPU utilization spikes.
The penalty of over-sizing VMs gets worse as VMs get larger. Using a 2-way VM is not that bad, but unneeded use of 4-way VM when one or two processors suffice can cost up to 15% of your system throughput. I presume that unnecessarily eight vCPUs would be criminal.

Item 3: ESX Has Not Strictly Co-scheduled Since ESX 2.5

I have documented ESX's relaxation of co-scheduling previously (Co-scheduling SMP VMs in VMware ESX Server). But this statement cannot be repeated too frequently: ESX has not strictly co-scheduled virtual machines since version 2.5. This means that ESX can place vCPUs from SMP VMs individually. It is not necessary to wait for physical cores to be available for every vCPU before starting the VM. However, as Item 3 pointed out, this does not give you free license to over-size your VMs. Be frugal with your SMP VMs and assign vCPUs only when you need them.

Item 4: The Cell Construct Has Been Eliminated in ESX 4.0

In the performance best practices deck that I give at conferences I talk about the benefits of creating small virtual machines over large ones. In versions of ESX up to ESX 3.5, the scheduler used a construct called a cell that would contain and lock CPU cores. The vCPUs from a single VM could never span a cell. With a ESX 3.x's cell size of four this meant that VMs never spanned multiple four-core sockets. Consider this figure:

What this figure shows is that a four-way VM on ESX 3.5 can only be placed in two locations on this hypothetical two-socket configuration. There are 12 combinations for a two-way VM and eight for a uniprocessor VM. The scheduler has more opportunities to optimize VM placement when you provide it with smaller VMs.

In ESX 4 we have eliminated the cell lock so VMs can span multiple sockets, as item one states. Continue to think of this placement problem as a challenge to the scheduler that you can alleviate. By choosing multiple, smaller VMs you free the scheduler to pursue opportunities to optimize performance in consolidated environments.

First Success of VMware's Performance Service Offering

drummonds — Thu, 13 Aug 2009 18:23:22 GMT

Just over a week ago I had the privilege of riding along with VMware's Professional Services Organization as they piloted a possible performance offering. We are considering two possible services: one for performance troubleshooting and another for infrastructure optimization. During this trip we piloted the troubleshooting service, focusing on the customer's disappointing experience with SQL Server's performance on vSphere.

If you have read my blog entries (SQL Server Performance Problems Not Due to VMware) or heard me speak, you know that SQL performance is a major focus of my work. SQL Server is the most common source of performance discontent among our customers, yet 100% of the problems I have diagnosed were not due to vSphere. When this customer described the problem, I knew this SQL Server issue was stereotypical of my many engagements:

"We virtualized our environment nearly a year ago and and quickly determined that virtualization was not right for our SQL Servers. Performance dropped by 75% and we know this is VMware's fault because we virtualized on much newer hardware on the exact same SAN. We have since moved the SQL instance back to native."

Most professionals in the industry stop here, incorrectly bin this problem as a deficiency of virtualization, and move on with their deployments. But I know that vSphere's abilities with SQL Server are phenomenal, so I expect to make every user happy with their virtual SQL deployment. I start by challenging the assumptions and trust nothing that I have not seen for myself. Here are my first steps on the hunt for the source of the problem:

Instrument the SQL instance that has been moved back to native to profile its resource utilization. Do this by running Perfmon to collect stats on the database's memory, CPU, and disk usage.
Audit the infrastructure and document the SAN configuration. Primarily I will need RAID group and LUN configuration and an itemized list of VMDKs on each VMFS volume.
Use esxtop and vscsiStats to measure resource utilization of important VMs under peak production load.

There are about a dozen other things that I could do here, but my experience in these issues is that I can find 90% of all performance problems with just these three steps. Let me start by showing you the two RAID groups that were most important to the environment. I have greatly simplified the process of estimating these groups' performance, but the rough estimate will serve for this example:

RAID Group	Configuration	Performance Estimate
A	RAID5 using 4 15K disks	4 x 200 = 800 IOPS
B	RAID5 using 7 10K disks	7 x 150 = 1050 IOPS

We found two SQL instances in their environment that were generating significant IO: one that had been moved back to native and one that remained in a virtual machine. By using Perfmon for the native instance and vscsiStats the virtual one, we documented the following demands during a one-hour window:

SQL Instance	Peak IOPS	Average IOPS
X (physical)	1800	850
Y (virtual)	1000	400

In the customer's first implementation of the virtual infrastructure, both SQL Servers, X and Y, were placed on RAID group A. But in the native configuration SQL Server X was placed on RAID group B. This meant that the storage bandwidth of the physical configuration was approximately 1850 IOPS. In the virtual configuration the two databases shared a single 800 IOPS RAID volume.

It does not take a rocket scientist to realize that users are going to complain when a critical SQL Server instances goes from 1050 IOPS to 400. And this was not news to the VI admin on-site, either. What we found as we investigated further was that virtual disks requested by the application owners were used in unexpected and undocumented ways and frequently demanded more throughput than originally estimated. In fact, through vscsiStats analysis (Using vscsiStats for Storage Performance Analysis), my contact and I were able to identify an "unused" VMDK with moderate sequential IO that we immediately recognized as log traffic. Inspection of the application's configuration confirmed this.

Despite the explosion of VMware into the data center we remain the new kid on the block. As soon as performance suffers the first reaction is to blame the new kid. But next time you see a performance problem in your production environment, I urge you to look at the issue as a consolidation challenge, and not a virtualization problem. Follow the best practices you have been using for years and you can correct this problem without needing to call me and my colleagues to town.

Of course, if you want to fly us out for to help you correct a specific problem or optimize your design, I promise we will make it worth your while.

Performance Debate at Burton Group's Catalyst 2009

drummonds — Mon, 03 Aug 2009 17:33:34 GMT

Last week Chris Wolf moderated a debate on virtual platform performance between myself and Simon Crosby, CTO of Citrix. A recording of the debate was put online shortly after its conclusion.

Simon and I disagreed on a few issues and demonstrated different strategies in the discussion. My goal in representing the fine efforts of our performance team was to show to the audience VMware's commitment to product performance. This commitment is demonstrated through a never ending series of benchmark publications and continual product improvement. In the years since I joined VMware we have quantified ESX's ability to serve web pages (SPECweb), enable massive numbers of database transactions (TPC-C, with disclaimers), and establish industry leadership in consolidated workloads (VMmark). As we released these and dozens of other numbers, Citrix has remained silent on its own product's performance.

I was pleased that the event's format gave me the opportunity to discuss our accomplishments. My only regret was that I lacked the time to dispense with the most important of several factual inaccuracies from Simon. At one point in the discussion Simon claimed that VMmark is not run by anyone except VMware. In fact, it is closer to the truth to say that VMmark is run by everyone except VMware. A quick view of the VMmark results page will show results from every major server vendor, with no submissions from VMware.

Thanks to the Burton Group and Chris Wolf for letting me participate. It was a pleasure.

Tweaking vSphere Performance For High-Consolidation Workloads

drummonds — Thu, 25 Jun 2009 10:41:57 GMT

I was recently copied on an internal thread discussing a performance tweak for VMware vSphere. The thread discussed gains that can be derived from an adjustment to the CPU scheduler. In ESX 3.5, ESX's cell construct limited vCPU mobility between different sockets. ESX 4.0 has no such limitations and its aggressive migrations are non-optimal in some cases.

This thread details the application of this change in ESX 4 and provides some insight into its impact. This scheduler modification is going to be baked in to the first update to ESX 4.

On 4socket (or more) Dunnington (or any non-NUMA) platform, VMmark score can be further improved by enabling CoschedHandoffLLC: In console OS, it can be enabled via vsish (available from VMware*debug-tools*.rpm):

vsish -e set /config/Cpu/intOpts/CoschedHandoffLLC 1
I believe that config parameter is also tunable through VC or VI client. (haven't confirmed myself)

The degree of improvement depends on the configurations but in one case, the improvement was about 10 - 20%.

In default setting, VMmark may suffer many inter-package vcpu migrations which causes performance degradation. Setting CoschedHandoffLLC reduces the number of inter-package vcpu migrations and recovers performance loss.

The fix is disabled by default in ESX 4.0 GA but will be enabled by default in ESX 4.0 u1.

Try this out and let me know if you see a significant change on any of your workloads.

Using Perfmon For Accurate, ESX Performance Counters

drummonds — Thu, 18 Jun 2009 00:22:58 GMT

My colleague in product management, Praveen Kannan, has been working to extend Perfmon to show some ESX performance counters. This capability is automatically installed with VMware Tools on vSphere 4. But Praveen and I have made a stand-alone version available to those of you that are still on VI3. Download it here to give it a try.

To install, place the file in an appropriately-named directory on any Windows VM on VI3. Double-click the executable, which will self-extract the files into the same directory. Run "install.bat" and you're done.

Once you bring up Perfmon you'll see two new performance objects on your computer: "VM Memory" and "VM Processor". These objects contain counters exposed by ESX that accurately reflect the VM's memory and CPU usage. Here's Perfmon on my test VM after I've installed the tool.

This makes collection of host stats a breeze. Windows Management Instrumentation (WMI) programs can now easily get access to reliable host statistics. And anyone with access to Perfmon can get see their VM's resource usage. Unlike guest-based statistics, the host-statistics shown through these counters accurately reflect resource usage in the presence of virtualization overheads and time slicing of VMs.

Disclaimer:

This is a pre-release "sneak peak" version. Eventually this tool will be available for download on vmware.com and supported by VMware. But today there is no support for this tool and you're using it "as-is". Use at your own risk and do not contact VMware support for help with this release.

That's VMware's official position on this tool. But feel free to comment here with any ideas about this great new feature.

ESX Monitor Modes

drummonds — Tue, 28 Apr 2009 20:40:05 GMT

VMware has supported Intel and AMD's virtualization assist since 2006. Long before then we were using an all-software approach that we call binary translation (BT). With the benefit of years of development and optimization, BT outperformed the early versions of hardware assist. But as hardware assist evolved the use of these new features became more attractive.

Because our support for hardware assist is rich and BT is heavily optimized, the monitor can benefit from using either technology in different situations. The following tables detail the defaults in ESX 4.0, which can be changed through VM settings if desired.

Monitor Defaults with Intel Processors

VM Configuration	Core-i7 (Nehalem)	45nm Core2 with VT-x	65nm Core2 with VT-x and FlexPriority	65nm Core2 with VT-x and No FlexPriority	P4 with VT-x	EM64T without VT-x	No EM64T
FT enabled	VT-x + SPT	VT-x + SPT	VT-x + SPT	VT-x + SPT	Not runnable	Not runnable	Not runnable
64-bit guests	VT-x + EPT	VT-x + SPT	VT-x + SPT	VT-x + SPT	VT-x + SPT	Not runnable	Not runnable
VMI enabled	BT + SPT	BT + SPT	BT + SPT	BT + SPT	BT + SPT	BT + SPT	BT + SPT
OpenServer, UnixWare, OS/2	VT-x + EPT	VT-x + SPT	VT-x + SPT	VT-x + SPT	VT-x + SPT	BT + SPT	BT + SPT
32-bit Linux and 32-bit FreeBSD	VT-x + EPT	VT-x + SPT	BT + SPT (*)	BT + SPT (*)	BT + SPT (*)	BT + SPT	BT + SPT
32-bit Windows XP, Windows Vista, Windows Server 2003, Windows Server 2008	VT-x + EPT	VT-x + SPT	VT-x + SPT	BT + SPT (*)	BT + SPT (*)	BT + SPT	BT + SPT
Windows 2000, Windows NT, DOS, Windows 95, Windows 98, Netware, 32-bit Solaris	BT + SPT (*)	BT + SPT (*)	BT + SPT (*)	BT + SPT (*)	BT + SPT (*)	BT + SPT	BT + SPT
All other 32-bit guests	VT-x + EPT	VT-x + SPT	VT-x + SPT	VT-x + SPT	VT-x + SPT	BT + SPT	BT + SPT

(*) When we use BT on an Intel system with VT-x capability, we dynamically switch to VT-x if the guest enters long mode.

Monitor Defaults with AMD Processors

Configuration	Barcelona, Phenom, and Newer	AMD64 pre-Barcelona	No AMD64
FT enabled	AMD-V + SPT	Not runnable	Not runnable
64-bit guests	AMD-V + RVI	BT + SPT	Not runnable
VMI enabled	BT + SPT	BT + SPT	BT + SPT
OpenServer, UnixWare, OS/2	AMD-V + RVI	BT + SPT	BT + SPT
32-bit Linux and 32-bit FreeBSD	AMD-V + RVI	BT + SPT	BT + SPT
32-bit Windows XP, Windows Vista, Windows Server 2003, Windows Server 2008	AMD-V + RVI	BT + SPT	BT + SPT
Windows 2000, Windows NT, DOS, Windows 95, Windows 98, Netware, 32-bit Solaris	BT + SPT	BT + SPT	BT + SPT
All other 32-bit guests	AMD-V + RVI	BT + SPT	BT + SPT

Legend

VT-x: Intel's virtualization hardware assist.
EPT: Extended Page Tables. Intel's on-board, virtualization-aware memory management unit (MMU).
EM64T: Intel's 64-bit extensions to the x86 architecture.
SPT: Shadow page tables. ESX's software memory management unit (i.e., not EPT or RVI.)
BT: Binary translation. ESX's software virtualization capability (i.e., not VT or AMD-V)
AMD-V: AMD's virtualization hardware assist.
RVI: Rapid Virtualization indexing. AMD's on-board, virtualization-aware memory management unit (MMU).

My Hyper-V Video

drummonds — Wed, 10 Jun 2009 20:22:47 GMT

There's been no shortage of comments on the Hyper-V video I posted. I made a comment on this action in a VMTN blog entry. Read up and comment here or there.

Scott

Drink From the Fire Hose

drummonds — Wed, 03 Jun 2009 21:04:48 GMT

A few weeks ago our communities' administrators setup an XML aggregation of all blogs in VMware's performance community. In addition to the regular postings coming from VROOM! and me, there are several other members of our performance team that irregularly contribute new content. If you follow the aggregator and its RSS feed then you'll be notified of new performance content as it goes live.

The aggregator can be found at http://www.vmware.com/vmtn/planet/vmware/performance.xml.

Enjoy!

Storage Workload Characterization and Consolidation in Virtualized Enviornments

drummonds — Wed, 03 Jun 2009 00:21:48 GMT

Newer Processors and Virtualization Performance

drummonds — Tue, 02 Jun 2009 20:08:42 GMT

Newer processors are much more important to virtualization than physical, un-virtualized environments. The generational improvements haven't just increased the raw compute power, they've also reduced the overheads associated with virtualization. This blog entry will describe three key changes that have particularly impacted virtual performance.

Hardware Assist Is Faster

In 2008, AMD became the first CPU vendor to produce a hardware memory management unit equipped to support virtualization. They called this technology Rapid Virtualization Indexing (RVI). This year Intel did the same with Extended Page Tables (EPT) on its Xeon 5500 line. Both vendors have been providing the ability to virtualize privileged instructions since 2006, with continually improving results. Consider the following graph showing the latency of one key instruction from Intel:

This instruction, VMEXIT, is called each time the guest exits to the kernel. The graph shows its latency (delay) in completing this instruction, which represents a wait time incurred by the guest. Clearly Intel has made great strides in reducing VMEXIT's wait time from its Netburst parts (Prescott and Cedar Mill) to its Core architecture (Merom and Penryn) and on to its current generation, Core i7 (Nehalem). AMD processors have shown commensurate gains with AMD-V.

Pipelines Are Shorter

The longest pipelines in the x86 world were in Intel's Netburst processors. These processor's pipelines had twice as many stages at their counterparts at AMD and twice as many as the generation of Intel CPUs that followed. The increased pipeline length would have enabled support for 8 GHz silicon, had it arrived. Instead, silicon switching speeds hit a wall at 4 GHz and Intel (and its customers) were forced to suffer the drawbacks of large pipelines.

Large pipelines aren't necessarily a problem for desktop environments, where single threaded applications used to dominate the market. But in the enterprise, application thread counts were larger. Furthermore, consolidation in virtual environments drew thread counts even higher. With more contexts in the processor, the number of pipeline stalls and flushes increased, and performance fell.

Because of decreased efficiency of consolidated workloads on processors with long pipelines, VMware has often recommended that performance-intensive VMs be run on processors no older than 2-3 years. This excludes Intel's Netburst parts. VI3 and vSphere will do a fine job at virtualizing your less-demanding applications on any supported processors. But use newer parts for the applications that hold your highest performance expectations.

Caches Are Larger

A cache is highly effective when it fully contains the software's working set. The addition from the hypervisor of even a small about of code will change the working set and reduce cache hit rate. I've attempted to illustrate this concept with the following simplified view of the relationship between cache hit rates, application working set, and cache sizes:

This graph is based on a model that greatly simplifies working sets and the hypervisor's impact on them. Assuming that ESX increases the working set by 256 KB, this graph shows the difference in cache hit rate due to the contributions of the hypervisor. Notice that with very small caches and very small application working sets, the cache hit rate suffers greatly due to the addition of even 256 KB of virtualization support instructions. And even up to 2 MB, a 10% decrease in cache hit rate can be seen in some applications. With a 256 KB contribution by the kernel, cache hit rates do not change significantly with cache sizes of 4 MB and beyond.

In some cases a 10% improvement in cache hit rate can double application throughput. This means that a doubling of cache size can profoundly effect the performance of virtual applications as compared to native. Given ESX's small contribution to the working set, you can see why we at VMware recommend that customers run their performance-intensive workloads on CPUs with 4 MB caches or larger.

vscsiStats: Fast and Easy Disk Workload Characterization on VMware ESX Server

drummonds — Mon, 01 Jun 2009 22:53:39 GMT

Final Thoughts on the Hyper-V Video

drummonds — Tue, 12 May 2009 22:50:15 GMT

Its been about 10 days since I posted the YouTube video showing Hyper-V's stability problems in consolidated environments. I immediately received a lot of questions about the configuration that I answered to the best of my ability in my "Video on Hyper-V Crashes" blog entry. Many respondents were not surprised by stability problems with a first-generation product and some people requested more detail on this issue for further discussion. But there were too many comments to address in all.

One of the more interesting emails I received pointed out that it unreasonable to blame Hyper-V for the collapse of these very large and very busy websites. Hyper-V's stability issues would bring down individual VMs or small groups when the parent partition blue screened. I think that this is a reasonable observation, so its worth including here. I can't say that Hyper-V was responsible for the MSDN and TechNet crashes. That would be for Microsoft to say, when and if they choose to expose the issue behind the outage.

Lastly, all comments come from people that fall into one of two categories: one camp thinks the video captures are bogus and the other believes they're based on a real, reasonable, repeatable workload. I'm not going to try and move you from one camp to the other.

It is clear that a small, vocal, and surprisingly profane number of you think that I made this whole thing up. The premise of this latter group appears to be that Microsoft wouldn't make a product that a customer could crash under normal conditions. If this is your reasoning then no video, discussion or demonstration is going to change your mind. I'll let everyone else make their decisions based on Microsoft's track record and his or her experience with Microsoft products.

Update: 5/15/09

The team responsible for the research has deciced to post details: Setting the Record Straight on the Hyper-V Video

Video on Hyper-V Crashes

drummonds — Wed, 06 May 2009 22:59:51 GMT

Since I posted the YouTube video showing Hyper-V blue screens last Friday I've received a lot of comments, questions, compliments and complaints. The video and descriptive text have raised more questions than answers, so here are a few details to help fill out the story.

The workload was not technically VMmark. There are two reasons for this:
- VMmark's run rules specify that the VMs must be configured with a single virtual disk. Because this configuration can't make use of Hyper-V's paravirtualized SCSI driver, which requires a second virtual disk, the run rules were violated to make Hyper-V produce its best results.
- The vendors that provided requirements for VMmark included use of SMP Linux guests. Hyper-V's lack of support for these configurations means that it is unable to run VMmark according to the rules. Those rules were ignored by the test team and the ESX and Hyper-V tests were run with uniprocessor Linux guests so that Hyper-V was able to produce some number.
The server ran 15 tiles* when ESX was installed. So, the hardware is good.
The server successfully ran 10 tiles* when Hyper-V was installed, although at a much higher CPU utilization and lower throughput than ESX. The server seems to run Hyper-V correctly.
The 11-tile* run was tried many, many times. Hyper-V was unable to run 11 tiles without guest blue screens or the parent partition crashing and bringing down the server.

(*) As detailed in the first bullet, these aren't real "tiles". They have been dumbed down (Linux SMP) and reconfigured (extra virtual disk) to work around Hyper-V limitations.

I'm hoping to convince the people responsible for the test to shed their anonymity and come out with an official paper. I'll provide those details as soon as I can get them.

Update: 5/15/09

The team reasonable for the research has posted details of the experiment. Read more at Setting the Record Straight on the Hyper-V Video.

Virtualizing Microsoft SQL Server

drummonds — Mon, 11 May 2009 20:20:29 GMT

At VMworld Europe 2009 my engineering colleague Chethan Kumar and I presented the results of a six-month investigation into the performance of SQL Server on ESX. Tomorrow (May 12 at 09:00 PDT) we're going to offer an updated version of this session to the general public. If you have any interest in virtualized SQL Server deployments, please register and attend the presentation to discover what we learned in our investigation.

I provided some notes on that presentation in a blog entry (SQL Server Performance Problems Not Due to VMware) right after the show. But the large numbers of attendees and exceptionally high ratings encouraged me to setup this encore session. And since Chethan's research on SQL Server performance tuning has continued, we have some updates to the experimental results.

In tomorrow's webinar we will tell the story of our exploration into persistent rumors of SQL Server performance problems. The search began after VMworld 2008 when I decided to engage every customer with a complaint on SQL Server performance. At the same time Chethan investigated every possible application, operating system, and hypervisor parameter that could impact SQL performance. I talked to dozens of customers and Chethan spent hundreds of hours on this work.

This presentation will detail the results of our investigation and leave its attendees with a clear understanding SQL performance on VMware. Our conclusions are surprisingly simple and certain to help you get the most out of your virtual infrastructure.

The Role of the ESX Monitor

drummonds — Wed, 29 Apr 2009 00:57:40 GMT

There's a lot of confusion out there on VMware's support for the CPU vendors' virtualization assist technology. VMware has always led the industry with its support for hardware assist. We were the first vendor to support AMD-v and Intel VT-x in 2006, the first to support AMD RVI in 2008, and will be the first to support Intel EPT when vSphere 4 becomes publicly available. These technologies--which we call hardware assist--provide value to the part of ESX we call the monitor.

As we prepare for vSphere's general availability we're generating a lot of documentation to help people get the most out of the new version of ESX. One of my colleagues started a document that details the role of the monitor and how it flexibly uses different hardware assist technologies. I've summarized the default behavior of our monitor in several situations in ESX Monitor Modes. Of course vSphere's users will be able to override these defaults if they want to experiment with their workloads.

I wanted to include a textual summary of the role of the monitor in virtualization but found myself getting bogged down with the writing. So, I thought I'd try something new. Let me know what you think of this short video clip explaining the role of the monitor and how it might leverage hardware assist.

{youtube}http://www.youtube.com/watch?v=PYqsxIE5P-U{youtube}

Storage Performance: VMFS and Protocols

drummonds — Fri, 13 Mar 2009 00:12:37 GMT

Introduction

VMware's customers are always asking us about the storage stack. Without exception, the two most common questions about our storage system performance are:

Which storage protocol performs best?
Does VMFS scale to meet the demands of many servers and VMs?

This document will contain a few of the points needed to help understand this issue.

Storage Protocols

VMware published a paper comparing storage protocols in 2008. This paper detailed the two key characteristics of ESX's storage stack:

The hypervisor is easily able to drive the storage connection to link speed.
Configurations where protocol management happens in the HBA (Fibre Channel and HW iSCSI) are more CPU efficient.

On the first note, take the following graph, taken from page three of the paper:

Note that in this case all four test cases drive the storage to link speed. That's 2 Gb/s with the Fibre Channel HBA and 1 Gb/s with the other three. In short, if throughput is your goal, make decisions based on link speed. If you check through the rest of the paper, you'll see that response time is similar for all of the configurations, as well. But you will see slight differences in throughput in some of the protocols.

This brings us to the second point from above: less work is done by the CPU when protocol management can be off-loaded to the HBA. This means that FC and HW iSCSI HBAs will have additional CPU cycles for the VMs' work. It can also explain the slight differences in throughput in the other graphs in the paper. The efficiency results quoted in the paper are here:

The increased overheads of running software iSCSI or NFS are due to the VMkernel managing those protocols. It's worth noting that the proliferation of iSCSI in the enterprise has led VMware to spend considerable effort to improve the efficiency of SW iSCSI. Expect its efficiency to improve dramatically in the following releases.

VMFS Scalability

Many in the industry erroneously believe that VMFS won't scale as storage demands grow. Often SCSI reservations and disk locking are cited as the technical-sounding but vaguely-supported reason for this claim. It's worth sampling data from our scalable storage performance paper to debunk this myth.

This chart is a favorite in our world-wide tours as we address VMFS scalability. It's was first introduced in a VMFS scalability blog article that went live in February of 2008. It shows the results of using 64 hosts to generate a variety of traffic on a single VMFS volume. And it's a wealth of information on VMFS and storage access patterns. For instance:

The aggregate number of random writes, in cyan in the middle, maintains perfectly flat linear scalability as the host count grows from 1 to 64.
The aggregate number of random reads is initially limited by the few disks being accessed but ultimately matches the throughput of random writes as many disks come to bear to serve the large number of random reads.
The sequential read activity, which highlights the strengths of today's arrays, demonstrates the largest total throughput which only slightly drops as the array manages so many connections.
But the sequential read activity drops off dramatically as hosts are added.

This last example showing degradation in aggregate sequential read capability is an artifact of the workload that is very important to database administrators: multiple sequential reads approximate random activity. Why is this? As many hosts request more and more sequential data, the array interleaves these requests to maintain response times. This means that the sequential accesses get "shuffled" which results in a random access pattern.

In short, VMFS has no scalability problems as many hosts drive tremendous amounts of traffic to a single volume. If the data isn't convincing enough, consider the following: there are no SCSI reservations used during normal data access. This means that there are no scalability limitations as a result of virtual machine storage access. A word of caution, though: the file system is locked during administrative operations that change the metadata on the volume. This means that virtual machine creation or destruction can will result in file system locks. Perform these operations off of peak hours.

Best Practices for Performance

drummonds — Fri, 16 May 2008 21:35:33 GMT

VMware and our partners have published a wide variety of white papers on best practices for installation and configuration of enterprise applications on VI3. We'll collect additional material on this page as we build this content out. Keep checking back!

Operating Systems

Windows Server 2003
SUSE Linux Enterprise Server
Red Hat Enterprise Linux
Ubuntu

Applications

Understanding ESX Memory Management at Partner Exchange 2009

drummonds — Fri, 10 Apr 2009 00:22:15 GMT

I recently attended a practice talk for next week's Partner Exchange hosted by Kit Colbert, one of our senior engineers, who is leading a whole bunch of cool efforts around performance. I wanted to "leak" one slide that his showed us that we'll be touching up for publication. Some of you that are curious about memory counters and want a different take from Memory Performance Analysis and Monitoring may find this interesting.

Some of this stuff won't make sense outside of Kit's presentation, but let me point out a few things that may help consume the information in this incredible chart:

One of the key messages from Kit's presentation is that ESX reports memory with respect to the guest (the VM) and the host. The very top rectange shows memory stats reported for each VM. The second rectangle shows the single VM's memory stats reported by each host.
As can be seen from the above, the consumed memory in the host represents everything in the VM, minus the savings due to page sharing.
This graph doesn't yet highlight the difference between ballooned memory and swapped memory from the guest perspective. From the guest's perspective, swapped memory is much more attractive then ballooned memory, as the guest doesn't know that the swapped memory is gone. But it does see the ballooned memory as pinned. ESX is clever enough to deflate the balloon driver, if possible, when the guest starts to access swapped memory to avoid the host's swapping of guest memory.
The final rectangle shows memory of all VMs from the host's perspective. Don't pay attention to the reserved and unreserved memory; I'm told those are unnecessary distractions that will be removed.

Kit is going to be in Orlando with me next week to talk about ESX and guest memory management. He's going to explain the difficult process of recovering unused memory from guests to enable over-commitment. Be sure and see him if you're in town!

SQL Server Performance Problems Not Due to VMware

drummonds — Fri, 13 Mar 2009 17:47:57 GMT

Microsoft SQL Server runs at roughly 80% of native on VI3 in most benchmarked environments. In production environments, and under loads that model those conditions, SQL Server runs at 90-95% of native on ESX 3.5. I can say this with confidence despite a large amount of the industry's skepticism because I've spent so much time on SQL Server in the past half year. I'd like to share some of my research on the subject and observations with you.

Two weeks ago my colleague Chethan Kumar and I presented on SQL Server in Cannes, France for VMworld Europe 2009. This presentation was the culmination of six months of investigation that was started at VMworld 2008 in Las Vegas. At that event I heard so many concerns about SQL Server performance that I was resolved to identify the problems. I talked with every customer I could find that claimed that SQL ran at anything less than 70% of native. So many of these contacts claimed that they had measured SQL at 25% of native or worse, that I knew that something was going wrong.

First, let me show you a slide that Chethan presented at the show in Cannes:

Chethan spent three months investigating SQL Server to find out how much he could improve virtual performance from the "out of the box" experience. As this figure details, the sum total of performance improvements was 15%. Here's another break-down of these results:

The only option that we found in ESX to improve virtual performance was static transmit coalescing, which is documented on page four of one of our SPECweb papers. Large pages and SQL's priority boost, which are best practices provided by Microsoft for SQL Server configuration, provide the largest gains in performance.

The key messages that we communicated to our audience were that a properly running SQL Server should run at 80% of native or better. In most production cases it can run at a performance indistinguishable from native speed. And if performance is lagging, there don't exist many changes that can be made to ESX that can yield and performance gains at all.

This begs the question: "If ESX can't be tuned to double SQL performance, what is causing these reports of terrible SQL Server throughput?" The great majority of the problems are coming from mis-configured storage. But a variety of other items such as poor hardware selection or use of the wrong virtualization software contribute to the confusion, as well. I've been documenting these issues in Best Practices for SQL Server on this community and will continue to update that document as more problems are discovered.

If you have a SQL Server running un-virtualized in your environment, I'd like you to try virtualizing it again. Follow our best practices document and pay close attention to your storage configuration during deployment. I feel confident that once you've setup your environment properly, you're going to like what you see.

Best Practices for SQL Server

drummonds — Mon, 01 Dec 2008 20:45:05 GMT

Introduction

At VMworld 2008 in Las Vegas several of us in our virtual performance team met with a variety of customers to talk about Microsoft SQL Server. We already had a large base of customers running very many SQL Server DBs on our products and we wanted to collect information on the challenges posed in the process of virtualizing this critical workload. We were pleased to see that ESX Server handled SQL VMs with excellent performance. But, for many customers, the first efforts at virtualizing SQL didn't yield high-performing SQL VM. After careful investigation and many, many discussions we've started to put together the puzzle as to where SQL Server performance problems come from. This page will document these common problems, borrowing slides from our presentations on the subject.

Virtualizing SQL: The Checklist

We've talked with dozens of customers in the past months to document the issues that resulted in poor SQL performance. Happily, none of the issues were due to underlying technologies. Here is a list of issues and an explanation of the impacts. These items are roughly listed in the order of decreasing likelihood of occurrence.

Item 1: Configure Storage Correctly

Storage configuration problems are the number one cause of SQL performance issues. Usually these problems arise because the DBA requests a virtual disk of the VI admin, the VI admin places the VMDK on a LUN that may or may not meet the DBA's performance needs. For instance:

VMs' VMDK files placed on VMFS volumes without enough spindles.
Many VMDK files placed on a single VMFS volume which could use more spindles.
Database and log files placed on the same LUN which, you guessed it, could use more spindles.

This may be obvious to some, but this problem occurs again and again. The VI administrator should be aware of a few technical items that can help understand and avoid this problem:

Based on the IO demands of the DB files, a certain number of spindles should be guaranteed to this file. This means that its VMDK must be placed on a VMFS volume to accout for the SQL Server's demands and all of the other demands on that volume.
Mixing sequential activity (such as log file update) and random activity (such as database access) results in random behavior. This means that the LUN configuration in the pre-virtual physical environment may not be sufficient for the consolidated environment. This is discussed some in Storage Performance: VMFS and Protocols.
When storage isn't meeting the SQL Server's demands, the device latency or kernel latency (queueing time) will increase. Read up on these counters in Storage Performance Analysis and Monitoring.

Item 2: Use Recent Hardware

Often companies that are dipping their metaphorical toes into
virtualization want to run proof-of-concept (POC) experiments to verify
that the virtual platform can meet their performance expectations. But
its surprising how many times these experiments are run on older,
poorly-performing hardware. Presumably the shiny, new systems were in
use for production applications so only the mothballed, cobweb-covered
servers from a previous generation were available for the POC. This
causes many problems. Check out this slide from a talk on SQL Server at VMworld Europe 2009:

The slide points out a couple of things. First, the larger caches and shorter pipelines on newer Intel processors results in a considerable drops in performance overheads. Second, the latency of the VMEXIT instruction, which determines the amount of time it takes to transition from the VM to the VMkernel, has shrunk by a large amount with subsequent generations of hardware. And don't forget the other additions from Intel and AMD such as hardware assisted memory management and IO virtualization.

Item 3: Follow SQL Server Best Practices

Microsoft has kindly provided a web page of best practices for SQL Storage configuration. These be practices should still be followed when configuring your virtual SQL deployments!

Item 4: Configure VM Identically to Native and Run The Right Test

For many SQL Server POCs the goal is to measure the VM's ability to perform, with respect to the virtual platform. If this comparison is to be performed, its critical that the VM be configured identically to the physical hardware. Obviously this means that the VM should be run on the same hardware using identically configured LUNs. Its also important to ensure that the VM has the same number of vCPUs and amount of memory as the physical baseline. This means restricting the number of pCPUs and amount of memory with NUMPROC and MAXMEM, respectively, in boot.ini.

It also means that the test being applied should be understood. If a benchmark is chosen that uses a very small database, the content will be cached and the storage system won't be used. This can skew the results and produce recommendations not consistent with production deployments. Here is another slide from the same VMworld Europe 2009 presentation detailing some of what we know about the SQL Server benchmarking alternatives:

We at VMware prefer DVD Store.

Item 5: Use VMware's ESX Server

VMware's hosting products, VMware Server, VMware Workstation, and even VMware Fusion, are all capable of running SQL Server. But if the database is going to be run in production on enterprise-class hardware, use VMware's enterprise-class hypervisor: ESX Server. These products are not often confused by the initiated but rogue members of large companies often run off-the-books proof-of-concept experiments on VMware's hosted products. When they produce results they don't like, the results get spread throughout the company which can slow the virtual deployment.

Consider the following data, again from the VMworld Europe 2009 SQL Server presentation:

This information is getting a bit dated now, as it was performed years ago on ESX Server 3.0. But the point stands: before believing results claiming that "VMware cannot run SQL Server" its worth investigating the platform used to generate the results.

Item 6: Understand Memory Management and Configure Correctly

Database performance is heavily dependent on the amount of memory available. Almost without exception, providing more memory to SQL Server will improve performance. However, if that memory is coming from a host that is already over-committed or is being provided through workarounds to 32-bit limitations, performance may suffer. Here are a few keys for SQL Server memory management:

If more than 3 GB is desired, use 64-bit versions of the OS and application.
If memory is over-committed on the box, set reservations for performance-critical SQL Server VMs to guarantee that those VMs' memory isn't ballooned or swapped out.
If SQL Server's "lock pages in memory" parameter has been set, provide set the VM's reservations to the amount of memory in the VM. This setting can adversely interfere with ESX Server's balloon driver. Setting reservations will stop the balloon driver from inflating into the VM's memory space.

Item 7: Align Disk Partitions

This item is really a special but very important case of item two, follow best practices. Partition alignment can impact storage performance which can be critical to some SQL Server VMs' performance. See VMware's paper on partition alignment for more information on this.

Whitepapers

Virtual Desktop Benchmarking

wponder — Thu, 19 Feb 2009 16:03:02 GMT

This is helpful do you have an example of the number of users and the size of the network connectivity even if its a rough target? At some point there has to be a rough allocation of bandwith per users. If we do not want to upgrade our networks what experiance and how many users can we get on that network? Is it 100Kbps, 500Kbps, 1.5Mbps, 100Mbps?

Building Block Architecture for Superior Performance

drummonds — Tue, 17 Feb 2009 16:21:46 GMT

If any of you have heard me speak in the numerous events I've done in the past two years, you may have heard me detail the areas where virtualization performance can exceed native. There are scalability limitations in traditional software that make nearly every enterprise application fall short of utilizing the cores that are available to them today. As the core explosion continues, this under-utilization of processors will worsen. Here is a graph that we've been showing to illustrate that point:

In 2008 I visited VMworld Europe and showed on using multiple virtual machines on a single physical host could circumvent the limitations in today's software. In that experiment we showed that 16,000 Exchange mailboxes could be fit on a single physical server when no one had ever put more than 8,000 on in a single native instance. We called this approach designing by "building blocks" and were confident that as the core count continued to increase, we'd continue to expose more applications whose performance could be improved through virtualization.

On Thursday last week SPEC accepted VMware's submission of a SPECweb2005 result. And last night we posted an article on VROOM! detailing the experiment and providing information on the submission. This submission is an incredible first for us: not only have we shown that we can circumvent limitations in web servers, but we posted a world record performance number in the process. Of course, if any of you have seen Sreekanth Setty's presentation at VMworld on his ongoing work on SPECweb2005, this result wouldn't surprise you:

Getting a benchmark standardization body like SPEC to approve these results isn't always easy. Most of the industry remains stuck in a mode of thinking of performance as a single instance's maximum throughput. But given the scale-out capabilities of a large number of enterprise applications I'd argue that benchmarking should account for scale-out capabilities on a single box. VMware's customers follow this practice faithfully in sizing their deployments to match their needs and everyone wants to know the platform's ability to handle this use-case. SPEC's willingness to accept results showing building blocks on a single host is commendable and progressive. As more benchmarks approve submissions like these VMware will continue to be able to show record numbers.

SPECweb Record

drummonds — Tue, 17 Feb 2009 16:43:09 GMT

Last Thursday SPEC accepted VMware's submission on a record-setting SPECweb2005 score. Over the weekend we posted an article on VROOM! with some of the details on this experiment, which had been many months in the making. I wanted to write some thoughts on the SPEC results (Building Block Architecture for Superior Performance) and gauge the industry's thoughts.

Is anyone surprised by this performance record? Do any of you continue to run web servers on native, unvirtualized hardware? Any ideas where this building block approach might be able to extract more throughput out of a software architecture than the traditional scale-up model?

Scott

Standardization of Virtual Desktop Benchmarking

drummonds — Fri, 06 Feb 2009 04:00:17 GMT

For years now VMware has been providing products that enable a virtual desktop experience. Historically, this would occur in virtual desktops on our hosted products but in some cases virtualization of Citrix XenApp (formerly presentation server) could provide a large number of desktops off a single virtual machine. And more recently VMware View offers a means of hosting a large number of desktops on a single server where each is granted its own operating system instance. As the number of virtual desktops and alternatives for implementing virtual desktops has grown, the need for a benchmark that can compare the performance of these alternatives has arisen.

Desktop benchmarking is not new to the industry, as people have been using PCs for decades. But standards in virtual desktop benchmarking are non-existent. Some might argue that traditional tools, common to PCs for years, should be used. But there are several reasons why this is not true:

Pre-virtual desktop benchmarking is built to completely saturate all memory and CPU resources provided. Fully saturating CPU on a single multi-way VM, as an example, results in far fewer VMs per host than is common in VDI deployments. Fewer VMs means less work for the hypervisor's scheduler.
Existing desktop benchmarks are often throughput-based, as opposed to latency-based. Because existing tools want to differentiate between powerful processors and large amounts of memory, they're designed to pack more and longer instructions in each run than is common in virtual desktop deployments. Most desktop deployments won't run massive video renders but the response times of individual button clicks and window appearance is critical.
No existing benchmarks are aware of the peculiarities of VM-based timing. VDI benchmarks need to be aware of this by either using host timing or invoking and measuring operations from remote, non-virtual locations.

At VMworld 2008 VMware presented a VDI workload that had been constructed from a collaborative effort between all VDI teams within VMware with review and qualification by several of our partners. The first measurements on this workload came from Dell and EqualLogic and we quickly made details on its characteristics available via white paper. Key features of this workload include:

A diverse set of applications (Word, PowerPoint, Excel, Acrobat, and Internet Explorer) common to business desktop deployments.
Load generation modeled after the most common VDI deployments.
Small (less than 500 ms) operation generation and measurement.
Host-based measurement and an architecture to support remote command invocation in the next release.

As an attempt at the world's first VDI benchmark, we're very pleased with our efforts. We found that it met the unique requirements in measuring virtual desktops of all kinds. And since it was generated with large group of internal collaborators and multiple partners, it's an excellent beginning at what the industry needs to standardize this process.

But today we realize that its just a beginning. I want to encourage everyone to bring your comments to VMware via this blog or the performance forums on what you think the characteristics of a industry standard virtual desktop benchmark should be. We'll never make one benchmark that meets everyone's needs and I suspect that there are even some common needs that will require significant development resources. But I expect that with your guidance and assistance in refining this workload we'll accelerate the process of getting this benchmark in a shape that the industry can embrace

Ready Time

drummonds — Wed, 27 Aug 2008 17:50:36 GMT

Introduction

Ready time is as important as it is confusing. I'm going to collect a few thoughts on ready time in this collection point with the hopes that some of the confusion around this important part of virtual system performance can be eliminated.

Details

Stated simply, ready time is the amount of time a VM wants to run but has not be provided CPU resources on which to execute. Somewhat confusingly, ready time is reported in two different values between esxtop and VirtualCenter. In esxtop is reported in an easily-consumed percentage format. A number of 5% means the VM spent 5% of its last sample period waiting for available CPU resources. In VirtualCenter ready time is reported as a time measurement. In VC's real-time data, which produces sample values every 20,000 ms, a number of 1,000 ms is reported for a 5% ready time.

There is so much more to know about ready time that I'm not going to reproduce here. Read the whitepaper on the subject for more details. There have been no changes in the details on ready time since ESX 3.0 that make that paper out-of-date.

Interpreting Ready Time Values

The most common question we get on ready time is, "what ready time numbers constitute a problem?" While there is no easy answer to this, we can offer some guidance on the acceptable values. But before I lay that out, let me say that ready time should not be the ultimate measurement of system performance. As always, user experience and latency should be. There are some situations where user experience is horrible on a system with no load and virtually zero ready time. This could happen with a mis-configured array, as an example. And occasionally we see aggressively-consolidated hosts showing very high ready times that are meeting user needs. There are no absolutes with ready time.

But, there are a few general regions into which ready time values can be binned. Note that these ready time values are per vCPU. esxtop reports ready time for a VM once its been summed up across all vCPUs. That means that 5% ready on each of four vCPUs will be reported as 20% ready at the VM level. This is the high end of a very light amount of ready time.

Value, per vCPU	Description
r == 0%	This doesn't happen. The very presence of a hypervisor between the operating system and the hardware means that there is a non-zero ready time on all operations. But on healthy systems this number is so small that end-users don't know their workload has been virtualized. See the next section.
0% < r <= 5%	This is the "normal" region for ready time. Very small single digit numbers result in a minimal impact to user experience. If performance problems exist on the system and ready time falls into this region, your problems lie elsewhere.
5% < r <= 10%	In this region ready time is starting to be worth watching. Most systems function healthily with ready time in this region but highly sensitive measurements may be suffering.
10% < r	While some systems continue to meet expectations, double-digit ready time percentages often mean some action is required to address performance issues. See the last section for guidance.

Again, remember that VirtualCenter performance numbers must be re-calculated to percentages to find the category on the above table. But since VC reports ready time per vCPU, no special arithmetic is needed to account for the number of vCPUs in the VM (as is needed with esxtop.)

Causes and Correction

There are two general areas that can cause unnecessarily high ready times:

Overloaded hosts.
Excessive use of SMP.

Host Overloading

The most common cause of high ready time is trying to get too much work out of too little hardware. Consider the following simple case: on a hypothetical system with only one physical CPU, if two 1-way VMs are fully loaded by their users then each wants to have an entire CPU. Because only one is available, ESX will time share that resource and give each of them only 50% of the CPU. As a result, each VM will spend 50% of its time waiting for the processor. This would be reported as 50% ready time.

Often this condition is observable when ready time is high and total host CPU utilization is also very high. The only fix for this is to back off the load on the system. VMs should be migrated off or processor resources should be increased.

Excessive SMP

In ESX Server 2.5, SMP guests had to be co-scheduled to start at the exact same moment. If a 2-way VM was ready to run but only one physical core was available, the VM would not be scheduled until a second core was freed up. This would increase its ready time. In ESX Server 3.0 and later versions, relaxed co-scheduling was introduced which meant that a subset of a VM's vCPUs could be scheduled ahead of others. However, guest operating systems still require some degree of co-scheduling which means that the relaxation isn't absolute. In short, increasing vCPUs still puts some burden on the scheduler to try and co-schedule the vCPUs that can increase ready time. This is one ready why VMware advises only allocating vCPUs to VMs that are using them. Read Co-scheduling SMP VMs in VMware ESX Server for more information on co-scheduling.

This condition is manifested by hosts that have sub-optimal CPU utilization and lots of SMP VMs. A host may have a dozen 4-way VMs with each showing high ready time but only be at an aggregate 40% CPU utilization. This is a clear sign that the scheduler is spending a great deal of time managing unneeded vCPUs.

New Documentation on esxtop

khughes — Mon, 05 Jan 2009 19:11:56 GMT

I attended the esxtop session at VMworld this past year and learned quite a bit but this looks like a very well made document which I have already saved into my folder of useful documentation. I will definatly have to find time to read through it.

Thanks for some thorough documentation on esxtop!

Kyle

esxtop Performance Counters

drummonds — Fri, 16 May 2008 18:59:39 GMT

Introduction

This article contains a list of some of the performance counters provided by esxtop. This is far from exhaustive, as this list was created to answer the question: "which are the most important esxtop counters?" Recently VMware has published an exhaustive list of esxtop information on Interpreting esxtop Statistics. Check that out for more information.

CPU Counters

Counter	Description
%RDY	The percentage of time that the world or group is waiting a processor to be available to execute its workload.
%USED	The percentage of CPU that is used by that world or group.
GID	Group ID
NWLD	The number of worlds in the group. When this number is greater than one, the row can be expanded to get information on each world.

Memory Counters

Counter	Description
%ACTV	Instantaneous view of the percentage of memory pages that have been used by the VM in the previous seconds. Unlike TCHD which counts pages by following working sets, %ACTV is a more frequently updated number that is based on a sample of the entire memory pool.
%ACTVS	Slow moving average of the %ACTV counter.
%ACTVF	Fast moving average of the %ACTV counter.
MCTL?	Set to "Y" when the balloon driver is active in the guest and "N" when not.
MCTLSZ	This counter reports the amount of memory that the balloon driver is currently holding for use by other VMs.
MEMSZ	The amount of memory (in MB) allocated to the VM at the time of its creation.
NHN	The NUMA home node. This is the node on which the VM is booted. Migrations that have occurred since the VM started running would result in this VM running on another node(s).
NMIG	The number of NUMA node migrations since the VM was booted. ESX Server's scheduler should avoid NUMA migrations so if this number continues to climb during normal operations some tuning of the VMs may be required.
NRMEM	The amount of memory that exists on a remote NUMA node.
NLMEM	The amount of memory that exists on the local NUMA node.
N%L	The percentage of the VM's memory that exists on the local NUMA node. N%L = NLMEM / (NRMEM+NLMEM)
OVHD	The amount of memory used by the VMkernel to maintain and execute the VM.
SHRD	The amount of the VM's memory that is shared with other VMs.
SHRDSVD	The amount of memory that was saved due to page sharing. This number may be less than or equal to SHRD. As one VM must always claim the single copy of a shared page, one VM with a shared page will not be able to claim savings.
SWR/s	The rate at which memory is being swapped in from disk. High swap rates indicate a need for more memory in the cluster.
SWW/s	The rate at which memory is being swapped out to disk. High swap rates indicate a need for more memory in the cluster.
TCHD	The amount of memory (in MB) that has been touched (recently used) by the VM. In this case "recently" means within a minute or two.

Storage Counters

Counter	Description
ABRTS/s	The rate at which disk operations are being aborted. Abort commands are issued by the guest when the storage system has not responded within an acceptable amount of time (as defined by the guest OS or application.)
ACTV	The number of IO operations that are currently active. This represents operations for which the host is processing and can serve as a snapshot view of storage activity. When this number hovers near zero, the storage system isn't being used. If is sustains non-zero numbers, the a constant interaction with the strorage is occurring.
DAVG/cmd	The average amount of time it takes a device (HBA, array, and everything in between) to service a single request.
GAVG/cmd	The total latency seen from the VM when performing an IO operation. GAVG = DAVG+KAVG.
KAVG/cmd	The average amount of time it takes ESX Server's VMkernel to service a disk operation. Since this number represents time spent by the CPU to manage IO and processors are orders of magnitude faster than disks, it should be much, much less DAVG.
QUED	The number of IO operations that require processing but have not yet be addressed. Commands are queued and awaiting management by the kernel when the driver's active buffer is full (see ACTV). Occasionally a queue will form and result in a small, non-zero QUED number but any significant (double-digit) average of queued commands means the storage hardware is unable to keep up with the host's needs.
READS/s	The number of disk reads per second. READS/s + WRITES/s = IOPS.
WRITES/s	The number of disk writes per second. READS/s + WRITES/s = IOPS.

Network Counters

Counter	Description
%DRPRX	The percentage of packets that were dropped that was supposed to be received.
%DRPTX	The percentage of packets that were dropped for which transmission was attempted.
MbRX/s	The megabits per second that are received at the network item.
MbTX/s	The megabits per second that are transmitted from the network item.

DRS/DPM Video Live!

mikepodoherty — Fri, 07 Nov 2008 22:25:54 GMT

Scott,

Thanks. Please send it to mikepodohert at aim dot com ( work email does not have the space for the file)

DPM Power/Performance Video

drummonds — Thu, 06 Nov 2008 22:18:05 GMT

Back in September the performance team here at VMware embarked on a project to measure power savings as a result of using VI3's distributed power management (DPM). This feature, experimentally supported in VI3 will full support planned for the next release, leverages DRS to consolidate idle and lightly-loaded VMs onto as few servers as possible. Once the workload has been consolidated to the bare minimum hardware required, spare servers are powered down. The end result is the flexible performance due to automated load balancing and a halving of total power usage.

The experiment that we performed was based on a workload derived from VMmark. In fact, it was precisely the VMmark workload. But the execution of the test against a cluster of systems makes the results invalid for comparison against other systems. VMmark run rules require the test run against VMs on a single server.

We started the test with 13 tiles worth of VMs (108 VMs in all) on the DRS cluster. With all of these VMs idle, DPM consolidated them to a single host and turned off three servers. As the load was applied to the VMs at 9:00 AM and driven through an eight-hour workday, DRS and DPM powered on servers and balanced load, as needed. When the day ended at 5:00 PM, the load was again consolidated and servers were powered down. The video we shot includes power meters of the systems under test and screenshots of activity induced by DRS and DPM.

Check out the video on YouTube and let me know what you think. I'm considering recording some of the other amazing things that we're doing with our products and would love your feedback on what you'd like to see.

VMkernel Scheduler

drummonds — Tue, 27 May 2008 22:56:02 GMT

Introduction

Details on the ESX Server scheduler are commonly requested when I engage customers and partners. People want to know more about how the scheduler works, when SMP should be used, and what the deal is with SMP co-scheduling. This page will answer these questions and others as they arise in the forum or the discussion portion of this page.

Terminology and Architecture

In VMware parlance, the monitor is the part of our products that provides a virtual interface to the guest operating systems. The VMkernel is the part of our products that manages interactions with the devices, handles memory allocation, and schedules access to the CPU resources, among other things. This is shown in the following figure.

This document will provide information on one part of the VMkernel: the scheduler.

Performance Scaling and the Scheduler

It is a critical requirement for enterprise deployments that an operating system provide fast and fair access to the underlying resources. As a critical part of this design, the scheduler has undergone countless engineer-years of development to guarantee that this requirement is met. We've now released dozens of papers showing linear scaling of workloads as vCPU count is scaled up within a single VM and VM count is scaled up within a single host. Here are a few such papers that contain supporting data.

VM scaling as demonstrated by Oracle databases.
VM and vCPU scaling under IBM DB2 load.
VM and vCPU scaling with SQL Server running in the VM.

The scheduler's ability to fairly scale up to and beyond totally committed CPU resources is no accident. In fact, in a conversation I had with a QA manager I was assured that the VMkernel's scheduler would fairly distribute CPU resources to all VMs at least up to 4x CPU overcommitment. Of course, on a system with the CPU over-committed by 4x each VM will only run at 1/4 native speed but the scheduler keeps the VMs running at that performance. Not one at 1/8 speed, one at 1/10 speed, and another at 1/4 speed.

SMP and the Scheduler

As ESX Server supports uniprocessor (UP) and symmetric multiprocessor (SMP) VMs, the fair-and-fast requirement for the scheduler must be upheld in the presence of concurrently executing UP and SMP VMs. Internal testing of this requirement shows fair scheduling even in the presence of concurrently executing 1-way, 2-way, and 4-way VMs.

In fact, the ability to fairly execute under such environments is a very tricky problem for a scheduler. We've run analysis on competitors' products and found that the ability to fairly balance differently-sized VMs is something of which ESX Server alone is capable. Stay tuned in the coming months as we back this claim up with performance data.

Cell Size

One construct that assists the scheduler in optimally placing VMs on a heavily utilized system is a cell. A cell is a logical grouping of a subset of CPU cores in the system. In ESX 3 versions the cell size is equal to four. Since the cell is statically assigned to physical cores, this means that each four-core processor is in exactly one cell. When only dual-core processors are present, a cell is comprised of two sockets. The most important thing to know about cells is the following:

A VM cannot span more than one cell.

This means that four-way VMs run on only one socket at a time in systems with quad-core CPUs. For this case, the number of options presented to the scheduler is equal to the number of sockets. In future versions of ESX we plan to increase the cell size to eight. In some cases (such as systems with hexa-core CPUs) a modification of the cell size can improve performance. See KB article 1007361 for more information.

UP or SMP?

When and if to use SMP is a common question from VMware users. The simple answer to this is to only use SMP when needed. Why only use SMP when needed? There are two reasons:

SMP schedulers are less efficient than UP schedulers. This is a simple experiment that can be confirmed with trivial benchmarks like Netperf or Passmark. On UP systems (either virtual or native) the UP hardware abstraction layer (HAL) will provide marginally better results than the SMP HAL.
Even when unused, virtualization of idle vCPUs requires resources by the kernel. Memory is needed to maintain data structures and CPU resources are needed to virtualize the idle system. The amount of work needed to support an idle CPU varies greatly but usually is in the realm of 1-2% of a single CPU core.
The work required to deliver timer interrupts increases quadratically with the number of vCPUs, like RHEL5, the number of timing interrupts delivered by the VMkernel can be quite high. See Red Hat Enterprise Linux for more information on this issue.

What About Co-scheduling?

Back in the days of ESX Server 2.5, SMP VMs had to have their vCPUs co-scheduled at the same instant to begin running. Because only 2-way VMs were supported at this time, that meant that two CPU cores had to be available simultaneously to launch a 2-way VM. On a server with a total of only two cores, this meant that the VM could not be launched concurrently with any other process on the server. This would include the service console, the web interface, or any other process.

This requirement was reduced in ESX Server 3.0 through a process called relaxed co-scheduling. Effectively SMP VMs can have their vCPUs scheduled at slightly different times and idle vCPUs didn't necessarily have to be scheduled concurrently with running vCPUs. More details on this are available in the Co-scheduling SMP VMs in VMware ESX Server page.

NUMA Considerations

Support for non-uniform memory access (NUMA) architectures was introduced in ESX Server 2. This meant that the scheduler became aware that memory was not uniform across each CPU. Each CPU node had access to its own local memory and a larger pool of remote memory (which was divided as local memory for the other CPU nodes.) Memory access to local memory is much faster than remote memory so the scheduler should favor the placement of processes on nodes that held the processes' memory.

Subsequent generations of ESX Server continued to optimize for the use of NUMA memory. This included placement of vCPUs next to needed memory and startup of VMs at NUMA nodes with resources available for execution. All of this is transparently handled by the scheduler but it should be noted that the newer your version of ESX Server, the better its NUMA scheduling is.

Memory Performance Analysis and Monitoring

drummonds — Fri, 23 May 2008 23:37:14 GMT

Introduction

This document is a living, up-to-date version of the performance analysis methods whitepaper.

Host memory utilization represents the entirety of memory usage due to the VM and all tasks required by ESX Server to manage and provide control of the VMs. Using ESX Server's monitoring capabilities there is no visibility into improper usage of configuration of memory within the guest. Continue to use traditional monitoring tools in the guest to identify memory-hungry applications or shortages that lead to in-guest swapping.

Navigating esxtop

As before, bring up esxtop to inspect system specifics. Hitting the ‘m' key will display the memory counters.

Once running, the following can be observed from the esxtop report:

The header data contains host data that impacts all VMs running on the host. The physical memory row (PMEM) contains the total RAM installed on the system, the amount used by the console operating system (COS), the memory used by the kernel (VMK), and other statistics.
The next few rows contains host-level memory statistics for various ESX subsystems:
- VMKMEM: shows memory statistics for the ESX Server VMkernel
- COSMEM: displays the memory statistics as reported by the ESX Server service console.
- PSHARE: displays the ESX Server page-sharing statistics.
- SWAP: displays the ESX Server swap usage statistics.
- MEMCTL: displays the memory balloon driver statistics.

Relevant Counters

Type	VirtualCenter	esxtop	Details
Total memory size		MEMSZ	The is the amount of memory that the VM has been sized to. The VM will never get more than this but most of the time will be using far less than this amount due to sharing, ballooning, and swapping.
Memory target		SZTGT	The amount of memory that the kernel would like to provide to the VM. This number is calculated by on the guest's memory usage. When memory is over-committed, it may not equal the amount of memory that is actually provided due to ballooning and swapping.
Granted memory	mem.granted.average		The amount of memory that has been provided to the VM. Memory is not granted to the VM until it has been touched once. In the case of Linux, which does not zero out pages upon boot, a 4G VM will only be granted the small portion (100M or so) needed to run the OS until the OS or applications start to access more.
Touched memory		TCHD	The amount of memory (in MB) that has been "touched" (read from or written to) in the past X minutes.
Consumed memory	mem.consumed.average		The amount of machine memory allocated to the VM. For instance, a Linux VM might have been sized to 4G. Half of the pages may not yet have been used by the OS. Perhaps 1G of this remaining 2G can be shared. That leaves a consumed memory of only 1G.
Shared memory	mem.shared.average		Shared memory represents the entire pool of shareable memory. For instance, if two VMs each have 500M of identical memory, the shared memory is 1G.
Shared common memory	mem.sharedcommon.average		Shared common memory represents the footprint in machine memory as a result of memory sharing. For instance, if two VMs each have 500M of identical memory, the shared common memory is 500M.
Active memory	mem.active.average	%ACTV, %ACTVS, %ACTVF	The amount of memory (as a percentage of the entire host's memory) that has been used by the VM in the past sample period. %ACTVS and %ACTVF are slow and fast counters showing recent and long-term averages.
Ballon driver usage	mem.vmmemctl.average	MCTLSZ	The amount of memory claimed by the balloon driver for us in other VMs.
Swap rate		SWW/s SWR/s	The rates at which memory is swapped out (written) or in (read).
Swap Totals	mem.swapout.average, mem.swapin.average		These are cumulative amounts of swapping that has occurred since the VM was powered on. It's important to check if swapin and swapout are increasing, rather than just seeing if they are nonzero. Because if they are non-zero, it could be the result of swapping in the past, and not swapping at the present time.
NUMA migrations		NMIG	The number of NUMA migrations that have occurred since the VM's creation.
NUMA memory		NLMEM, NRMEM	The amount of the VM's memory that is on the local and remote NUMA nodes.
Overhead	mem.overhead.average	OVHD	The amount of memory required by the VMkernel to maintain and execute the VM.

Evaluate the Data

Memory analysis on an ESX Server means not just investigation of server-side statistics but also a solid understanding of the application that is running in the VM. When memory is short on the host, ballooning and swapping may be visible in esxtop, with swapping having a great impact on performance. When memory is short within the VM the guest will swap.

How much memory are the VMs actually using? While they may have been allocated large amounts of memory, its likely that the OS and applications are only using a small percentage of what the VM was assigned. Check the active and touched memory counters for accurate numbers on guest memory usage.
Is memory short in the host? Swapping (SWW/s and SWR/s) is a certain sign of this problem. Heavy use of the balloon driver may also suggest this but ballooning has a very slight impact to guest performance.
Can memory deficiencies be addressed through VM resizing? Checking memory usage of critical apps within the VMs can help inform decisions to decrease the amount of RAM provided to those VMs. Some operating systems will expand to utilize all available memory at little or no value to the application. Reducing the memory space and correcting over-sized caches frees up memory for other VMs.
Is the collection of all VMs' active memory (TCHD or %ACTV) sustaining at an amount that exceeds the total available memory? If so, then either more memory must be added to the host or VMs must be migrated to another DRS cluster.
Are the guests swapping? If the VM has been sized with too little memory then the guest OS will swap inside the VM. This will appear to ESX Server as any other disk activity but should be investigated and solved with traditional OS analysis tools.
Can NUMA migrations (NMIG) be seen on the system? NMIG reports total migrations since the VM has been powered on. If this number continues to climb then the VM is being migrated from node to node which most certainly degrades performance.
Does the amount of memory located on a remote NUMA node (NRMEM) remain at a non-zero number? This may be a sign that the VM has been sized to exceed the memory of a single NUMA node. If the VM is using more memory than fits on a single node, some of its memory is certain to be located on a remote node. Remote memory access is quite slow relative to local memory access.

Correct the System

The prescriptive advice for memory shortages is fairly simple: use less memory or buy more. The following recommendations are variations on this theme:

Verify that VMware Tools has been installed on every VM on the system and that the memory balloon driver has not been disabled. (The balloon driver is always on by default and disabled manually through text-based advanced configuration in extremely rare cases.) When provide the ability to balloon memory within the guests, ESX Server is able to take memory from VMs that are not using it and make it available to those that do need it.
Provide more memory to the DRS cluster. As total resources go up, VirtualCenter will balance VMs across the cluster so VMs that need the memory are able to get it.
Set memory reservations to minimally provide the amount of memory required of the OS and critical applications. This will allow for sustained, fast access for critical code and provide hints to VirtualCenter for optimal VM positioning across the DRS cluster.
Make sure the amount of memory used by the VMkernel to maintain the VMs is acceptable. This value, reported for each VM with the overhead counter (OVHD), is dependent on the memory size of the VM, the number of vCPUs provided to it, and whether or not it is executing a 64-bit OS. Fewer VMs on the host, fewer aggregate vCPUs, and lower precision OSes (32-bit as opposed to 64-bit) will lower this number. Reducing any of these in the cluster will free up resources for every VM in the cluster.
Size VMs on NUMA systems to guarantee that each VM's memory will fit on a single node. This means either decreasing the memory allocated to a VM or increasing the node memory size.
Size guests appropriately according to their needs. For example:
1. Depending on the access pattern of the data, databases may not benefit from the last doubling of cache size. Experiment with smaller cache sizes and see if performance drops. If not, decrease the VM's available memory so it can be used by other VMs.
2. Check the guest OS's statistics for in-guest swapping. Provide memory as its needed and pay attention to esxtop statistics to see if the additional memory provided generates a new bottleneck in the host.

Understanding Page Sharing

One cannot fully optimize an ESX Server's memory without understanding the performance implications of page sharing. VMware's page sharing algorithm was presented at EMC World 2008 as resulting in a 2% increase in CPU load. But the benefits of page sharing have been demonstrated to provide overcommitment of memory safely to 2X and beyond.

The value of page sharing can be seen int the following counters:

esxtop	VirtualCenter	Description
SHRD	memory.shared	The amount of memory in the VM that is sharable.
SHRDSVD	No equivalent.	The amount of memory saved due to page sharing.
No equivalent.	memory.sharedcommon	The size of the memory after redundant pages have been removed.

Note that missing counters can be calculated using the other two. Shared memory minus shared common memory equals shared savings.

References

The top-level Performance Monitoring and Analysis paper.

The esxtop Performance Counters index.

The Understanding VirtualCenter Performance Statistics page.

CPU Performance Analysis and Monitoring

drummonds — Fri, 23 May 2008 19:05:48 GMT

Introduction

CPU load is generated by the guest and its applications as well ESX Server as it provides a virtual interface to the hardware. While the work performed by the host does result in some increase in load, the great majority of processing is due to the applications in the VM. A solid understanding of the workload profile regardless of the virtual environment can assist CPU analysis.

Check Utilization

Invoke esxtop. By default, it should show CPU utilization but pressing ‘c' will ensure this data is being displayed. The following figure shows example data produced on a test system.

Observe the following:

The PCPU(%) line in the header shows utilization for the processor(s) by core and in total. The comma-delimited data first displayed shows core utilization followed by "used total" which averages utilization of all cores.
The LCPU(%) line shows the percentage of CPU utilization per logical CPU. The percentages for the logical CPUs belonging to a package add up to 100 percent. This line appears only if hyperthreading is present and enabled.
The CCPU(%) line shows the percentages of total CPU time as reported by the ESX Server service console. Use of any third party software, such as management agents and backup agents, inside the service console, may result in high CCPU(%) number.
There is an idle world running whose %USED entry displays the amount of CPU cycles that remain unused. If the idle world is reported at less than 100% utilization then only a fraction of one physical core remains for additional work. As this number can max out at many hundreds of percentages (100% for each core) small numbers here represent heavily loaded systems.
Check the utilization (%USED) of the interesting VMs. The VMs are reported here with the names specified at their time of creation. Like the idle row, utilization for each VM can exceed 100%. A VM that was provided two vCPUs, as an example, can max out at 200% CPU utilization.
Expand the group data for the VM that is most interesting. This is done by hitting ‘e' and then entering the group ID number (GID) for the VM. The figure below contains a CPU-expanded version for GID "30" in the previous figure. Once expanded, esxtop will expand rows and provide counter data for every world in the group. This includes:
- vmmX: For each vCPU provided to the VM, a virtual machine monitor (VMM) world is displayed. This world will perform the majority of the work required to execute and virtualize the guest code (OS, application, and hypervisor).
- vcpu-X: A vcpu-X world is created to assist the VMM world for each vCPU. Primarily this work revolves around the virtualization of the IO devices.
- mks: Mouse, keyboard, and screen interrupt servicing.
- vmware-vmx: The VMX worlds assist in maintenance and communications with other worlds and should not represent a material portion of the group utilization.

Evaluate the Data and Correct the System

The general flow for evaluation starts by considering the system's load. Is the system overloaded with too many VMs? Is the guest using all of its vCPUs and simply requires more or faster processors? Are all guests waiting for IO? For example:

Check the PCPU(%) line to see if all cores' utilization is near 100%. In this case the system is saturated. If multiple VMs are competing for the CPUs, try to reduce the VMs on the system or find other means of decreasing the load on the system. See "CPU Saturation of Host" below.
See if the PCPU(%) line shows an unequal load across processor cores with some at saturation and some remaining near idle. This would indicate applications within the VM utilizing all of the cores provided to them. Increase its vCPU count, if possible, and verify that the guest is making use of the additional cores. If the application supports horizontal scalability, you may run multiple VMs to use the additional cores. See "CPU Saturation of VM" below.
If all CPUs remain underutilized, either the application in the VM is misconfigured or the VM is waiting for IO operations to complete. See "Low CPU Utilization" below.

CPU Saturation of Host

As stated above, both the PCPU(%) and %USED counters can be used to identify systems hosts that are using all physical CPUs. It is possible, however, for the VMs on the system to be utilized nearly all of the processor cycles without actually requesting more that is available. This near-saturation case is the sign of a heavily loaded system.

A better sign of over-utilization on a host is ready time (%RDY). When any world's ready time starts to climb, that world is spending the reported percentage of its time waiting for some CPU to become available for work. Ready time above 10% is worth investigation and may be a sign of an over-utilized host. For a more detailed discussion on ready time, see Ready Time.

Host saturation is a clear sign that too much work has been loaded onto a single server. This is usually due to overly aggressive consolidation ratios. Overcommiting CPU resources in this case will only worsen the performance. Consider the following remedies:

Verify that VMware Tools has been installed on every VM on the system. In addition to many other benefits, VMware Tools provides a network driver (vmxnet) without which guest networking will be unnecessarily inefficient.
Verify that the all systems in the DRS cluster are carrying load when the server of interest is overloaded. If they aren't, increasing aggression of DRS algorithm and check VM reservations against other hosts in the cluster to ensure migrations will happen. Lastly, increase the number of servers in the DRS cluster so VMs from this server can be migrated to servers with available resources.
Increase the CPU resources available to the VMs by increasing or improving CPUs or cores on some of the systems in the DRS cluster.
Set CPU reservations for the VMs that most need the processing power to guarantee that they get the CPU cycles they need.
Ensure the newest version of ESX Server is being used. The newer versions of ESX Server provide better efficiency and CPU-saving features such as TCP segmentation offload (TSO), large memory pages, and jumbo frames.
Reduce the CPU resource footprint of running VMs. As examples:
1. Decrease disk and or network activity for applications that cache data by increasing the amount of memory provided to the VM. This may lower IO and reduce ESX Server's responsibility to virtualize the hardware.
2. Assist CPU by replacing software I/O with dedicated hardware (such as iSCSI HBAs or TCP segmentation offload NICs).
3. Reduce vCPU count for guests to only the number required to execute the workload. For instance, a single-threaded application in a 4-way guest will only benefit from a single vCPU. But the hypervisor's maintenance of the three idle vCPUs takes CPU cycles that could be used for other work.
4. For VMs created using P2V conversion, analyze the VM resources as well as the applications running inside the VM. Stop the unnecessary services that may running inside the P2V'ed VM. Also reduce the number of vCPUs and memory count to only the number required to execute the workload.

The easiest general comment for addressing CPU bottlenecks given correctly-configured VMs is to address processing power at the cluster level. If VirtualCenter reports fully utilized CPUs for all hosts in the cluster, there is little possibility of avoiding a need to increase cluster resources or decrease VM count.

One last nuance of virtual system tuning, mentioned in item 6c above, is the correct balancing of virtual CPU count. Few applications fully utilize two or more vCPUs and many VMs are often committed to a special purpose with a single application. The guest OS and the hypervisor must expend CPU cycles managing multiple vCPUs. If the applications are not using them, the system efficiency as a whole will improve by reducing vCPU count for VMs.

CPU Saturation of VM

Like host CPU saturation, VM CPU saturation can be seen when the %USED for a VM is high. Unlike host CPU saturation, the idle world may report a large amount of free computational resources and the VM's ready time (%RDY) may remain low. This behavior can be seen when a single VM utilizes all of the processors allocated to it but additional CPUs remain unused on the host. The VM's utilization of all of its vCPUs can be confirmed by expanding the VM's world on the CPU screen. Once this has been confirmed, the following options are available:

Verify that VMware Tools has been installed on every VM on the system. In addition to many other benefits, VMware Tools provides a network driver (vmxnet) without which guest networking will be unnecessarily inefficient.
If possible, increase the number of vCPUs provided to the VM. As the application in the guest is successfully using all of its vCPUs, it may continue to scale as the vCPU count is increased. Pay attention to the vmmX world for each vCPU after increasing vCPU count to verify that the VM is making use out of its newly provided resources. As detailed in item 6c in the "CPU Saturation of Host" section, the addition of vCPUs imposes an overhead on the host whether they are being used or not. So carefully assess the guest's needs to avoid unneeded vCPU count increases.
If possible, you can power on multiple VMs running the same application. This will depend upon whether how well an application supports horizontal scalable configuration. It is possible that an application may perform better when running as multiple single vCPU Vms, rather a single SMP VM.
Utilize faster processors. As processor performance is continually increasing the option of upgrading processors or migrating the VM to systems with newer processors can provide more total throughput to the VM.
Set CPU reservations for the VMs that most need the processing power to guarantee that they get the CPU cycles they need.
Decrease the work as a result of running the VM. As examples:
1. Decrease disk and or network activity for applications that cache data by increasing the amount of memory provided to the VM. This may lower IO and reduce ESX Server's responsibility to virtualize the hardware.
2. Assist CPU by replacing software I/O with dedicated hardware (such as iSCSI HBAs or TCP segmentation offload NICs).

Low CPU Utilization

Assuming performance problems have been confirmed, low CPU utilization is usually a sign of inefficiently designed datacenter architecture. The design could be flawed in an individual VM or in the connectivity between various components. The Performance Monitoring and Analysis will walk through investigation of system-level components such as memory and then system-wide components such as network and storage.

References

esxtop Performance Counters

Understanding VirtualCenter Performance Statistics

Storage Performance Analysis and Monitoring

drummonds — Tue, 27 May 2008 19:02:23 GMT

This document is a living, wiki version of the performance analysis methods whitepaper . That document will ultimately be replaced with this one.

Introduction

Storage often bounds the performance of enterprise workloads. More so than CPU or memory performance investigation, traditional means of analysis continue to be sound for storage performance in virtual deployments. This section will introduce the tools for identifying heavily-used resources and VMs that have high demands of their storage system. Traditional correction methods will then apply.

iSCSI storage using software initiators is not covered in this section. When accessed through the hypervisor's iSCSI initiator or an in-guest initiator traffic will show up on the VMkernel network or the VM's network stack. Check the Network section for more information.

Navigating esxtop

As before, esxtop is the best place to start when investigating potential performance issues. To view the disk adapter information in esxtop, hit the ‘d' key once it is running.

On ESX Server 3.5, the storage system can be displayed per VM (using ‘v') or per storage device (using ‘u'). But the same counters are displayed on each. Look at the following items:

For each of the three storage views:
- On the adapter view (‘d'), each physical HBA is displayed on a row of its own with the appropriate adapter name. This short name may be checked against the more descriptive data provided through the Virtual Infrastructure Client to identify the hardware type.
- On ESX Server 3.5's VM disk view (‘v'), each row represents a group of worlds on the ESX Server. Each VM will have its own row and rows will be displayed for the console, system, and other less-important (from a storage perspective) worlds. The groups' IDs (GID) match those on the CPU screen and can be expanded by pressing ‘e'.
- On ESX Server 3.5's disk device view (‘u'), each device is displayed on its own row.
As with the other system screens, the disk displays can have groups expanded for more detailed information:
- The HBAs listed on the adapter display can be expanded with the ‘E' key to show worlds that are using those HBAs. By finding a VM's world ID the activity due to that world can be seen on the expanded line with the matching world ID (WID) column.
- The worlds for each VM can be displayed by expanding the VM row on the VM disk view with the ‘e' key.
- The disk devices on the device display can be expanded to show usage by each world on the host.

Relevant Counters

Type	VirtualCenter	esxtop	Details
Queued Disk Commands	disk.queueLatency.average	QUED	Queued commands are queued in the kernel queue. They are awaiting an open slot in the device driver queue. A large number of queued commands means a heavily loaded storage system. See Storage Queues and Performance for information on queues.
Queue Usage	Not available	%USD	This counter tracks the percentage of the device driver queue that is in use. See Storage Queues and Performance for info on this queue.
Command Rate	disk.commands.summation	ACTV	VirtualCenter reports the number of commands that have been issued in the previous sample period. esxtop provides a live look at the number of commands that are being processed at any one time. Consider these counters a snapshot of activity. But don't consider any number here "too much" until large queues start developing.
HBA Load	Not available	LOAD	In esxtop the LOAD counter tracks how full the device queues are. Once LOAD exceeds one, commands will start to queue in the kernel. See Storage Queues and Performance for information on these queues.
Storage Device Latency	disk.deviceReadLatency disk.deviceWriteLatency	DAVG/cmd	These counters track the latencies of the physical storage hardware. This includes everything from the HBA to the platter.
Kernel Latency	disk.kernelReadLatency disk.kernelWriteLatency	KAVG/cmd	These counters track the latencies due to the kernel's command processing.
Total Storage Latency	Not available	GAVG/cmd	This is the latency that the guest sees to the storage. It is the um of the DAVG and KAVG stats.
Aborts	disk.commandsAborted.summation	ABRTS/s	These counters track SCSI aborts. Aborts generally occur because the array is taking far too long to respond to commands.

Evaluate the Data

It is important to have a solid understanding of the storage architecture and equipment before attempting to analyze performance data. Consider the following questions:

Is the host or any of the guests swapping? The guest's swap activity must be checked with traditional OS tools and the host can be checked with SWR/s and SWW/s counters detailed in Memory Performance Analysis and Monitoring.
Are commands being aborted? This is a certain sign that the storage hardware is overloaded and unable to handle the requests in a manner in line with the host's expectations. Corrective action could include hardware upgrades, storage redesign (increasing spindles on the RAID), or guest redesign.
Is there a large queue? While less dangerous than abortions, queued commands are similarly a sign that hardware upgrades or storage system redesign is necessary.
Is the array responding at expected rates? Storage vendors will provide latency statistics for their hardware that can be checked against the latency statistics in esxtop. When the latency numbers are high, the hardware could be overworked by too many servers. As examples, 2-5 ms latencies are usually a sign of a healthy storage system reading data on the array cache, 5-12 ms latencies reflecting a healthy storage architecture were data is being randomly read across the disk, and 15 ms latencies or greater possibly representing an over-utilized or misbehaving array.

Identifying a Slow Array

Its worth pausing at this moment to point out that 95% of all storage performance problems are not fixed in ESX. Believe me, I (Scott) have been called into a dozen performance escalations where poor storage performance was blamed on the hypervisor and not a single one was being caused by ESX. If you're seeing high latencies in VirtualCenter or esxtop to the storage device, its worth treating this problem as an array configuration issue. Check ESX's logs for obvious storage errors, check array stats, and make sure that there are no fabric configuration problems.

At the point of high storage latencies you shouldn't be using complex benchmarks to reproduce and solve this problem. Go with Iometer and make certain you're doing an apples-to-apples comparison against a physical system (ideally dual-booted from the ESX server under test) to make sure of what your expected, non-virtual results are. Check Storage System Performance Analysis with Iometer for information on using Iometer for problems like this.

Correct the System

Corrections for these problems can include the following:

Reduce the guests and host's need for storage.
1. Some applications such as databases can utilize system memory to cache data and avoid disk access. Check in the VMs to see if they may benefit from increased caches and provide more memory to the VM if resources permit. This may reduce the burden on the storage system.
2. Eliminate all possible swapping to reduce the burden on the storage system. First verify that the VMs have the memory they need by checking swap statistics in the guest. Provide memory if resources permit. Next, as described in the "Memory" section of this paper, eliminate host swapping.
Configure the HBAs and RAID controllers for optimal use. It may be worth reading Storage Queues and Performance for information on how disk queueing works.
1. Increase the number of outstanding disk requests for the VM by adjusting the "Disk.SchedNumReqOutstanding" parameter. For detailed instructions, check the "Equalizing Disk Access Between Virtual Machines " section in the "Fibre Channel SAN Configuration Guide". This step and the following one must both be applied for either to work.
2. Increase the queue depths for HBAs. Check the section "Setting Maximum Queue Depth for HBAs " in the "Fibre Channel SAN Configuration Guide" for detailed instructions. Note that you have to set two variables to correctly change queue depths. This step and the previous one must both be applied for either to work.
3. Make sure the appropriate caching is enabled for the disk controllers. You will need to the vendor provided tools to verify this.
If latencies are high, inspect array performance using the vendor's array tools. When too many servers simultaneously access common elements on an array the disks may have trouble keeping up. Consider array-side improvements to increase throughput.
Balance load across the physical resources that are available.
1. Spread heavily used storage across LUNs being accessed by different adapters. The presence of separate queues for each adapter can yield some efficiency improvements.
2. Use multi-pathing or multiple links in case the combined disk I/O is higher than a single HBA capacity.
3. Using VMotion, migrate IO-intensive VMs across different ESX Servers, if possible.
Upgrade hardware, if possible. Storage system performance often bottlenecks storage-intensive applications but for the very highest storage workloads (many tens of thousands of IOs per second) CPU upgrades at the ESX Server will increase the host's ability to handle IO.

Resources

Top-level performance analysis page: Performance Monitoring and Analysis

VirtualCenter performance counters: Understanding VirtualCenter Performance Statistics

esxtop performance counters: esxtop Performance Counters

Fibre Channel SAN Configuration Guide

Storage Queues and Performance

Large Memory Pages

drummonds — Fri, 08 Aug 2008 16:02:31 GMT

In ESX Server 3.5 VMware introduced support for large memory pages in the guest. Large memory pages, an architecteral feature available in x86 microprocessors for decades, can be used to improve performance on workloads that make use of them. With CPU, hypervisor, OS, and application support, throughputs can go up and CPU utilization can go down. Since applications such as Oracle databases and Java have been using large pages on Linux and Windows for years, the introduction of this support on ESX Server allows for increased gains in performance over previous virtual installs. VMware is currently the only virtualization vendor to support large pages.

VMware's support for large memory pages is detailed in the Large Page Performance performance study. That paper includes data on throughput gains in SPECjbb. The results are duplicated here:

At LinuxWorld 2008 VMware presented further data on the value of large memory pages with Oracle databases. Here is a chart showing those gains with VMware binary translation (BT):

Data also shared at LinuxWorld on the value of large memory pages with AMD Rapid Virtualization Indexing (RVI; formerly called NPT):

The gains derived from the presence of large pages in these Oracle/Swingbench results are atypical. Large pages have been documented in numerous locations to provide benefits between 5-20% on most database applications. The increases shown here (of up to 350%) are due to the specialized configuration which is less about demonstrating real-world application performance and more about stressing the underlying configuration to uncover strengths and weaknesses.

Storage System Performance Analysis with Iometer

drummonds — Thu, 27 Mar 2008 18:02:53 GMT

Introduction

Iometer is an open source tool originally developed by Intel that remains the simplest and best means of generating load on a system for performance analysis. Because of minute fluctuations in the in-guest timers many in-guest benchmarks are unable to produce accurate results. We at VMware have used Iometer for years and find its results to be accurate in most situations. But see the disclaimer below for a word of caution.

Figure 1. Iometer with disk targets tab selected.

Figure 1 shows the topology and disk target tab for a particular VM. As the manager only has one host visible beneath it and only one worker is present on another host, you can tell that this is a single server with only one processor. Had Iometer been started on a 4-way box, four workers would be visible. As the UI remains operational, dynamo workers can be connected from other hosts to drive load across multiple VMs that may be on multiple servers.

Configuring the Test

For all Iometer tests, under "Disk Targets" always increase the "# of Outstanding I/Os" per target. When left at the default value of ‘1', a relative low load will be placed on the array. By increasing this number some the OS will queue up multiple requests and really saturate the storage. The ideal number of outstanding IOs can be determined by running the test multiple times and increasing this number all the while. At some point IOPS will stop increasing. Generally an increase in return diminishes around 16 IOs/target but certainly more than 32 IOs/target will have no value due to the default queue depth in ESX. See Storage Queues and Performance for more information on queues. In Figure 1 you can see that "# of Outstanding I/Os defaults to 1."

When choosing the system to test in the topology frame, the "Disk Targets" tab will provide options as to the storage target. The options here include formatted disks (yellow) or unformatted disks (blue). In the former case Iometer address the storage through the OS's file system (FS). In the latter, direct calls are made to the hardware without using a FS. Storage specialists are usually more interested in just the hardware so evaluation of unformatted LUNs (blue) is preferable. There is some cost of virtualizing the OS's interface to the disk through the FS so formatting the disk with the correct FS and testing the yellow target can be helpful. Figure 1 shows two testable drives: the yellow-iconed C drive on which the OS was installed and the blue-iconed unformatted drive that is preferable for benchmarking.

Always make certain that the "maximum disk size" in the "Disk Targets" tab is larger then the available memory! For instance, when testing a formatted disk, setting the maximum size to 200,000 sectors (or 100 MB) could be cached by the guest OSes in a VM provided 1 GB of RAM. In this case all Iometer calls to storage will be intercepted and cached by the guest, host, or storage cache. Setting the disk maximum disk size to a number at least four times greater than the memory available in the largest cache will avoid caching.

Under the "Access Specifications" tab, choose a workload that matches the most interesting profile. Real workloads that are dominated by database performance often randomly read and write small, fixed-size block IO. SQL Server on Windows, for instance, uses 16K blocks, 66% read (which implies 34% write), and 100% random (thus 0% sequential). Exchange 2007 uses a similar profile but with an 8K block size. Oracle on Linux has flexibility to use the block size set when the file system was created. Depending on the DB specialists needs, this can range from 2K to 64K but will again be random with a 2:1 read-to-write ratio. Note: you can approximate this Linux performance on a Windows guest but do not run Iometer on Linux (see "Iometer On Linux" below.)

Application	Block Size	Randomness	Read/write Ratio
Exchange 2003	4K	80%	60% read (40% write)
Exchange 2007	8K	80%	55% read (45% write)
SQL Server	16K, 64K	100%	66% read (34% write)

Table 1. Example Iometer profiles.

Note the number of workers that has been specified in under the manager. This will default to one worker (thread) for each physical or virtual processor on the system. In the event that Iometer is being used to compare native to virtual performance, make sure that the worker numbers match! For instance, the work count will be one on a UP VM but four for the same native measurements if the system is quad-core. Correct the native worker count by detaching workers.

Before invoking the test, be aware of the potential impacts of data alignment. VMware has demonstrated substantial differences in performance based on alignment of data on storage arrays in our partition alignment paper. Make sure that partitions and virtual disks have been created by Virtual Center to guarantee that the partitions and files are properly aligned.

Disclaimer (or: When Not To Trust Iometer)

As discussed in Time-based Measurements in Virtual Machines, the hypervisor may introduce some inaccuracy to in-guest time measurement. The likelihood of measurement error increases as the server's load increases. Generally servers that are using less than 30% of their available CPU resources can be trusted. In the event that a large VM on a small server is driving all CPUs to high utilization, Iometer results may suffer from some inaccuracy. This is rarely the case with Iometer runs.

Analyzing the Results

As mentioned above, the results provided by Iometer tend to be trustworthy. But for unimpeachable results use the analysis techniques provide by VMware. See the Performance Monitoring and Analysis that's already been dedicated to this subject.

Storage Queues and Performance

drummonds — Thu, 24 Jul 2008 00:51:16 GMT

Introduction

VMware recently published a paper titled Scalable Storage Performance that delivered a wealth of information on storage with respect to the ESX Server architecture. This paper contains details about the storage queues that are a mystery to many of VMware's customers and partners. I wanted to start a wiki article on some aspects of this paper that may be interesting to storage enthusiasts and performance freaks.

Two Important Queues

Let's use the following figure as a starting point for this discussion.

For the purposes of this paper, I'm going to call the two different queue types the "kernel queue" and the "device driver queue". The device driver queue is specified in the device itself and has historically been configured through Linux-like module commands in the console operating system. More on that in "Changing Queue Depth" below. The kernel queue should be thought of as infinitely long, for all practical purposes. Any time the device driver queue gets full, commands to the storage will queue up in the kernel.

Note that each LUN gets its own queue. This means that when you change the queue depth in the device driver, you're changing the queue depths for many queues. The underlying device (HBA) is going to have a hard limit on the number of active commands it will allow at one time. This should be considered when setting queue depth. If your HBA can support only 2,000 active commands but it is addressing 40 LUNs, a specified queue depth of 64 won't allow that many commands to all LUNs. This being due to the fact that 64*40 = 2,560--which is more than the 2,000 maximum commands. In practice this is rarely a concern, though, as rarely are so many LUNs being simultaneously addressed through a single HBA and so many outstanding commands being issued to these LUNs.

Device Driver Queue Function

The device driver queue is used for a low-level interaction with the storage device. It controls how many active, or "in flight", commands there can be at any one time. This is effectively the concurrency of the storage stack. Set the device queue to 1 and each storage command becomes sequential: each one must complete before the next starts.

But if the device queue is left at its default of 32, as an example, 32 commands will be concurrently processed by the storage system. All 32 will be shipped off to the storage device by the kernel and new commands are queued when completions arrive.

Kernel Queue Function

The kernel queue can be thought of as kind of an overflow queue for the device driver queues. But it's not just an overflow queue. ESX Server contains all kinds of cool optimizations to get the most out of your storage. And these features apply to commands in the kernel queue only. Here are some examples of features provided to commands queued at the kernel queues:

Multi-pathing for failover and load balancing.
Prioritization of storage activities based on VM and cluster shares.
Optimizations to improve efficiency for long sequential operations.

There are others, as well.

Impacts of Queue Depths

So, increasing queue depths in the device driver can greatly improve the performance of the storage at the device level. Decreasing the device driver queue will result in increases in usage of the kernel queues. This decreases the device efficiency, but introduces opportunities for optimizations across multiple VMs and devices. So, what's the right ratio of these two depths? We think that the sweet spot lies with a depth 32 device driver queue. That's why we've set 32 as the default device driver queue length.

But your configuration and workloads may benefit from a change to this default queue depth. I'll refer you to the aforementioned storage paper for information on when you might want to change the driver queue depth. I'll just point out a couple of broad observations here:

With fewer, very high IO VMs on a host, larger queues at the device driver will improve performance.
As the VM count grows and storage performance features--like shares, load balancing, failover, etc.--become more important, the default queue depth is best.
With too many servers each having too large of device queues, your storage array could easily be overloaded and see its performance suffer.

Improving Storage Performance

Now that we've covered how storage queuing works, you may be wondering how you can monkey around with these queue sizes for optimal performance. I can tell you as someone that has been involved with many, many performance analysis projects that changing queue size is rarely a fix to an acute storage performance problem. You should first go through the analysis techniques in Storage Performance Analysis and Monitoring. That may or may not lead to changing queue depths.

But, in the event that you do end up changing queue depths...

Changing Queue Depth

We have a helpful knowledge base article that describes the process of changing the device driver queue. Unfortunately, as of today (7/24/08) this document only describes how to change queues through the console operating system. No information is provided for ESXi. I've contacted the KB owner and will have that document updated ASAP.

Network Performance Analysis and Monitoring

drummonds — Tue, 27 May 2008 21:05:20 GMT

Introduction

This page is a living, up-to-date version of the performance analysis methods whitepaper.

Check Utilization

esxtop will provide network information on the network screen which is displayed with the ‘n' key.

The following properties of this screen are worth particular attention:

Each row represents one of several relevant network items on the server: a physical NIC (vmnicX), a virtual switch interface (vswifX), a VM (contains the VM name), the VMkernel network stack (vmk-tcpip-A.B.C.D), and others.
The network items are organized by the virtual switch to which they are attached. The virtual switch name is listed under the DNAME column.
Network traffic on the hypervisor's iSCSI initiator will show up on the VMkernel network row which will contain the name "vmk-tcpip-A.B.C.D", where A.B.C.D is the VMkernel IP address.
Network traffic on an iSCSI initiators that were configured in the guest will show up on the vNIC displayed using the VM's name on the network panel.
Total throughput for each item can be observed by summing the total transmitted data (MbTX/s) and received data (MbRX/s) for each item. As the physical hardware becomes saturated transmitted and received packets will start to be dropped (%DRPTX and %DRPRX, respectively) which, depending on protocol, may result in a retransmission at a later time.

Evaluate the Data

Does the physical NIC's reported speed and duplex setting match the expectation of the hardware? Hardware connectivity issues may result in a NIC autonegotiating to a lower speed or half duplex mode.
Is there a significant load on the appropriate network items? For instance, is a network-intensive load in a guest actually generating the network activity on its vNIC that is expected? Are storage-intensive loads generating traffic on the vNIC or vmkNIC when the hypervisor or guest initiators are used?
Verify that the network traffic is flowing on appropriate NICs. A typical ESX host may have network traffic generated by VMs, network traffic from iSCSI protocol, VMotion related network traffic and service console associated network activity. It is recommended to have to separate NICs to handle these different network packets.
During periods of saturation, is the total throughput (MbTX/s summed with MbRX/s) matching expectations? Either the guest or the other end of the communication link may be throttling the performance.
Are packets being dropped? When overworked the hardware will refuse packets which get reported as dropped transmitted (%DRPTX) and received (%DRPRX) packets.

Correct the System

Make sure that the hardware is configured to run at its maximum capability. This means verifying that 1 Gb NICs are not autonegotiating down to 100 Mb/s for having been connected to an older switch. Similarly, ensure that NICs are running in full duplex mode.
When network throughput seems lower than expected, apply traditional network diagnosis techniques to investigate every link in the connection. Low throughput at the ESX Server is not necessarily due to server configuration.
Verify that VMware Tools is installed on the guests and TSO, Jumbo Frames, and 10 Gb Ethernet are enabled, where possible.
Bond multiple physical NICs to virtual switches with high utilization.
Provide separate virtual switches their own physical NICs and separate network-intensive VMs on their own vSwitches.
If VMs running on the same ESX Server communicate with each other, connect them to a dedicated virtual switch so that all network transfers occur in memory and not packets are shipped over the wire.

VMFS and RDM Considerations

ESX Server supports the mapping of physical LUNs to virtual machines via a method called raw device mapping (RDM). RDM eliminates VMFS from the stack which is incorrectly believed to be a source of performance problems. Removing VMFS reduces the total number of addressable LUNs, eliminates the ability to perform storage migrations (storage VMotion), and greatly increases the effort required for simplified maintenance activities provided by site recovery manager. And the performance benefits derived from the removal of VMFS are negligible.

See the performance characteristics of VMFS and RDM whitepaper for more information on this subject.

Resources

The top-level performance analysis page: Performance Monitoring and Analysis

VirtualCenter performance counters: Understanding VirtualCenter Performance Statistics

esxtop performance counters: esxtop Performance Counters

Guest-based Performance Measurement

drummonds — Tue, 03 Jun 2008 16:50:47 GMT

Because VMware products provide a virtual interface to the hardware, traditional performance instrumentation that is based on measuring hardware resources may not be accurate. As a result, Perfmon (in Windows) and top (in UNIX variants) will not provide accurate measurements of CPU utilization. The problems seen as a result of usage of traditional in-guest performance measurements come from three areas:

They are unaware of work being performance by the virtualization software, they will not have complete information on the resources being used by the virtualization software. This includes memory management, scheduling, and other support processes like the service console in ESX.
The way in which guest OSes account time is different and ineffective in a virtual machine.
Their visibility into available CPU resources is based on the fraction of the CPU that they have been provided by the virtualization software.

Items two and three are covered in more detail in a KB on the subject. Performance analysis on virtual deployments should always use host-based tools. On ESX Server, this means esxtop or VirtualCenter.

Understanding VirtualCenter Performance Statistics

drummonds — Thu, 15 May 2008 19:15:58 GMT

Introduction

VirtualCenter (VC) is the entry point for virtual platform management but is less frequently used for performance analysis than esxtop. On the surface, VC is insufficient for performance analysis. But this is not necessarily the case. The VirtualCenter performance counter collection is reduced by default to minimize the data maintained by VC's database. The performance counters maintained by VC can be modified and detailed analysis can be performed based on those counters. This document will provide details necessary for understanding and enabling VC's performance monitoring capabilities.

Refer to the Performance Monitoring and Analysis for information on using these counters.

VirtualCenter Statistic Archival

Our stats infrastructure has a lot of counters but our documentation has traditionally been quite thin in terms of descriptions. I got so sick of asking what stats are available at what stats level that I decided to start this page. Obviously it needs to be made more readable, but hopefully it is a start.

Remember that stats in VC are generally organized into 2 archival categories:

Not archived: these are the "real-time" (past-hour) stats, which are refreshed every 20 seconds, and are displayed for the past hour in the VI client. These stats are not stored in the database.
Archived stats. These stats are aggregations (rollups) of the real-time stats. They are aggregated at different sampling intervals and stored in the database. We follow the MRTG standard.
- Past day: past day stats take the real-time stats and roll them up so that there is 1 data point for every 5 minutes. Thus, there are 12 data points per hour and 288 per day.
- Past week: past week stats take the past day stats and roll them up so that there is 1 data point for every 30 minutes. Thus, there are 48 data points per day and 336 per week.
- Past month: past month stats take the past week stats and roll them up so there is 1 data point per 2 hours. Thus, there are 12 data points per day and 360 per month (30-day month).
- Past year: past year stats take the past month stats and roll them up so there is 1 data point per day. Thus, there are 365 data points per year.

The basic flow is this: an ESX host stores statistics at 20s granularity for a period of 1 hour. Therefore, using the Host Client one can view the stats for a host/VM for the past-hour, or one can view those stats using the VI client attached to VirtualCenter. ESX will also aggregate the statistics into past-day statistics and store them for up to 1 day. These past-day statistics are sent to VC periodically and then stored in the database. The database is responsible for periodically taking these past-day stats and rolling them up into 30-minute weekly stats, and then doing the same for converting the weekly stats to monthly stats, etc. Because past-day, past-week, and past-month stats are stored in the database, I call them "archived" stats.

VirtualCenter Statistics Level

Statistics level is a means of organizing statistics for archiving purposes. Its worth noting that only stats levels one and two are useful for deployment performance monitoring and analysis. Levels three and four provide granularity and visibility that is useful only for developers.

The concept of "stats level" applies only to the archived stats: we only store a stat in the database if we are at the appropriate stats level for that particular statistic. Non-archived stats are unaffected by stats level. In other words, every metric listed below is collected at 20s granularity and stored on the ESX host for 1 hour. However, unless VC is set to the stats level appropriate to that statistic, we will not store the data in the database or rollup the stat into a past-day stat on the ESX host. You can specify the stats level independently for each of the archiving interval. In other words, you might want to store level 4 stats for up to 1 day, but level 3 stats for 1 week.

In practice, we use stats level to vary the level of detail for statistics that are archived. At stats level 1, we have pretty coarse-grained stats, while stats level 4 contains very detailed statistics, and also includes statistics for various instances (e.g., for each NIC of a VM).

There are 3 important calls that I often use for stats (please refer to the SDK documentation for more information):

QueryStatsByLevel: this tells you what stats are available at what stats level. This is what I used to generate the tables below.
QueryAvailableMetrics: this tells you what stats are available for a given entity during a specified time period.
QueryPerf: this call takes a QuerySpec as an input and collects the stats for the specified entity over the specified time interval.

Let me give a concrete example of stats level.

Suppose I want to know the value of mem.consumed.maximum for a given VM. This is the maximum amount of machine memory allocated to a VM (including overhead memory) over a specified interval. As shown below, this is a "level 4" statistic. This means that if I've set the stats level to 4 for past-day stats and then formulate a QuerySpec that asks for the value of this data 20 minutes ago at "past-day" granularity (i.e., at 5-minute granularity), then I will get a value. If the stats level is 2 for past-day (5-minute granularity) statistics, however, then such a query will not return a value, because it is level-4 stat and only level-2/level-1 stats are being stored at 5-minute granularity. In contrast, even if the stats level is 1, then if I formulate a QuerySpec with 20s (i.e., "real-time" or "past hour") as the interval of collection, I will get this value, because this data is stored for up to one hour at 20s granularity no matter what the stats level.

Update Interval

Understanding the update interval is a key component to understanding the performance statistics. The Virtual Infrastructure Client (VIC) displays live stats at a 20s update frequency. Archived stats are archived at their archive frequency. This is key to understanding the relative amounts of data presented by VC.

For instance, a ready time of 1,000ms in the VIC's live stats graph translates into 5% ready time (1,000 / 20,000.) The same amount of ready time in a five minute archival frequency would be 15,000 ms.

Counter Index

For a list of all counters, see the vCenter Performance Counters page.

Linux Timer Rate

drummonds — Fri, 14 Mar 2008 16:30:54 GMT

Introduction

A hardware timer is used by modern systems for a variety of fine-grained operations at the operating system level. VMware's virtualization platforms virtualize this timer in the kernel. Because the virtual timer provided to the VM is actually software, it is subject to the same resource restrictions as other processes. The busier the system the more the timer execution must contend with other hypervisor activities. There are two implications of this:

When the system is very busy, the software timer may not execute as regularly and virtual time may fall behind.
Depending on how frequently the OS wishes to be interrupted by the timer, the hypervisor must do different amount of work.

From VMware KB article #1420.

Linux guest operating systems keep time by counting timer interrupts. Unpatched 2.4 and earlier kernels program the virtual system timer to request clock interrupts at 100Hz (100 interrupts per second). 2.6 kernels, on the other hand, request interrupts at 1000Hz - ten times as often. Some 2.4 kernels modified by distribution vendors to contain 2.6 features also request 1000Hz interrupts, or in some cases, interrupts at other rates, such as 512Hz.
Furthermore, an SMP-capable Linux kernel requests additional timer interrupts from the virtual local APIC timer. An SMP-capable kernel running on a one-CPU system generates twice as many total timer interrupts as the corresponding UP kernel, while such a kernel running on a two-CPU system requests three times as many. In general, an SMP-capable kernel running on <n> CPUs requests <n+1> times as many interrupts per second as a UP kernel. For example, an unmodified 2.6 Linux kernel running on a two-CPU virtual machine requests a total of 3000 clock interrupts per second.
When a guest asks for more than 1000 clock interrupts per second, it can be difficult for the virtual machine to keep up, especially if other applications are running on the host at the same time. This can cause the clock in the guest operating system to fall so far behind real time that it is unable to catch up. The overhead of delivering so many virtual clock interrupts can also hurt guest performance and increase host CPU consumption.

The amount of work required to manage the virtual timer is greatest with Red Hat Enterprise Linux 5 (RHEL5) SMP systems, which use a clock frequency of 1000 Hz and suffer from a multiplicative amount of work due to SMP support. For instance, the following table of timer interrupts was created on a 1000 Hz RHEL VM:

vCPU Count	Interrupts/sec
2	6,000
4	20,000
8	72,000

So, the amount of work that needs to be done by the hypervisor increases dramatically with the addition of vCPUs. In addition, decreasing the timer interrupt rate greatly decreases the work that needs to be done by the VMkernel to virtualize the timer. In RHEL 5.1, a Linux kernel that enables reducing the timer rate was included. By adding the parameter "divider=10" to the boot parameters, the amount of work required of the VMkernel to virtualize the timer goes down by an order of magnitude.

Performance Monitoring and Analysis

drummonds — Wed, 26 Mar 2008 20:12:44 GMT

Introduction

A common question that crosses my desk is, "how do I analyze and correct performance?" There are a variety of techniques for doing this and many sources of information spread around the interweb, so I'm going to collect a few thoughts here.

Monitoring

Guest-based performance monitoring is an inaccurate and unhelpful means of evaluating performance in virtual deployments. See Guest-based Performance Measurement for more inforamtion. Monitoring and analysis of VMware ESX Server should be performed with esxtop and VirtualCenter.

esxtop

esxtop is the tried-and-true means of collecting every performance stat needed and making it available in a way that is conducive to analysis. The best source of information on launching esxtop can be found in the Resource Management Guide (page 159). It's worth nothing that the directions in that guide came from ESX Server 3.0.1. Since then, the developers have been kind enough to simplify the process of including all performance counters. This is done with the "-a" switch. So,

esxtop -a -b > analysis.csv

runs esxtop in batch mode and prints all performance counters. Let me give you the quick "do-s" and "don't-s" of esxtop:

With esxtop on ESX Server 3.5 and newer, always include the "-a" option to display all counters.
With esxtop on versions of ESX Server prior to 3.5 always follow the resource management guide to enable all counters. The storage latency statistics, for instance, are not displayed by default!
Always start your VMs before running esxtop in batch mode. If you start them after starting "esxtop -b", then esxtop will only produce VM data based on the VMs that were running at the time of its start.

VirtualCenter

VirtualCenter doesn't provide all of the performance counters you might need to analyze performance. But, it provides more than you might think! The default setup for VirtualCenter performance counter collection is fairly minimal. But this can be expanded by reconfiguring VC's performance counter collection. This is done as follows:

From the VI client, Administration->VirtualCenter Management Server Configuration...
On the left, click "Statistics".
Now increase the stats level as you see fit.

More information on VC performance counters and archival can be found in the Understanding VirtualCenter Performance Statistics wiki article. A few things are worth considering before going crazy with this:

You probably will never need level four stats. Those are mainly for debugging.
Use the DB size estimator after you monkey around with these levels. The DBs can get quite large and you want to know this before it happens and your VC system performance suffers.

Analysis

Performance analysis techniques were previously detailed in the performance analysis whitepaper. That has been migrated to this wiki and new material will appear here.

The following pages will provide guidance for identifying and correcting performance problems on ESX-based systems. We recommend following each of these pages in the order they're presented here.

Check and correct CPU utilization: CPU Performance Analysis and Monitoring
Identify memory bottlenecks and remove: Memory Performance Analysis and Monitoring
Characterize storage performance and correct: Storage Performance Analysis and Monitoring
Understand and improve the network utilization profile: Network Performance Analysis and Monitoring

Within each of these pages are techniques for using counters from VirtualCenter and esxtop. Information on those counters is provided in vCenter Performance Counters and esxtop Performance Counters, respectively.

Also, note that, while useless in collecting performance data, Perfmon can help with analysis of large esxtop output files. See Using Perfmon for esxtop-based Performance Analysis for more information.

Windows Server 2003

drummonds — Fri, 16 May 2008 22:04:43 GMT

Please be patient while we build out this content. If you have ideas for additions, mail me!

Short list:

Always install Service Pack 2. Microsoft changed the interaction with the APIC to improve efficiency on virtual platforms.
Read up on idle loop behavior for Service Pack 1 .

Using Perfmon for esxtop-based Performance Analysis

drummonds — Fri, 09 May 2008 16:48:20 GMT

Introduction

By now everyone knows that guest performance metrics are not reliably in virtual machines (Guest-based Performance Measurement). Slight fluctuations in very small time measurements and a lack of knowledge of the hypervisor's activities produces misleading numbers. Windows' performance monitoring tool, Perfmon, suffers from the same problems. However, Perfmon remains highly valuable for virtual machine performance analysis.

The performance analysis methods document that was published early in 2008 provided tips on esxtop-based performance analysis. To record esxtop data, page 159 of the resource management guide describes running esxtop in batch mode. But on ESX Server 3.5, running esxtop in batch mode with all counters enabled results in an incredibly large CSV file that cannot easily be parsed. But Perfmon can help with this process.

esxtop was constructed so that its CSV-formatted batch output file can be readily consumed by Perfmon. This means that Perfrmon can be used for two key activities:

Quickly analyzing results.
Generating smaller CSV files of a subset of the data that can be more easily consumed by other analysis tools (such as Microsoft Excel.)

Quick Results Analysis

esxtop's batch output CSV file can be opened and viewed in Perfmon with the following steps:

Transfer the CSV file to a Windows sytem.
Launch Perfmon (Run: "perfmon".)
Right click on the graph and select "Properties..." from the drop-down menu.
Select the "Source" tab.
Select the "Log files:" radio button from the "Data source" section.
Click the "Add..." button.
Browse to and select the CSV file created by esxtop and click "OK".
Click the "Apply" button.
Optionally: reduce the range of time over which the data will be displayed by using the sliders under the "Time Range" button.
Click "OK".

Once the data has been loaded into Perfmon you may select ESX performance counters and display them using Perfmon's graphing system just like normal Windows performance counters. Refer to the Perfmon documentation for instructions on doing this.

Pruning CSV File Data

Because so many people use Microsoft Excel to analyze performance data but its row and column limitations are quickly exceeded when generating large esxtop batch files, a means of removing unneeded data will assist analysis. Once Perfrmon has been loaded with the esxtop data, it can be used to generate smaller CSV files that can be easily consumed by Excel. Follow these steps:

Start Perfmon and load the esxtop batch data as described above.
Have the Perfmon graph display the data of interest (that you'd like to import into Excel.)
On the graph, right click "Save Data As..."
In the popup box, select "CSV" as type and save the file.

This will save only the counters that were displayed in the graph. By iteratively selecting subsets of counters and saving off individual CSV files it becomes possible to quickly build performance graphs in Excel using esxtop batch output data.

Benchmarking

drummonds — Wed, 28 May 2008 19:12:13 GMT

Introduction

Many VMware users wish to perform analysis on their own virtual deployments. This page will collect information on setting up and executing your own tests to analyze performance.

General Best Practices

Always measure performance from a native (non-virtual) system. Be aware that time measurements in virtual machines can be subject to minute fluctuations. Many benchmarks produce results by summing times from large number of small operations so these small inaccuracies can be compiled to produce a large error. See Time-based Measurements in Virtual Machines for more information on this subject. The only way to guarantee correct measurement is to run the measurement tool on a native system. This is easy for client-server test architectures but may require clever architecture for in-guest testing.

Always ensure apples-to-apples comparison. Make sure that the benchmark or application under test are both constrained by the same resources. For instance, if the virtual machine was configured with 512M of RAM and two virtual CPUs, restrict the native system to the same resources if a virtual-to-native comparison is desired.

Collect accurate host-based performance statistics. Guest OS performance metrics (such as CPU utilization) are not accurate. Use VirtualCenter or esxtop to collect accurate performance counters during the test. See the Performance Monitoring and Analysis for more information on analysis.

Application Benchmarking

Microsoft Exchange.

Microsoft SQL Server.

Subsystem Benchmarking

Storage

Internally at VMware we've used Iometer for a variety of storage analyses. See the Storage System Performance Analysis with Iometer for more information.

Time-based Measurements in Virtual Machines

drummonds — Thu, 29 May 2008 18:05:22 GMT

Introduction

All benchmarking relies on accurate time keeping so production can be measured with respect to the passage of time. Because hosted and hypervisor products virtualize the hardware timer, minute fluctuations in guest time keeping can occur. Details are provided on this in many locations including the following:

During benchmarking these fluctuations in time can cause unexpected results. Performance measurements can appear inflated if time slows down while work occurs, depressed if time accelerates while work is occurring, or somewhere in between. This topic will provide some details on this phenomenon.

If the hypervisor (or host operating system for a hosted product) is busy with other tasks it may stall slightly when delivering timer interrupts to the VM. This means that the guest timer appears to run slow. VMware products will correct for these deviations by pushing time back to its correct position but these "slow downs" and "catch ups" may occur at different points in a benchmark.

Artificially High Results

Consider the case where a great number of IO operations are measured over a long period of time. If a benchmark wants to run 10,000 IOPS and the native system takes 10ms to process each operation, the benchmark would measure 10ms on a native system. However, if the benchmarking is run on a virtual system and the virtualization software is busy servicing the IO operation instead of updating time, time may not progress properly during the operation. Although 10ms or more would have passed, perhaps the VM was only informed of the passing of 9ms. In this case, the operation appeared to run faster on the VM than the native system.

For benchmarks where many operations are measured of a large time window, this isn't a problem. On our 10K operation benchmark, if time were started before operation one and stopped after operation 10,000, a fluctuation of 1ms will make no difference. After all, that's only a 1ms inaccuracy on a sequence that would take 100s to run.

However, if the benchmark measured each operation individually and summarized them all to product a result, each individual 1ms inaccuracy would be summed over the entire run. The benchmark would report that the average IO length was 9ms even though observation of wall time would still show the passage of 100s during the 10,000 IO operation run.

Hyper-Threading on ESX Server

drummonds — Fri, 09 May 2008 21:31:11 GMT

Introduction

Those of us at VMware that regularly engaged with our field or directly with customers are often asked how Hyper-Threading impacts the performance of a system. I've now been asked this question enough times to have my canned "It Depends" response on the tip of my tongue for every conference I present at. I'm going to use this document to elaborate on that point a bit and provide a little more detail.

What Is Hyper-Threading?

Hyper-Threading is a technology included by Intel first in their Netburst line of parts. Hyper-Threaded processors present their individual processing cores to the system as if they are two processing cores. To use Intel's parlance, that means that each physical core appears in the operating system as two logical cores. While the OS can distinguish between a system that has two logical cores (i.e. a single physical core with Hyper-Threading enabled) and two physical cores, applications cannot. It is up the the OS's scheduler to choose if it wishes to use logical cores in the same manner as physical cores.

Is It Supported In ESX Server?

Hyper-Threading (HT) has been supported in ESX Server since version 2. ESX Server's scheduler is aware of the presence of HT and treats logical cores differently from physical cores. Virtual CPUs (vCPUs) requesting resources are assigned first to physical cores until all physical cores are loaded. If there are additional vCPUs requesting CPU resources they will then be assigned to the additional logical cores. By this method HT has no impact on performance until more vCPUs are concurrently executing than there exist physical cores.

How Does It Perform on ESX Server?

Understanding HT performance on native systems is tricky enough. Try Googling "hyperthreading performance" and you'll discover a world of information on this feature. By faking the presence of another processing core, Hyper-Threading removes some SMP scheduling from the guest operating system which is then handled by the processor's thread scheduler. Since the processor can manage context switches between its threads much faster than the OS can, this means that often heavy parallelism in applications results in improved performance due to Hyper-Threading.

The exact gains due to HT even on native systems is dependent on the workload. The industry has cited numbers that range from 0% to 40% gain when HT is enabled on supported processors. For the most part, HT improves performance nominally. There are a few cases where performance can slow down, but these are exceptions rather than the norm.

The best generalizations we can provide about HT on ESX are:

Until you have more vCPUs requesting processing power than there are physical cores, HT cannot hurt and provides no value.
Once you have more vCPUs requesting CPU than physical cores on the system, HT usually provides small gains.
While very early versions ESX may have sub-optimally handled HT, since ESX Server 2.5.3 robust support of HT in the scheduler means that it should not hurt performance.

For More Information

Hyper-Threading configuration options for performance optimization were provided in ESX Server 3 and beyond. See the Resource Management Guide (page 123) for more information.

Best Practices for Oracle

drummonds — Wed, 28 May 2008 00:01:33 GMT

While Oracle makes a lot of software for a bunch of purposes, the only best practice guidance available today is for Oracle databases. We'll build out pages dedicated to other Oracle products if and when other best practices are available.

Real Time Scheduling for Oracle DBs on Linux

The following nugget came from a database expert in performance engineering. Due to the possibility of this change screwing up your DBs it was only with great reluctance that he even committed this to ink. You're taking your own job in your hands if you try this with your production DBs. But, for those of you with a test/dev Oracle DB and an unrepentant need for speed, give this a try. --drummonds

Traditional Unix/Linux timeshare scheduler policies attempt to provide good interactive response times by favoring a process that has just had an I/O request completed over a running process. This can create havoc on a high transaction-rate database server, but can be avoided by manipulating the scheduling policies for the database processes.

The scheduler policies date back to the 1970s when we would be running an editor session simultaneously with nroff (a text formatting program that often ran for minutes to process a document) on a single-processor system. Allowing the long-running process to execute without preemption would mean unacceptably long response times for interactive applications. So, the scheduler decays the priority of a process as it runs and accumulates CPU time. Furthermore, when a timeshare priority process goes to sleep waiting for I/O completion, it sleeps at a stronger priority than any running process with a timeshare priority. This ensures, for example, that a text editor user will get immediate feedback for every character typed on the keyboard even if the system is busy running CPU hungry tasks.

The problem with this scheme comes to the forefront when we run an application such as Oracle on a modern computer system. In an online transaction processing environment, the database management system threads/processes typically run for a short period of time, in the order of milliseconds or tens of milliseconds, then issue a disk I/O or send a network packet. A typical large system may issue many thousands of disk accesses per second. Every time an I/O completes, the scheduler makes the issuing process (or thread) runable, and at a stronger priority than all running timeshare processes. So, we are guaranteed a preemption unless the system can find an idle CPU. And, the preempted process goes to the end of the run queue for its priority level.

Now, if a preemption occurs, and the running process was holding an important database resource, say, a latch, all other processes that may need that latch, possibly including the preemptor process itself, will go into a spin loop waiting for the latch, and will eventually put themselves to sleep until the process holding the latch runs again and releases the latch. This is very expensive in term of CPU usage. The spinning and the context switches will drive up the CPU utilization without an increase in the system throughput.

An even worse phenomenon can occur in a large system with a very high I/O rate, where the problem may actually exhibit itself as excess CPU idle due to processes putting themselves to sleep waiting for latches and not waking up when the latch is released. Another possible symptom is the thundering herd problem: once the process holding the latch runs and releases it, a large number of processes become runable, and chaos and inefficiency follows.

The simplest solution is to maintain a constant priority for the DBMS processes, even when sleeping. This way, we allow the running process to voluntarily give up the CPU, at which time it has supposedly released all latches. Because of the nature of OLTP workloads, processes will voluntarily give up the CPU after a few, or at most tens of, milliseconds. So avoiding involuntary preemptions will fix the problem.

One way of achieving this is using the real-time priority feature. We are not really interested in running the database processes at a strong priority. We only use real time priorities because of an additional feature that this scheduling policy offers: the priority of a real time process does not change when the process sleeps on I/O. As a result when an I/O completion occurs, a process becomes runable at the same priority as the currently running process, and is put at the end of the run queue. We avoid the involuntary preemption and the associated spinning and sleeping costs.

Here is a sample RHEL4.4 script that accomplishes this. This script will run the Oracle processes at a stronger priority than all timeshare priority processes and could result in starvation of other applications. It should be used only when you are certain that running these processes at a strong priority won't deny resources to other applications.

for DAEMON_PID in `ps -u oracle -f|grep -v grep|grep ora_|awk '{print $2}'`
do
sudo chrt --rr -p 82 --pid ${DAEMON_PID}
done
LGWR_PID=`ps -u oracle -f|grep -v grep|grep lgwr|awk '{print $2}'`
sudo chrt --rr -p 83 --pid ${LGWR_PID}
LSNR_PID=`ps -u oracle -f|grep -v grep|grep tnslsnr|awk '{print $2}'`
sudo chrt --rr -p 81 --pid ${LSNR_PID}
for SHADOW_PID in `ps -u oracle -f|grep -v grep|grep -v -E 'ora_|tnslsnr'|awk '{print $2}'`
do
sudo chrt --rr -p 81 --pid ${SHADOW_PID}
done

To avoid starving the Oracle daemons, we run them at a stronger priority than the shadow processes, with particular attention paid to the logwriter.

Co-scheduling SMP VMs in VMware ESX Server

drummonds — Fri, 02 May 2008 15:12:56 GMT

Background

VMware ESX Server efficiently manages a mix of uniprocessor and multiprocessor VMs, providing a rich set of controls for specifing both absolute and relative VM execution rates. For general information on cpu scheduling controls and other resource management topics, please see the official VMware Resource Management Guide.

For a multiprocessor VM (also known as an "SMP VM"), it is important to present the guest OS and applications executing within the VM with the illusion that they are running on a dedicated physical multiprocessor. ESX Server faithfully implements this illusion by supporting near-synchronous coscheduling of the virtual CPUs within a single multiprocessor VM.

The term "coscheduling" refers to a technique used in concurrent systems for scheduling related processes to run on different processors at the same time. This approach, alternately referred to as "gang scheduling", had historically been applied to running high-performance parallel applications, such as scientific computations. VMware ESX Server pioneered a form of coscheduling that is optimized for running SMP VMs efficiently.

Motivation

An operating system generally assumes that all of the processors it manages run at approximately the same rate. This is certainly true in non-virtualized environments, where the OS manages physical processor hardware. However, in a virtualized environment, the processors managed by a guest OS are actually virtual cpu abstractions scheduled by the hypervisor, which time-slices physical processors across multiple VMs.

At any particular point in time, each virtual cpu (VCPU) may be scheduled, descheduled, preempted, or blocked waiting for some event. Without coscheduling, the VCPUs associated with an SMP VM would be scheduled independently, breaking the guest's assumptions regarding uniform progress. We use the term "skew" to refer to the difference in execution rates between two or more VCPUs associated with an SMP VM.

Inter-VCPU skew violates the assumptions of guest software. Non-trivial skew can result in severe performance problems, and may even induce failures when the guest expects inter-VCPU operations to complete quickly. Let's first consider the performance implications of skew. Guest OS kernels typically use spin locks for interprocessor synchronization. If the VCPU currently holding a lock is descheduled, then the other VCPUs in the same VM will waste time busy-waiting until the lock is released. Similar performance problems can also occur in multi-threaded
user-mode applications, which may also synchronize using locks or barriers. Unequal VCPU progress will also confuse the guest OS cpu scheduler, which attempts to balance load across VCPUs.

An extreme form of this performance problem may also lead to correctness issues. For example, a guest kernel may perform inter-processor operations, such as TLB shootdowns, that are expected to complete quickly on physical hardware (e.g. several microseconds). The guest OS may timeout if it finds that such operations have not completed after an unreasonably long period of time (e.g. several milliseconds). Without coscheduling, we have observed this behavior in practice for several different guest operating systems, including Windows BSODs, and Linux kernel panics.

Strict coscheduling in ESX Server 2.x

VMware introduced support for running SMP VMs with the release of ESX Server 2 in 2003. ESX Server 2.x implemented coscheduling using an approach based on skew detection and enforcement.

The ESX scheduler maintains a fine-grained cumulative skew value for each VCPU within an SMP VM. A VCPU is considered to be making progress when it is running or idling. A VCPU's skew increases when it is not making progress while at least one of its sibling VCPUs is making progress. A VCPU is considered to be "skewed" if its cumulative skew value exceeds a configurable threshold, typically a few milliseconds.

Once any VCPU is skewed, all of its sibling VCPUs within the same SMP VM are forcibly descheduled ("co-stopped") to prevent additional skew. After a VM has been co-stopped, the next time any VCPU is scheduled, all of its sibling VCPUs must also be scheduled ("co-started"). This approach is called "strict" coscheduling, since all VCPUs must be scheduled simultaneously after skew has been detected.

In some situations, such as when the physical machine has few cores, and is running a mix of UP and SMP VMs, coscheduling may incur "fragmentation" overhead. For example, consider an ESX Server with two physical cores running one dual-VCPU VM and one single-VCPU VM. When the UP VM is running, the scheduler cannot use the remaining physical core to run just one of the SMP VM's two VCPUs. This effect is typically negligible in systems with larger numbers of cores (or with hyperthreading enabled), due to the increased flexibility available when mapping VCPUs to hardware execution contexts.

Note that a VCPU executing in the guest OS idle loop can be descheduled without affecting coscheduling, since the guest OS can't tell the difference. In other words, an idle VCPU does not accumulate skew, and is treated as if it were running for coscheduling purposes. This optimization ensures that idle guest VCPUs don't waste physical processor resources, which can instead be allocated to other VMs. For example, an ESX Server with two physical cores may be running one VCPU each from two different VMs, if their sibling VCPUs are idling, without incurring any coscheduling overhead. Similarly, in the fragmentation example above, if one of the SMP VM's VCPU is idling, then there will be no coscheduling fragmentation, since its sibling VCPU can be scheduled concurrently with the UP VM.

Relaxed coscheduling in ESX Server 3.x

The coscheduling algorithm employed by the ESX scheduler was significantly enhanced with the release of ESX Server 3 in 2006. The basic coscheduling approach is still based on skew detection and enforcement.

However, instead of requiring all VCPUs to be co-started, only those VCPUs that are skewed must be co-started. This ensures that when any VCPU is scheduled, all other VCPUs that are "behind" will also be scheduled, reducing skew. This approach is called "relaxed" coscheduling, since only a subset of a VM's VCPUs must be scheduled simultaneously after skew has been detected.

To be more precise, suppose an SMP VM consists of multiple VCPUs, including VCPUs A, B, and C. Suppose VCPU A is skewed, but VCPUs B and C are not skewed. Since VCPU A is skewed, VCPU B can be scheduled to run only if VCPU A is also co-started. This ensures that the skew between A and B will be reduced. But note that VCPU C need not be co-started to run VCPU B. As an optimization, the ESX scheduler will still try to co-start VCPU C opportunistically, but will not require this as a precondition for running VCPU B.

Relaxed coscheduling significantly reduces the possibility of coscheduling fragmentation, improving overall processor utilization.

Conclusions

ESX Server employs sophisticated cpu scheduling algorithms that enforce rate-based quality-of-service for both uniprocessor and multiprocessor VMs. For multiprocessor VMs, coscheduling techniques ensure that virtual CPUs make uniform progress, faithfully implementing the illusion that the VM is running on dedicated multiprocessor hardware, ensuring efficient execution of guest software. Optimizations such as relaxed coscheduling and descheduling idle VCPUs provide a high-performance execution environment that efficiently utilizes physical host resources.

Appendix: ESX Server coscheduling statistics

ESX Server 3.x exports statistics related to the coscheduling behavior of multiprocessor VMs. The "esxtop" utility can be used to examine these statistics on a live ESX system.

The %CSTP column in the CPU statistics panel shows the fraction of time the VCPUs of a VM spent in the "co-stopped" state, waiting to be "co-started". This gives an indication of the coscheduling overhead incurred by the VM. If this value is low, then any performance problems should be attributed to other issues, and not to the coscheduling of the VM's virtual cpus.

Best Practices for Apache

drummonds — Tue, 27 May 2008 23:24:29 GMT

Apache web servers are a common target for virtualization in almost every data center. This page will collect best practices for setting up and configuring Apache for best performance.

No Apache-specific information has yet been provided. But tips are available on the Best Practices for Web Servers page.

Best Practices for Web Servers

drummonds — Tue, 27 May 2008 23:10:36 GMT

Introduction

This page is a collection point for ESX Server configuration options that can maximize web server performance regardless of the choice of web server.

TCP Transmit Coalescing

In a recent whitepaper published on SPECweb performance on VMware ESX Server, the use of TCP transmit coalescing was described to improve web server performance. The general idea behind transmit coalescing is to buffer TCP transmits at the ESX Server for a brief period of time to allow multiple packets to be transmitted at one time. This introduces a very slight increase in latency but can provide a dramatic increase in efficiency.

Transmit coalescing can be turned on with the following steps:

Using the VMware Infrastructure Client, choose the ESX Server host on which the virtual machine is deployed.
Click the Configuration tab.
Click Advanced Settings in the Software panel.
Click the Net tab.
Edit the Net.vmxnetThroughputWeight value to 128, then click OK.
Reboot the virtual machine.

Details and performance results are provided in the SPECweb paper .

Resources

Apache Best Practices

IIS Best Practices

Red Hat Enterprise Linux

drummonds — Fri, 16 May 2008 21:55:13 GMT

Page under construction.

Quick notes:

RHEL5 always uses a Linux Timer Rate timer rate of 1000 Hz. This can be decreased in RHEL5.1 for greater efficiency of RHEL SMP guests.
RedHat has not enabled VMI in their kernels, so out-of-the-box paravirtualization is not possible. But custom kernels can easily be built to take advantage of this feature. See VMware KB article #1003644.

Ubuntu

drummonds — Fri, 16 May 2008 22:13:21 GMT

Ubuntu supports paravirtualization!

I'll have more to say about Ubuntu later.

EMC World 2008

drummonds — Mon, 19 May 2008 15:29:29 GMT

I'm going to be at EMC World 2008 from May 19 through May 23. I'll be presenting on VMware's VI3 architecture with respect to performance, tips and techniques for performance monitoring and analysis, and best practices for best performance. My session, titled "VMware ESX Server Performance Analysis", is offered on Monday at 4:30 and Tuesday at 11:30. I'll also be hosting birds-of-a-feather sessions on the same subject at 2:30 on Tuesday. Please drop by if you're at the conference!

Welcome to the VMware Performance Community Forum!

drummonds — Fri, 16 May 2008 18:58:21 GMT

Bring your performance topics to this forum. Questions and comments from all are welcome.

Scott