- Contributors:
-
drummonds
-
RDellimmagine
Virtual Performance
Scott Drummonds works in a variety of performance areas at VMware: VDI, application best practices, competitive analysis, customer performance investigations, and outward bound communications. This blog will detail some of my musings on these subjects.
20 Posts
1
2
Previous
Next
Results of two experiments are shown on this graph: in one memory is reclaimed only through ballooning and in the other memory is reclaimed only through host swapping. The bars show the amount of memory reclaimed by ESX and the line shows the workload performance. The steadily falling green line reveals a predictable deterioration of performance due to host swapping. The red line demonstrates that as the balloon driver inflates, kernel compile performance is unchanged.
Kernel compilation performance remains high with ballooning because this workload needs very little memory and the guest OS can easily take unused pages from the application. Performance falls with swapping because ESX randomly selects virtual machine pages for swapping, whether those pages are in use by the application or not. The guest OS is better at selecting pages for reclamation than ESX is.
As before, the virtual machine using only ballooning maintains higher performance under memory pressure than the virtual machine whose memory is being swapped away by the host. Performance is constant and shows no negative impact due to ballooning until the balloon encroaches on the SGA. Again, ESX's host swapping randomly selects pages to send to disk which degrades performance even at small swap amounts.
As with kernel compile, the balloon driver safely reclaims memory from over-provisioned VMs with little impact to application performance.
Neither ESX nor the guest OS can efficiently take memory from the JVM without significantly degrading performance. Memory in Java is managed internal to the JVM and efforts by the host or guest to remove pages will equally negatively impact Java applications performance. In these environments it is wise to manually set the JVM's heap size and specify memory reservations for the virtual machine in ESX to account for the JVM, OS, and heap.
So here is my special recommendation for you: never, ever disable the balloon driver. This forces the host to swap that virtual machine's memory, should that resource become scarce. And where ballooning usually will not hurt performance, swapping always will. If you must protect an application from memory reclamation due to memory over-commitment, use reservations. They make admission control more effective, they self-document the needs of the VM, and they are easily configured.
The X-axis represents different combinations of VM and vCPU counts. SPECjbb is memory intensive and shows great gains with increases in CPU cache. The few cases that show dramatic benefit due to the ESX 4.0 scheduler are benefiting from the distribution of vCPUs across sockets. Very large gains are possible in this somewhat uncommon case.
In this experiment we ran VMmark against VMs that were configured outside of VMmark specifications. In one case some of the virtual machines were given too few vCPUs and in another they were given too many. Because VMmark's workload is fixed, changing VM sizes does not alter the amount of work performed by the VMs. In other words, the system's score does not depend on the VMs' vCPU count. Until CPU saturation, that is.
Notice that the scores are similar between the undersized, right-sized, and over-sized VMs. Up until tile 10 (60 VMs) they are nearly identical. There is a slight difference in processor utilization that begins to impact throughput (score) as the system runs out of CPU. At that point wasted cycles dedicated to unneeded vCPUs negatively impact the system performance. Two points I will call out from this work:
What this figure shows is that a four-way VM on ESX 3.5 can only be placed in two locations on this hypothetical two-socket configuration. There are 12 combinations for a two-way VM and eight for a uniprocessor VM. The scheduler has more opportunities to optimize VM placement when you provide it with smaller VMs.
In ESX 4 we have eliminated the cell lock so VMs can span multiple sockets, as item one states. Continue to think of this placement problem as a challenge to the scheduler that you can alleviate. By choosing multiple, smaller VMs you free the scheduler to pursue opportunities to optimize performance in consolidated environments.
This instruction, VMEXIT, is called each time the guest exits to the kernel. The graph shows its latency (delay) in completing this instruction, which represents a wait time incurred by the guest. Clearly Intel has made great strides in reducing VMEXIT's wait time from its Netburst parts (Prescott and Cedar Mill) to its Core architecture (Merom and Penryn) and on to its current generation, Core i7 (Nehalem). AMD processors have shown commensurate gains with AMD-V.
Large pipelines aren't necessarily a problem for desktop environments, where single threaded applications used to dominate the market. But in the enterprise, application thread counts were larger. Furthermore, consolidation in virtual environments drew thread counts even higher. With more contexts in the processor, the number of pipeline stalls and flushes increased, and performance fell.
Because of decreased efficiency of consolidated workloads on processors with long pipelines, VMware has often recommended that performance-intensive VMs be run on processors no older than 2-3 years. This excludes Intel's Netburst parts. VI3 and vSphere will do a fine job at virtualizing your less-demanding applications on any supported processors. But use newer parts for the applications that hold your highest performance expectations.
This graph is based on a model that greatly simplifies working sets and the hypervisor's impact on them. Assuming that ESX increases the working set by 256 KB, this graph shows the difference in cache hit rate due to the contributions of the hypervisor. Notice that with very small caches and very small application working sets, the cache hit rate suffers greatly due to the addition of even 256 KB of virtualization support instructions. And even up to 2 MB, a 10% decrease in cache hit rate can be seen in some applications. With a 256 KB contribution by the kernel, cache hit rates do not change significantly with cache sizes of 4 MB and beyond.
In some cases a 10% improvement in cache hit rate can double application throughput. This means that a doubling of cache size can profoundly effect the performance of virtual applications as compared to native. Given ESX's small contribution to the working set, you can see why we at VMware recommend that customers run their performance-intensive workloads on CPUs with 4 MB caches or larger.
Kit is going to be in Orlando with me next week to talk about ESX and guest memory management. He's going to explain the difficult process of recovering unused memory from guests to enable over-commitment. Be sure and see him if you're in town!
1
2
Previous
Next
A couple of days ago we finally got out one of my favorite papers from our ongoing vSphere launch activities. This paper on ESX memory management, written by Fei Guo in performance engineering, has three graphs that are absolute gems. These graphs show balloon driver memory savings next to throughput numbers for three common benchmarks. The conclusion is inescapable: the balloon driver reclaims memory from over-provisioned VMs with virtually no impact to performance. This is true on every workload save one: Java.
Example 1: Kernel Compile
Linux kernel compilation models a common developer environment involving a large number of code compiles. This process is CPU and IO intensive but uses very little memory. Results of two experiments are shown on this graph: in one memory is reclaimed only through ballooning and in the other memory is reclaimed only through host swapping. The bars show the amount of memory reclaimed by ESX and the line shows the workload performance. The steadily falling green line reveals a predictable deterioration of performance due to host swapping. The red line demonstrates that as the balloon driver inflates, kernel compile performance is unchanged.
Kernel compilation performance remains high with ballooning because this workload needs very little memory and the guest OS can easily take unused pages from the application. Performance falls with swapping because ESX randomly selects virtual machine pages for swapping, whether those pages are in use by the application or not. The guest OS is better at selecting pages for reclamation than ESX is.
Example 2: Oracle/Swingbench
Oracle's database is best tested against Swingbench, the OLTP load generation tool provided by Oracle. Database workloads utilize all system resources but show a non-linear dependence on memory. Memory can be safely reclaimed from OSes running databases until the cache becomes smaller than needed by the workload. The following figure shows this. As before, the virtual machine using only ballooning maintains higher performance under memory pressure than the virtual machine whose memory is being swapped away by the host. Performance is constant and shows no negative impact due to ballooning until the balloon encroaches on the SGA. Again, ESX's host swapping randomly selects pages to send to disk which degrades performance even at small swap amounts.
As with kernel compile, the balloon driver safely reclaims memory from over-provisioned VMs with little impact to application performance.
Example 3: Java/SPECjbb
Java provides a special challenge in virtual environments due to the JVM's introduction of a third level of memory management. The balloon driver draws memory from the virtual machine without impacting throughput because the guest OS efficiently claims pages that its processes are not using. But in the case of Java, the guest OS is unaware of how the JVM is using memory and is forced to select memory pages an arbitrarily and inefficiently as ESX's swap routine. Neither ESX nor the guest OS can efficiently take memory from the JVM without significantly degrading performance. Memory in Java is managed internal to the JVM and efforts by the host or guest to remove pages will equally negatively impact Java applications performance. In these environments it is wise to manually set the JVM's heap size and specify memory reservations for the virtual machine in ESX to account for the JVM, OS, and heap.
Conclusions and Scott's Special Recommendation
Love your balloon driver. Your application owners are always asking for more memory than they need. With great comfort you can over-provision memory some and rely on ESX and the balloon driver to reclaim what is not in use. Without the balloon driver, ESX will be forced to use its last technology for managing memory over-commit: host swapping. And host swapping always decreases performance.So here is my special recommendation for you: never, ever disable the balloon driver. This forces the host to swap that virtual machine's memory, should that resource become scarce. And where ballooning usually will not hurt performance, swapping always will. If you must protect an application from memory reclamation due to memory over-commitment, use reservations. They make admission control more effective, they self-document the needs of the VM, and they are easily configured.
I spent a great deal of time answering customers' questions about the scheduler. Never have so many questions been asked about such an abstruse component for which so little user influence is possible. But CPU scheduling is central to system performance, so VMware strives to provide as much information on the subject as possible. In this blog entry, I want to point out a few nuggets of information on the CPU scheduler. These four bullets answer 95% of the questions I get asked.
Item 1: ESX 4's Scheduler Better Uses Caches Across Sockets
On UMA systems with low load levels, virtual machine performance improves when each virtual CPU (vCPU) is placed on its own socket. This is because providing each vCPU its own socket also give it the entire cache on that CPU. On page 18 of a recent paper on the scheduler written by Seongbeom Kim, a graph highlights the case where vCPU spreading improves performance. The X-axis represents different combinations of VM and vCPU counts. SPECjbb is memory intensive and shows great gains with increases in CPU cache. The few cases that show dramatic benefit due to the ESX 4.0 scheduler are benefiting from the distribution of vCPUs across sockets. Very large gains are possible in this somewhat uncommon case.
Item 2: Overuse of SMP Only Slows Consolidated Environments At Saturation
For years customers have asked me how many vCPUs they should give to their VMs. The best guidance, "as few as possible", seems too vague to satisfy. It remains the only correct answer, unfortunately. But a recent experiment performed by Bruce Herndon's team sheds some light on this VM sizing question.In this experiment we ran VMmark against VMs that were configured outside of VMmark specifications. In one case some of the virtual machines were given too few vCPUs and in another they were given too many. Because VMmark's workload is fixed, changing VM sizes does not alter the amount of work performed by the VMs. In other words, the system's score does not depend on the VMs' vCPU count. Until CPU saturation, that is.
Notice that the scores are similar between the undersized, right-sized, and over-sized VMs. Up until tile 10 (60 VMs) they are nearly identical. There is a slight difference in processor utilization that begins to impact throughput (score) as the system runs out of CPU. At that point wasted cycles dedicated to unneeded vCPUs negatively impact the system performance. Two points I will call out from this work:
- Sloppy VI admins that provide too many vCPUs need not worry about performance when their servers are under low load. But performance will suffer when CPU utilization spikes.
- The penalty of over-sizing VMs gets worse as VMs get larger. Using a 2-way VM is not that bad, but unneeded use of 4-way VM when one or two processors suffice can cost up to 15% of your system throughput. I presume that unnecessarily eight vCPUs would be criminal.
Item 3: ESX Has Not Strictly Co-scheduled Since ESX 2.5
I have documented ESX's relaxation of co-scheduling previously (Co-scheduling SMP VMs in VMware ESX Server). But this statement cannot be repeated too frequently: ESX has not strictly co-scheduled virtual machines since version 2.5. This means that ESX can place vCPUs from SMP VMs individually. It is not necessary to wait for physical cores to be available for every vCPU before starting the VM. However, as Item 3 pointed out, this does not give you free license to over-size your VMs. Be frugal with your SMP VMs and assign vCPUs only when you need them.Item 4: The Cell Construct Has Been Eliminated in ESX 4.0
In the performance best practices deck that I give at conferences I talk about the benefits of creating small virtual machines over large ones. In versions of ESX up to ESX 3.5, the scheduler used a construct called a cell that would contain and lock CPU cores. The vCPUs from a single VM could never span a cell. With a ESX 3.x's cell size of four this meant that VMs never spanned multiple four-core sockets. Consider this figure: What this figure shows is that a four-way VM on ESX 3.5 can only be placed in two locations on this hypothetical two-socket configuration. There are 12 combinations for a two-way VM and eight for a uniprocessor VM. The scheduler has more opportunities to optimize VM placement when you provide it with smaller VMs.
In ESX 4 we have eliminated the cell lock so VMs can span multiple sockets, as item one states. Continue to think of this placement problem as a challenge to the scheduler that you can alleviate. By choosing multiple, smaller VMs you free the scheduler to pursue opportunities to optimize performance in consolidated environments.
My colleague in product management, Praveen Kannan, has been working to extend Perfmon to show some ESX performance counters. This capability is automatically installed with VMware Tools on vSphere 4. But Praveen and I have made a stand-alone version available to those of you that are still on VI3. Download it here to give it a try.
To install, place the file in an appropriately-named directory on any Windows VM on VI3. Double-click the executable, which will self-extract the files into the same directory. Run "install.bat" and you're done.
Once you bring up Perfmon you'll see two new performance objects on your computer: "VM Memory" and "VM Processor". These objects contain counters exposed by ESX that accurately reflect the VM's memory and CPU usage. Here's Perfmon on my test VM after I've installed the tool.
This makes collection of host stats a breeze. Windows Management Instrumentation (WMI) programs can now easily get access to reliable host statistics. And anyone with access to Perfmon can get see their VM's resource usage. Unlike guest-based statistics, the host-statistics shown through these counters accurately reflect resource usage in the presence of virtualization overheads and time slicing of VMs.
Disclaimer:
This is a pre-release "sneak peak" version. Eventually this tool will be available for download on vmware.com and supported by VMware. But today there is no support for this tool and you're using it "as-is". Use at your own risk and do not contact VMware support for help with this release.That's VMware's official position on this tool. But feel free to comment here with any ideas about this great new feature.
Newer processors are much more important to virtualization than physical, un-virtualized environments. The generational improvements haven't just increased the raw compute power, they've also reduced the overheads associated with virtualization. This blog entry will describe three key changes that have particularly impacted virtual performance.
Hardware Assist Is Faster
In 2008, AMD became the first CPU vendor to produce a hardware memory management unit equipped to support virtualization. They called this technology Rapid Virtualization Indexing (RVI). This year Intel did the same with Extended Page Tables (EPT) on its Xeon 5500 line. Both vendors have been providing the ability to virtualize privileged instructions since 2006, with continually improving results. Consider the following graph showing the latency of one key instruction from Intel:This instruction, VMEXIT, is called each time the guest exits to the kernel. The graph shows its latency (delay) in completing this instruction, which represents a wait time incurred by the guest. Clearly Intel has made great strides in reducing VMEXIT's wait time from its Netburst parts (Prescott and Cedar Mill) to its Core architecture (Merom and Penryn) and on to its current generation, Core i7 (Nehalem). AMD processors have shown commensurate gains with AMD-V.
Pipelines Are Shorter
The longest pipelines in the x86 world were in Intel's Netburst processors. These processor's pipelines had twice as many stages at their counterparts at AMD and twice as many as the generation of Intel CPUs that followed. The increased pipeline length would have enabled support for 8 GHz silicon, had it arrived. Instead, silicon switching speeds hit a wall at 4 GHz and Intel (and its customers) were forced to suffer the drawbacks of large pipelines.Large pipelines aren't necessarily a problem for desktop environments, where single threaded applications used to dominate the market. But in the enterprise, application thread counts were larger. Furthermore, consolidation in virtual environments drew thread counts even higher. With more contexts in the processor, the number of pipeline stalls and flushes increased, and performance fell.
Because of decreased efficiency of consolidated workloads on processors with long pipelines, VMware has often recommended that performance-intensive VMs be run on processors no older than 2-3 years. This excludes Intel's Netburst parts. VI3 and vSphere will do a fine job at virtualizing your less-demanding applications on any supported processors. But use newer parts for the applications that hold your highest performance expectations.
Caches Are Larger
A cache is highly effective when it fully contains the software's working set. The addition from the hypervisor of even a small about of code will change the working set and reduce cache hit rate. I've attempted to illustrate this concept with the following simplified view of the relationship between cache hit rates, application working set, and cache sizes:This graph is based on a model that greatly simplifies working sets and the hypervisor's impact on them. Assuming that ESX increases the working set by 256 KB, this graph shows the difference in cache hit rate due to the contributions of the hypervisor. Notice that with very small caches and very small application working sets, the cache hit rate suffers greatly due to the addition of even 256 KB of virtualization support instructions. And even up to 2 MB, a 10% decrease in cache hit rate can be seen in some applications. With a 256 KB contribution by the kernel, cache hit rates do not change significantly with cache sizes of 4 MB and beyond.
In some cases a 10% improvement in cache hit rate can double application throughput. This means that a doubling of cache size can profoundly effect the performance of virtual applications as compared to native. Given ESX's small contribution to the working set, you can see why we at VMware recommend that customers run their performance-intensive workloads on CPUs with 4 MB caches or larger.
I recently attended a practice talk for next week's Partner Exchange hosted by Kit Colbert, one of our senior engineers, who is leading a whole bunch of cool efforts around performance. I wanted to "leak" one slide that his showed us that we'll be touching up for publication. Some of you that are curious about memory counters and want a different take from Memory Performance Analysis and Monitoring may find this interesting.
Some of this stuff won't make sense outside of Kit's presentation, but let me point out a few things that may help consume the information in this incredible chart:
- One of the key messages from Kit's presentation is that ESX reports memory with respect to the guest (the VM) and the host. The very top rectange shows memory stats reported for each VM. The second rectangle shows the single VM's memory stats reported by each host.
- As can be seen from the above, the consumed memory in the host represents everything in the VM, minus the savings due to page sharing.
- This graph doesn't yet highlight the difference between ballooned memory and swapped memory from the guest perspective. From the guest's perspective, swapped memory is much more attractive then ballooned memory, as the guest doesn't know that the swapped memory is gone. But it does see the ballooned memory as pinned. ESX is clever enough to deflate the balloon driver, if possible, when the guest starts to access swapped memory to avoid the host's swapping of guest memory.
- The final rectangle shows memory of all VMs from the host's perspective. Don't pay attention to the reserved and unreserved memory; I'm told those are unnecessary distractions that will be removed.
Kit is going to be in Orlando with me next week to talk about ESX and guest memory management. He's going to explain the difficult process of recovering unused memory from guests to enable over-commitment. Be sure and see him if you're in town!