Large Memory Pages

[#1]

In ESX Server 3.5 VMware introduced support for large memory pages in the guest. Large memory pages, an architecteral feature available in x86 microprocessors for decades, can be used to improve performance on workloads that make use of them. With CPU, hypervisor, OS, and application support, throughputs can go up and CPU utilization can go down. Since applications such as Oracle databases and Java have been using large pages on Linux and Windows for years, the introduction of this support on ESX Server allows for increased gains in performance over previous virtual installs. VMware is currently the only virtualization vendor to support large pages.

VMware's support for large memory pages is detailed in the Large Page Performance performance study. That paper includes data on throughput gains in SPECjbb. The results are duplicated here:

At LinuxWorld 2008 VMware presented further data on the value of large memory pages with Oracle databases. Here is a chart showing those gains with VMware binary translation (BT):

Data also shared at LinuxWorld on the value of large memory pages with AMD Rapid Virtualization Indexing (RVI; formerly called NPT):

The gains derived from the presence of large pages in these Oracle/Swingbench results are atypical. Large pages have been documented in numerous locations to provide benefits between 5-20% on most database applications. The increases shown here (of up to 350%) are due to the specialized configuration which is less about demonstrating real-world application performance and more about stressing the underlying configuration to uncover strengths and weaknesses.

[Ken_Cline]

Scott,

Can you clarify this for me:

"Since applications such as Oracle databases and Java have been using large pages on Linux and Windows for years, the introduction of this support on ESX Server allows for increased gains in performance over previous virtual installs."

The graph then compares "Native" to "Virtual" - is this showing that Oracle and Java in a VM perform better than Oracle and Java on bare metal?

Thanks!

KLC

Ken Cline

Technical Director, Virtualization

Wells Landers[/url]

VMware Communities User Moderator

[drummonds]

While the graph is showing performance gains for native and virtual, its not necessarily meant to highlight differences between them. I've included both data sets so that people can see that large pages are not a virtualization-specific feature and provide benefits regardless of virtualization. The presence of support for LP unlocks value that the hardware, OS, and application already provide.

It is true, though, that there are slightly more gains to be had in a virtual environment than a physical environment. This is due to the fact that TLB misses cost a little bit more in a VM than in a native environment. It is possible that this trend reverses with the presence of a hardware MMU such as AMD RVI, however. But you'll have to read my (not-yet-published) article on AMD RVI for more details on that.