In my recent whitepaper on enterprise Java performance on tc Server 2.0 and vSphere 4.1, I discuss tests that I ran with native and virtualized configurations using 1, 2, and 4 total CPUs. The setup that I used when I collected the published results reflected the results of some tuning experiments that aren't covered in the paper. At the JVM level, the two main tunings that I investigated were the maximum heap size, and the impact of using large pages. In this blog I will look at the impact of changing the maximum heap-size.
As I mention in the paper, the application that I used was the Java version of Olio, a Java web application that implements a complete social networking website. When choosing my JVM tunings, one of the first things that I wanted to understand was how changing the maximum Java heap size affected the peak number of Olio users that could be supported by the application. I also wanted to understand how changing the number of CPUs affected the choice of heap size.
The chart below shows the change in peak-throughput as I increased the maximum heap size for the different VM configurations. In order to avoid confusion with the published results, which reflect additional test-bed tuning, I normalized all of the values to the lowest datapoint.
There are a few interesting observations to be made about this data. There is clearly a tradeoff to be made between heap size and performance. However, after a certain point, the benefit of larger heap sizes is limited. Depending on memory availability, it may worthwhile to give up a bit of peak throughput in order to save memory. This is particularly true if reducing the heap size, and therefore the VM’s total memory size, allows you to avoid over-committing memory on an ESX host. While memory over-commit works well for most workloads, Java applications tend to suffer when ESX has to reclaim memory from the VM.
It is also clear that the same heap size won’t work for all VM configurations. In my tests I chose to use a maximum heap size of 2048MB for a 1-vCPU VM, 2560MB for a 2-vCPU VM, and 4096MB for a 4-vCPU VM. If I had used 2048MB for all three configurations, I would have lost about 5% off of the peak-throughput in the 2-vCPU case, and a painful 35% in the 4-vCPU case. Of course, the specific heap-size needed, and the impact of changing the VM configuration and heap-size, will vary for different Java applications. However, the message is clear. When moving from one hardware configuration to another, it is important to re-assess the choice of maximum heap-size. This is true whether you are changing the VM configuration, or moving from physical on one system to virtual on another.
