You are quite correct that with any application, Java or not, there are a large number of possible sources of performance problems. One item that you left out of your list is storage performance, which is the source of many problems that are initially blamed on ESX.

I have just published at performance troubleshooting guide for vSphere that focuses on identifying sources of performance problems in ESX 4.0. Most of the information in this guide is applicable to previous versions of ESX as well. The guide is available at Performance Troubleshooting for VMware vSphere 4 and ESX 4.0, and has some simple checks that can help to identify or rule out common problems.

I do not currently have access to a good servlet-based application for benchmarking, and so I have not been able to do any testing on Tomcat. However, I have been doing some testing with an enterprise-class three tier EJB-based application on SLES10 SP2. Since I am working with a partner, I will hold off mentioning the specific application-server and database involved. However, this application does have a significant networking load, as well as relying heavily on an external database (also running in a VM on a separate ESX host).

The tunings that I have found most significant are:

• Database performance: Ensuring that the database is performing properly is the number one factor in ensuring that the application provides acceptable response-times. No amount of JVM-level tuning will help if the DB performance is poor. My tuning was mostly related to the layout of the tablespaces on the storage, and proper sizing of the DB's in-memory caches.
• Use the vmxnet virtual adapters: Switching from the default e1000 adapter to the vmxnet3 adapter gave me a 10% increase in peak throughput (at acceptable response-times) for this particular workload. A nice paper about the benefits of the vmxnet3 adapter has been published in the VMware technical resources. For network-intensive VMs, tuning interrupt-coalescing can also help (see this document: Best Practices for Web Servers), but be careful with this one as it affects all VMs on a host, and there is a small tradeoff between the efficiency gains and higher latency.

Your approach of breaking out possible problems into small test-cases is a good one. However, be careful not to rule things out to quickly. A small test program to test DB access might miss something like improperly sized JDBC connection pools, which will only become obvious by monitoring the application when under load. I have seen one Java application in which the problem was a JDBC connection pool that was too slow. However, the best solution was not to increase the size of the pool, but to switch the VM to use vmxnet2, which decreased network latency and took enough pressure off of the connection pool. This is just one example of why it always makes sense to ensure that you are using best practices. This document is a good read.

Let us know what you find out. These case studies are helpful to everyone.