Hi everybody,
I have a situation here and I kinda run out of ideas, I would really appreciate some help.
I have two new built RHEL 5.4 VMs with 4 vcpus, 4gb of memory LSI Logic and four E1000 vnics running on an ESX 4.0 each. The Red Hat servers work as a Tomcat application cluster with other 2 physical servers and uses LVS for load balancing.
This servers host the web interface through which all business offices connect to register sell transactions and as you can imagine the load is pretty heavy on some particular days of the year, and it is very critical for the business. The customer has been stressing the servers to test performance and determine wheter to virtualize all four servers or leave them running as physical servers. The results show the processors in the physical servers (two dual-core cpus) at approximately 30% of use, while the virtual ones go up to 70% with peaks of 90%.
I have verified the VMs and found out the kernel is smp, detecting all processors, no KVM at all; a top command from the guest OS would show CPU is most of the time idle, while the performance monitor from vCenter shows the usage levels previosly mentioned. So we did P2V of the physical servers these ones were replacing to check if we saw any difference in performance, it is the same.
What I found really interesting is that the performance monitor shows the value "cpu wait" pretty much stucked at 80% (no matter if the VMs are idle or working a lot, and it happens in the new builts and the converted ones) so I verified memory, network and disk I/O rates, they are not going over their capacity in any way, and the host cpu is being used by the 25/30% on average, so there is no apparent reason for this VMs to be waiting this badly to access the CPU, what could be happening?
