>>I think you may have found the problem already.
On I'm glad that the issue is pretty straightforward (though personally wished it was little bit more challenging 
)
>>As for FT we do not have it setup in our environment.
Thanks for confirming this. I had to ask this, because in esx40 the way we charge CPU cycles changes when your run a FT VM(s) on the host.
Let me attempt to explain what you are seeing here.
Your workload (custom script that pulls information from AD server) seem to consume 80% of CPU on average. The "Dynamic power savings mode" mode in HP systems controls the processor P-States based on the PCPU Utilization. This feature puts the processor into low frequency state by default and steps up the clock frequency only if the PCPU utilization increases above 60%. When you are using a 2 or 4 vCPU VM you are spreading the load across multiple PCPUs such that average utilization of any PCPU remains below 60%, so the processor always run in lower frequency mode and hence the slow performance. On AMD systems lowest processor frequency could be as low as 50% of the rated processor frequency (i.e. a 3Ghz processor will run at 1.5 Ghz). Ideally this should result only in 2x performance drop but I guess the "Low power halt state" is also possibly impactingh performance (see next paragraph)
"Low Power Halt state" also known as C1E halt puts the processor into more deeper sleep when idle (like for instance processor cache can be flushed to save power). This means the processor will have to pay a penalty (few wasted CPU cycles and cache misses) when it wakes up. This is usually fine if your system is mostly idle but if your processor frequently goes in and out of idle state (typical of bursty or I/O bound workloads) then this could affect performance quite a lot. Since the load spreads across the vCPUs when using a vSMP VM the chances of the processor going in and out of the idle state also increases. So I suspect that this also contributes to the performance loss. Especially since windows ping-pongs single threaded application across multiple vCPUs to evenly distribute the load (even though it has undesirable effects both natively as well as in a VM ).
In esx40 we now measure CPU utilization in two different ways one with respect P-MAX frequency (i.e the rated clock frequency) and other with respect to the current clock frequency. %USED is based on rated clock frequency, %UTIL is based on the varying clock frequency. So (if %UTIL is 100 then it means you cannot any more juice out of the processor, but if the processor is running at half its rated max frequency then %UTIL would be 100 but %USED would be only 50. In your esxtop screenshot I spotted that %USED and %UTIL were not matching thats why suspected power management. Also %RUN of 100 means that the VM was scheduled 100% of the time during the last refresh interval, if %RUN is 100 and %USED is 50 then it means the VM used the CPU all the time but burnt only 50% of the processors cycles with reference to its rated max.
I;m guessing your Oracle Forms App also has similar CPU utlization pattern so disabling power management should fix your problem. In general for benchmarking we recommend all BIOS power management features to be disabled. For production its a personal tradeoff so we leave it to the customer choice. Also as you might be already be aware that single threaded apps are better run in a single vCPU VM. Ping-ponging single threaded apps to multiple CPUs has performance impact both at the processor micro-architectural level and also in the virtualization layer.
ESX 4.0 has power management feature but it is disabled by default. If you want to use it you should set the BIOS option to "OS Controlled Mode" and flip power management in VMKernel (Advanced settings) and should reboot the host.
Hope this helps.
I think you may have found the problem already.
On I'm glad that the issue is pretty straightforward (though personally wished it was little bit more challenging
I think you may just get your wish...
When the DBAs ran the Oracle Forms compile it took 4 minutes on the physical server, but on the VM it took 9 minutes. This is using the "same" server I P2Ved the physical server to take the Oracle configuration out of the variables. I removed all the "old hardware" and HP hardware programs from the VM. I also had them try running the Oracle Compile on a VM that was built as a VM and not P2Ved and we got a similar 9 minute time frame. The physical and VM servers are comparable in number of CPUs and amount of ram. I did another esxtop in batch collection while the Oracle process was running on the P2Ved Oracle server (ITBTAPP01) and have attached it.
Thank you for all of the information in your last post, it was very good information and helped me understand what those settings are really doing. The description in the BIOS of each makes it sound like the performance hit of each of them is very minor, but understanding how ESX relates to the BIOS settings is great.
