(Please forgive my submission of this issue in this forum; we do not have an active support contract and cannot submit a ticket using the normal channel.)
I represent MiserWare, Inc. (http://miserware.com). We are working to provide a cross-platform power management solution to a large government agency. Our product Granola (http://grano.la) utilizes DVFS as the underlying technology, managing DVFS states according to a patented algorithm that guarantees a user-defined level of performance. On the ESXi platform, we take advantage of the Low and Static processor power management modes to affect state changes. However, we have run into an issue on (at least) one platform using ESXi 5.1.
As part of the contract mentioned above, we are attempting to evaluate the power envelope and ultimately the effectiveness of Granola on an IBM 3850 server platform. However, we have seen very little power savings across the range of utilization on this platform on ESXi. In fact, across the spectrum of utilization, we saw 2% or less reduction in energy between Static and Low modes. This is the same order of savings as utilizing C-states on this machine.
The setup was 4 Windows Server 2008 VMs spanning the 80 CPU threads of the system (20 VCPUs each), with each VM allocated 128GB of RAM. First, we verified that ESXi was indeed managing P-states via the settings in VSISH (it was). Next, we checked that the P-state changes were occuring with the 'p' command in esxtop. For lower utilization levels, most of the time was spent in the lowest P-state (P11). Despite this, there was still no significant change in power consumption versus Static mode, where all time was spent in P0, except for the ~2% savings mentioned above, attributed to C-state management. Next, we verified that the C-state management was indeed causing the small change we were seeing by enabling Custom mode and disabling C-state management via VSISH.
Finally, we loaded Fedora 15 on the machine in order to verify that the issue was not caused by the BIOS. On a Linux system, running the exact same workloads, we saw a power reduction up to 33% between the highest and lowest CPU speeds, set via the cpufreq subsystem. Even in the lower utilization ranges, we saw significant savings; at a per-thread CPU utilization of 30%, we saw a reduction in energy of over 17%.
In order to fulfill our contract with the agency, we need to exhibit energy savings similar to those on Linux. I understand that there is management going on within the Low setting, and as such the energy reduction at higher utilization levels may be negligible. However, based on our tests above, I would conclude that ESXi 5.1 is failing to affect P-state changes on this system. We are currently working to test ESXi 5.1 on other systems, as well as testing older versions of ESXi on the IBM system. If you have any questions, please contact me at turner@miserware.com. Thanks in advance.
