VCenter reports much higher CPU usage for the VM than the Guest VM's CPU usage in task manager

[jwnchoate]

Hi, I have a 2Cpu ESX3 (2 single cores w/ hyperthreading), running 2 Virtual Machines. One VM runs a busy database server the other VM runs a nearly idle A/V (1 vcpu) server.

The A/V server runs maybe 1-2% on both the Guest VM and in VCenter, not using much resources.

However, when I look at the CPU performance tab of the database server in VCenter it shows 80% much of the time, but task manager in the VM itself shows 2-5% cpu usages. The database guest is a 2 VCPU VM. I am puzzled at the discrepancy. Other DB Servers on other ESX hosts show as expected. Actual ESX Host CPU mirrors the Graph for the VM in VCenter. Its runs around 80% as well.

How can I explain why task manager says the guest is idle, yet the ESX host says its running full bore?

Here is an esxtop sampling.

12:40:39pm up 26 days, 10:58, 46 worlds; CPU load average: 0.90, 0.89, 0.90

PCPU(%): 100.00, 88.14 ; used total: 94.07

LCPU(%): 99.43, 1.08, 80.15, 7.99

CCPU(%): 0 us, 1 sy, 99 id, 0 wa ; cs/sec: 91

ID GID NAME NMEM %USED %SYS %OVRLP %RUN %WAIT %BWAIT %TWAIT %CRUN %CSTP %IDLE %RD

1 1 idle 4 14.31 0.00 0.19 50.22 0.00 0.00 0.00 0.00 0.00 0.00 200.0

2 2 system 5 0.00 0.00 0.00 0.00 500.00 0.00 500.00 0.00 0.00 0.00 0.0

6 6 console 1 1.36 0.00 0.00 1.30 98.77 0.85 99.62 0.00 0.00 99.62 0.4

7 7 helper 13 0.01 0.00 0.00 0.00 1300.00 0.00 1300.00 0.00 0.00 0.00 0.0

8 8 drivers 9 0.00 0.00 0.00 0.01 900.00 0.00 900.00 0.00 0.00 0.00 0.0

9 9 vmotion 1 0.00 0.00 0.00 0.00 100.00 0.00 100.00 0.00 0.00 0.00 0.0

12 12 vmware-vmkauthd 1 0.00 0.00 0.00 0.00 100.00 0.00 100.00 0.00 0.00 0.00 0.0

13 13 lgvoracle 7 185.51 0.31 0.26 187.79 452.35 66.28 518.63 0.00 0.00 13.84 3.3

14 14 lgvsav 5 1.34 0.01 0.15 2.00 407.26 97.06 500.00 0.00 0.00 99.28 0.6

12:41:14pm up 26 days, 10:58, 46 worlds; CPU load average: 0.92, 0.89, 0.90

PCPU(%): 99.56, 79.78 ; used total: 89.67

LCPU(%): 90.63, 8.94, 21.40, 58.38

CCPU(%): 1 us, 0 sy, 99 id, 0 wa ; cs/sec: 93

ID GID NAME NMEM %USED %SYS %OVRLP %RUN %WAIT %BWAIT %TWAIT %CRUN %CSTP %IDLE %RD

1 1 idle 4 23.07 0.00 0.12 46.75 0.00 0.00 0.00 0.00 0.00 0.00 200.0

2 2 system 5 0.00 0.00 0.00 0.00 500.00 0.00 500.00 0.00 0.00 0.00 0.0

6 6 console 1 2.13 0.00 0.00 2.11 88.80 9.71 98.50 0.00 0.00 98.50 0.5

7 7 helper 13 0.01 0.00 0.00 0.01 1300.00 0.00 1300.00 0.00 0.00 0.00 0.0

8 8 drivers 9 0.00 0.00 0.00 0.01 900.00 0.00 900.00 0.00 0.00 0.00 0.0

9 9 vmotion 1 0.00 0.00 0.00 0.00 100.00 0.00 100.00 0.00 0.00 0.00 0.0

12 12 vmware-vmkauthd 1 0.00 0.00 0.00 0.00 100.00 0.00 100.00 0.00 0.00 0.00 0.0

13 13 lgvoracle 7 175.76 0.28 0.33 178.57 435.53 91.64 527.17 0.00 0.00 22.99 2.6

14 14 lgvsav 5 1.25 0.00 0.07 1.80 428.55 74.27 500.00 0.00 0.00 98.62 1.3

12:42:09pm up 26 days, 10:59, 46 worlds; CPU load average: 0.90, 0.90, 0.90

PCPU(%): 99.66, 95.17 ; used total: 97.42

LCPU(%): 79.00, 20.66, 72.80, 22.37

CCPU(%): 0 us, 1 sy, 98 id, 1 wa ; cs/sec: 94

ID GID NAME NMEM %USED %SYS %OVRLP %RUN %WAIT %BWAIT %TWAIT %CRUN %CSTP %IDLE %RD

1 1 idle 4 7.86 0.00 0.60 36.94 0.00 0.00 0.00 0.00 0.00 0.00 200.0

2 2 system 5 0.00 0.00 0.00 0.00 500.00 0.00 500.00 0.00 0.00 0.00 0.0

6 6 console 1 1.51 0.00 0.00 1.42 89.13 10.15 99.28 0.00 0.00 99.27 0.7

7 7 helper 13 0.00 0.00 0.00 0.00 1300.00 0.00 1300.00 0.00 0.00 0.00 0.0

8 8 drivers 9 0.00 0.00 0.00 0.01 900.00 0.00 900.00 0.00 0.00 0.00 0.0

9 9 vmotion 1 0.00 0.00 0.00 0.00 100.00 0.00 100.00 0.00 0.00 0.00 0.0

12 12 vmware-vmkauthd 1 0.00 0.00 0.00 0.00 100.00 0.00 100.00 0.00 0.00 0.00 0.0

13 13 lgvoracle 7 190.29 1.86 1.64 193.79 437.14 74.87 512.01 0.01 0.00 6.73 3.8

14 14 lgvsav 5 1.96 0.01 0.70 2.69 414.95 85.49 500.00 0.00 0.00 96.73 3.7

[etieseler]

Good question, something that I have seen and wondered about.

As was explained to me: Task Manager is not a very good metric of CPU usage. Task manager is a metric of the amount of time it takes for a request to be serviced by the CPU. The longer it takes for the CPU to finish, the busier the CPU must be.

VC shows you the MHz being used by that VM. So ESX is throwing gobs of resource at this VM and the VM happily gets its CPU cycles completedy quickly. It see's a low load on the CPU because of the fast processing tim; therefore task manager shows low CPU usage.

I have the same situation where some VM's show low CPU in Taskmanager but using alot of MHz, while others show a larger load in Task manager while not necesarilly using alot of MHz.

Anyone else care to chime in?

-Ed

[avarcher]

I've seen it with P2V'd VMs and the wrong HAL is being used by the guest OS, have a look at this, http://communities.vmware.com/thread/128253 and .

[Texiwill]

Hello,

The task manager expects full cpu cycles and it will only get them when the VM is running. It does its math based on that. Running within a VM the task manager is only a metric that will tell you that the utilization either went up or down, but not exact values. The values are pretty much meaningless unless pCPU is NOT over committed, which in most cases does not happen. VirtualCenter or esxtop is what you should be using. Use task manager performance to perhaps give you a 'oh I did something that took up more CPU' instead of 'oh it took up 50% of the CPU'. What virtual center or esxtop sees is the final say in performance metrics. Most tools out there that do performance monitoring of ESX look at the same data esxtop does.

Best regards,

Edward L. Haletky

VMware Communities User Moderator

====

Author of the book 'VMWare ESX Server in the Enterprise: Planning and Securing Virtualization Servers', Copyright 2008 Pearson Education. As well as the Virtualization Wiki at http://www.astroarch.com/wiki/index.php/Virtualization

[jwnchoate]

Yes, I did figure the ESX values were correct, however the VM has pretty much no competiton and its strange to see the nearly no usage. Looking at other busy hosts, its the same story.

[vikern]

I experienced 100% CPU usage inside vCenter on my VM after I changed the number of vCPU from 2 to 1. Inside the VM using VM Perfmon it gave me CPU usage between 1 - 5%.

I followed avarchers suggestion and changed the HAL from multi to single processor and rebootet. vCPU usage inside vCenter is now correct.

Thank you avarcher!