3 Replies Latest reply: May 18, 2009 4:57 PM by kjb007 RSS

    High CPU utilization w/Debian guests

    johnphilips42 Lurker

      Hi,

       

      I'm running ESXi 3.5 (158874) on a Dell PowerEdge 1950 w/two quad-core 2.0GHz CPUs and 32GB RAM.

       

      There are 18 64-bit VMs running Debian 5.0.1 (lenny).  Each VM has 4GB of RAM allocated to it, some have 2 vCPUs and some have 4.  This is just temporary until I get additional hardware purchased, then I'll spread out the VMs.

       

      It seems to be running ok right now, but the CPU load is much higher than expected.  Each VM averages 90-180MHz and the host CPU usage runs between 2.5-4.5GHz consistently.  The host CPU usage is always 1-2GHz above the sum of each guest's CPU usage.  See attached pics.

       

      These VMs aren't doing any production work yet, it's a fresh base install.  The only apps running are rsyslog, ssh, snmpd and ntpd.

       

      I installed open-vm-tools 2008.11.18 and VMware recognizes it just fine.  I used these kernel parameters:

       

      notsc clocksource=acpi_pm nohz=off highres=off

       

      Things I've tried:

       

      1.  Looked through the logs.  No errors, nothing suspicious.

      2.  Removed open-vm-tools and manually installed VMware tools.  Didn't make a bit of difference.

      3.  Installed Ubuntu.  The CPU utilization numbers were lower, but not much expected.

      4.  Switched from 4 vCPUs to 1.  That definitely brought down the load.  Each VM averaged 30MHz instead of 90-180MHz.

       

      Anyway, is this the expected behavior?  Specifically, when a Linux guest is doing essentially nothing, should it be using >100Mhz?

        • 1. Re: High CPU utilization w/Debian guests
          kjb007 Guru

          When you have a high SMP value, the host has to schedule and manage all of those vCPU's.  So, vm's typically perform better as 1 or 2 vCPU machines.  Even at idle, those vCPUs have to be scheduled.  You can look through esxtop on your host, and see how  your vm's are using their CPUs from a host  perspective.

           

           

           

           

          -KjB

          VMware vExpert

          • 2. Re: High CPU utilization w/Debian guests
            johnphilips42 Lurker

             

            kjb007,

             

             

            I changed all the VMs to have 1 vCPU instead of 4.  Now each one averages 40MHz of usage, which is much better than before.  However, the total CPU usage of the host is still incredibly high.  It ranges from 1GHz to over 6GHz.  This doesn't correlate at all.  Please look at the new screenshots, they were taken at the same time.  Adding up the CPU usage for each VM, it comes to 597MHz, while the host CPU usage is reported as 5165MHz, a difference of nearly 10 times!

             

             

            I ran esxtop as you suggested but don't really know what to look for.  There are several columns which aren't in the documentation and I couldn't find on google.  Attached is some sample output.

             

             

            • 3. Re: High CPU utilization w/Debian guests
              kjb007 Guru

              The esxtop confirms what you see from the vm side, 'PCPU(%):   3.97,   8.90,   5.12,   7.34,   1.44,   1.98,   5.48,   7.78 ;   used total:   5.25'  Your pCPU utilization is low.  However, there is a high %WAIT stats on several processes, and low %IDLE,  meaning, they are spending a lot of time waiting on some I/O to complete.  Seems I have similar numbers for some of the processes, but my utilization reports normal.

               

               

               

               

              -KjB

              VMware vExpert