Skip navigation
VMware

This Question is Possibly Answered

1 "correct" answer available (10 pts) 2 "helpful" answers available (6 pts)
4,636 Views 44 Replies Last post: Mar 11, 2011 3:14 AM by deejay.dfc@ka… RSS
1 2 3 Previous Next
steveh2 Hot Shot 10 posts since
Nov 16, 2006
Currently Being Moderated

Aug 3, 2009 8:56 AM

Bug? VM's CPU usage increasing over time?

We use NAGIOS to graph the CPU consumed by all of our VM's, it's very apparent that over time, with no activity on the VM, the amount of CPU consumed per 5 minutes increases?

 

This is on VM's running both Linux and Windows.

 

 

Any ideas on this, has anybody else noticed it?

Attachments:
mmarcha Lurker 2 posts since
Aug 17, 2009
Currently Being Moderated
1. Aug 17, 2009 10:50 AM in response to: steveh2
Re: Bug? VM's CPU usage increasing over time?

 

I have the exact same behavior, with the same load, my vm's CPU usage on the host increase all the time. After two month it is now 5x the normal amount. Shutting down the vms and restarting them bring back a "normal" CPU usage. Restarting the guest OS is not enough, the vmware guest process needs to be stopped and a new process created.

 

 

There is defenetively something very wrong here.

 

 

RHEL4 host with Linux and Windows server 32bits guests (1 CPU per guest).

 

 

 

 

 

Antoninio Novice 24 posts since
Jul 29, 2009
Currently Being Moderated
2. Aug 18, 2009 7:47 AM in response to: steveh2
Re: Bug? VM's CPU usage increasing over time?

 

Hi,

 

 

I'm not sure I understood well how use monitor your VM... Does nagios look the cpu time consumed by the vm thread or does it look at the global cpu consumed inside a vm?

 

 

I'm running long time vms and just begin to migrate from VMS 1 to VMS2... So this worries me a bit:)

 

 

robgolding63 Novice 18 posts since
Dec 9, 2006
Currently Being Moderated
3. Aug 30, 2009 5:19 AM in response to: Antoninio
Re: Bug? VM's CPU usage increasing over time?

 

I have exactly the same problem. I graph my host's CPU usage with Cacti, and here are my results:

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Has anyone found a solution, or evern a workaround to this problem? I've just bought a replacement CPU for the host server - faster clock speed and Intel-VT is supported, so I am planning to migrate all VMs to 64-bit, to see if that has any effect.

 

 

ferryvanaesch Lurker 1 posts since
Sep 26, 2009
Currently Being Moderated
4. Sep 26, 2009 12:18 PM in response to: robgolding63
Re: Bug? VM's CPU usage increasing over time?

Just wondering if VMWare has picked up on this one yet? This particular bug is becoming unworkable here.

ajm83 Lurker 2 posts since
Aug 20, 2009
Currently Being Moderated
5. Sep 29, 2009 1:14 AM in response to: ferryvanaesch
Re: Bug? VM's CPU usage increasing over time?

Still seeing this here as well. Here are our Nagios graphs for CPU usage. (Where it goes back to zero are when we have rebooted the machine - the missing data in Sept. is when we tried disabling Nagios)

 

Note that the number of VMs has not increased, nor has the load placed on the individual VMs.

 

 

 

 

 

 

 

 

 

 

 

 

 

mpco Novice 21 posts since
Jul 3, 2008
Currently Being Moderated
6. Oct 6, 2009 4:53 AM in response to: steveh2
Re: Bug? VM's CPU usage increasing over time?

I'm having this problem as well, as seen in this thread:

 

http://communities.vmware.com/thread/217345?tstart=0

 

This problem is becoming chronic, as I have to reboot all of my vmware servers about once a month to keep acceptable performance.  Anyone have any ideas?

mcssoftware Enthusiast 71 posts since
Dec 27, 2006
Currently Being Moderated
7. Oct 15, 2009 6:56 AM in response to: mpco
Re: Bug? VM's CPU usage increasing over time?

Yep.  Same problem here.  When I was on 1.x with the same number of VMs and the same server my host stayed up for months (almost a year!) with no issues. Now with server 2.x it looks like I may have to hard reboot  (power on/off) the VMs every 2-4 weeks as my CPU usage starts climbing dramatically.  This is the only fix that works.  The funny (or frustrating thing) is that the VMs themselves CPU usage will report 0-5% or so, but the vmx process on the server reports 25%.  This seems to indicate a problem with vmware server itself.  Sounds like a big bug and reduces my confidence in server. And I used to LOVE vmware server pre 2.x days  

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

BTW my layout is 64bit RHEL 5.x on the host, 16GB RAM, 8 VMs with mix of 32/64bit Windows and Linux.  Both my production and development server show the same symptoms and they are different hardware.

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

One of my graphs (using Zabbix - this is user %CPU over time):

 

 

 

 

robgolding63 Novice 18 posts since
Dec 9, 2006
Currently Being Moderated
8. Oct 15, 2009 10:56 AM in response to: mcssoftware
Re: Bug? VM's CPU usage increasing over time?

 

Hmmm, this is getting really frustrating now.

 

 

I'm just about to move my last VM over to 64-bit, in the hope that that will cure the problem. I'm not entirely hopeful that it will, though.

 

 

 

 

 

Rob

 

 

mcssoftware Enthusiast 71 posts since
Dec 27, 2006
Currently Being Moderated
9. Oct 15, 2009 2:59 PM in response to: robgolding63
Re: Bug? VM's CPU usage increasing over time?

Please let us know your results.  It would interesting if the problem was a result of mixing 23/64 bit guests - not a whole lot of help for those of us that have to run 32 guests, but at least we have a possible culprit.

 

 

To vmware :  Is there any official method of reporting bugs to vmware besides hoping they see this in the forums?  I'll gladly provide a detailed report if it helps get this fixed.

LucasAlbers Expert 563 posts since
Mar 29, 2005
Currently Being Moderated
10. Oct 15, 2009 6:20 PM in response to: mcssoftware
Re: Bug? VM's CPU usage increasing over time?

 

I think you need a support contract to have a chance of them fixing your bug in a reasonable timeframe.

 

 

I think our vmware server support contract was like 300 bucks or something cheap.

 

 

mpco Novice 21 posts since
Jul 3, 2008
Currently Being Moderated
11. Oct 20, 2009 6:39 AM in response to: LucasAlbers
Re: Bug? VM's CPU usage increasing over time?

It doesn't seem like this problem is isolated enough to warrant having to pay to fix it.  It seems like everyone's problem.  Are you not having this problem, LucasAlbers?

HelgeAlg Novice 10 posts since
May 22, 2009
Currently Being Moderated
12. Oct 21, 2009 11:36 PM in response to: steveh2
Re: Bug? VM's CPU usage increasing over time?

 

I have seen similar problem, but in my case it's the memory that's reaching the end.

 

 

I have a system with 32 GB memory and only 5 Active Clients.

 

 

After a while (1 Week) all the memory is used, but when I check the memory us of the clients it doesn't add up?

 

 

Is the CPU using you all see contributed to swapping? Perhaps we all have the same error, and it's a memory leak somewhere?

 

 

 

 

 

   

 

 

ajm83 Lurker 2 posts since
Aug 20, 2009
Currently Being Moderated
13. Oct 22, 2009 1:10 AM in response to: HelgeAlg
Re: Bug? VM's CPU usage increasing over time?

HelgeAlg, I don't think that issue is related (or at least we do not see it here). Memory usage remains consistent for us at 99% almost all the time.

 

 

 

CPU on the other hand continues to grow at a constant rate.

 

 

 

 

You can see at the start of week 41 we stop one VM, another at the end of week 41 and yet another at the start of week 42. And yet, growth continues at a rate of around 10% a week.

 

 

This is with no load change on the individual VMs at all!

LucasAlbers Expert 563 posts since
Mar 29, 2005
Currently Being Moderated
14. Oct 22, 2009 9:23 PM in response to: mpco
Re: Bug? VM's CPU usage increasing over time?

 

Of my 4 vmware 1.0 server's  and 4 gsx 3.2 server's, I have not encountered this problem.  They get rebooted once every 1 to 2 years, when we have catastrophic power failures caused by a hardhat digging through the electrical grid.

 

 

Of my two vmware 2.0 servers's they run client vm's that get started and stopped on a weekly basis, so they don't run long enough to encounter this issue.

 

 

Can't you restart the vm server service without restarting the vm's? I remember kicking the service when the hostd or what not vmware host service was high, without restarting the vm's.  At that point does the cpu utilization for the server drop?

 

 

This is tricky issue to resolve, I think the only method that jumps to mind is to compare the timings of an strace run of the host process.

 

 

But even then the information is too coarse to be of much use.

 

 

 

 

 

Bookmarked By (0)

Share This Page

Communities