VMware Cloud Community
dgrace
Enthusiast
Enthusiast

VM Tools using 14% CPU after Tools update (ESX 4.1U2)

I'm running 4.1 Update2 on all my hosts. I updated a bunch of Win2k3 (32 and 64bit) VMs and that seemed to go ok. Next day it was noticed that in those VMs vmtoolsd.exe was eating up 14% of a CPU constantly (averaging to 7% on a dual cpu, etc). Uninstall/reinstall Tools and VM rebooting does not clear up the issue. I opened a ticket but no luck so far.  They are working on getting me info on deeper level logging.

I have an older test Win2k3 VM that wasn't up to date on Windows Updates. I made a snapshot, did the Tools update on it and did not see the CPU issue. I reverted on the snapshot, ran Windows Updates and upgraded Tools but it was still fine.

I do have a Win2k8 VM that is updated and running fine.

Any thoughts on what else to check on? Thanks Smiley Happy

Reply
0 Kudos
10 Replies
VIVector
Contributor
Contributor

Same problem here.

After update to VMwareTool V8.3.12 on a Server 2003 SP2 (32bit) I notice the 14% CPU load from vmtoolsd.exe.

But this happens only with a Apache Tomcat 6 service installed.

If I start the Apache service and one client makes a query on the DB, the vmtoolsd.exe process goes permanent to 14%.

Kill the tomcat6.exe process, vmtoolsd.exe goes to 0%.

Vice versa, too.

Any ideas ?

Bug or Feature ?

dgrace
Enthusiast
Enthusiast

Using VIVector's idea of a service conflict, I start stopping what services I could. I found the culprit: Ultra VNC 1.0.5.6 32bit (we also run the 32 bit version on  our 64bit VMs). Each of the affected servers has it. I tested it on two VMs and saw the CPU usage by Tools only when the uvnc service was active.

I installed the latest verison of UVNC and still had the same issue. Uninstalled it and installed Real VNC and things were fine.

So it seems to be a conflict with Ultra VNC and the 4.1 Update 2 Tools. Didn't  see this issue under the last versions of Tools under 4.1 Update 1. Reported findings to VM support.

Thanks VIVector. The problem isn't solved yet but at least there is a lead now.

EDIT: Oh, I also ran a problematic VM through Converter to Workstation format and opened it using VM Player. Without further updating tools I still saw the CPU usage. Once I did the Tools update for VM Player the problem went away.

Reply
0 Kudos
Gadgetgeek
Enthusiast
Enthusiast

If someone gets an official answer from VMware please post an update. I am also experiencing this problem but am not running VNC. We do run TeamViewer, but stopping the TeamViewer service does not stop the constant 7% CPU usage by VMTOOLS.

I notice this doesn't happen on every one of my virtual machines, so I'm not sure what the problem is.

Thanks!

Geoff

Reply
0 Kudos
dgrace
Enthusiast
Enthusiast

End of December: Support said the bug was found and the patch would be out "shortly". Still waiting on it.

1/31 EDIT: New patches came out today. I don't see mention of this issue. I updated my ticket asking about the patch.

Reply
0 Kudos
Gadgetgeek
Enthusiast
Enthusiast

One of the two ESXi patches that I installed last week said that an updated VMTools was available. I am not prompted for any VMTools upgrades, however, after installation. I manually installed the tools on one of my affected machines but saw no difference.

By the way -- some machines that were using 7% before the patch upgrade are now using 14%. These are single CPU vms, not 7% per CPU. Some vms are still using 7%.

Many of these machines were built from the same template, and are at the same patch levels. I do not understand the logic behind which machines are affected. In one case, I have a 3-server application where all three server VMs were made at the same time from the same template. All at the same OS and patch level. 1 of the VMs out of the 3 is affected at 14%. That particular VM is running a SQL Server Reporting Server. The sharepoint portal server and application server are not affected. I do have a duplicate of this environment for PROD and DEV, and in both environments the SSRS servers are affected.

Roughly 10% of my machines are affected by this (7 out of 69 VMs.)

Reply
0 Kudos
Gadgetgeek
Enthusiast
Enthusiast

Do you have a ticket number I could reference when calling them? I want to put in a ticket as well.

Geoff

Reply
0 Kudos
dgrace
Enthusiast
Enthusiast

Got a response:

"No unfortunately the patch was not included in the latest release as it was not cleared through QA testing yet. The following KB article (http://kb.vmware.com/kb/2010732) is attached to this issues bug report to track it's current status, as soon as the patch is released this article will be updated with that information. To be automatically alerted as soon as the patch is available please subscribe to this article."

I haven't gotten to try it out yet.

Reply
0 Kudos
Gadgetgeek
Enthusiast
Enthusiast

I also found that article this morning while searching. The problem says that if the clock timer interval is set too low, then you will have this problem. This does seem to be the issue. Here is a screen shot from two of my VMs, one affected and one not.

clock res.jpg

What I want to know is, how does this clock interval get set? It's not something that I specifically changed for any of my virtual machines. Can I safely change it to some other setting?

Geoff

Reply
0 Kudos
Gadgetgeek
Enthusiast
Enthusiast

After looking up a few references online, the high-resolution CPU timer interrupt setting may be something that the applications set on the OS when running. Discussions for setting options for Java applications are in this document http://www.vmware.com/resources/techresources/1087

The applications I am seeing this on are:


- GE Cimplicity (SCADA system for controls). I can understand the need for high-resolution timer control, some control points are highly monitored. Uses Java.
- Zetafax with a network-based fax board. Rapid communication with the board is probably a requirement to not lose a modem connection.
- Symantec Endpoint Protection server. Running a Java database POSTGRES. 
- Asigra backup software. Java required.
- Microsoft SQL Reporting Services. No java installed.

The VMware KB article http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=201073... suggests installing the prior version of the VMware tools. This did resolve the problem in my testing.

I had to:
1. uninstall the current version of VMware tools.
2. Reboot. Windows said it found new hardware and prompted for a reboot, but I didn't. Display resolution was set to 640x480.
3. Set the graphics acceleration to Full

4. Run install from the VMware tools ISO available for download at http://packages.vmware.com/tools/esx/4.1p03/windows/index.html
5. Reboot. This reboot took a VERY long time. Sat at the "Windows Server 2003" startup screen for several minutes, and then was several more minutes before the Press Control-Alt-Delete screen came up.

After logging in the system seems normal--VMtools not using high CPU, and the system is responsive as normal. I just have a exclamation prompt to update my VMware tools in my system tray.

Hope this information helps someone.

Geoff

Reply
0 Kudos
Gadgetgeek
Enthusiast
Enthusiast

I upgraded to vSphere 5 this weekend and upgraded the tools in the virtual machines that were having issues. There are no problems with VMware Tools consuming CPU power like in ESX 4.1.

Geoff

Reply
0 Kudos