Upgraded to v4 This Weekend; Odd VMware Tools App Log Entry

[pnewell]

Hi All,

This past Friday night my boss and I upgraded our ESX hosts to v4.0 U1 (we had already upgraded our vCenter the week before due to a database problem).

The upgrading of the hosts went pretty well with only a couple hiccups, but today when I looked at the logs on my servers (a normal weekly task) I find that the vast majority of my guest OSes' Application logs are being flooded with the following two events:

(First event)

Event Type: Information

Event Source: vmStatsProvider

Event Category: Guest Library API

Event ID: 258

Date: 11/23/2009

Time: 2:55:31 PM

User: N/A

Computer:

Description: The "vmGuestLibrary" is successfully initialized for this Virtual Machine.

(Second event)

Event Type: Information

Event Source: vmStatsProvider

Event Category: General

Event ID: 256

Date: 11/23/2009

Time: 2:55:31 PM

User: N/A

Computer:

Description: The "vmStatsProvider" is successfully initialized for this Virtual Machine. WMI namespace: "root\cimv2".

Does anyone have any idea why these are being created? They are merely "Information" entries and thus I am not overly concerned about them, but they are written to the app log every ten minutes, which makes it a little difficult to parse the logs.

I already checked the services and didn't see anything that corresponds to the above items. I searched the registry of one of my servers and found a bunch of entries relating to "C:\Program Files\VMware\VMware Tools\Guest SDK\vmStatsProvider\win32\vmStatsProvider.dll", "C:\Program Files\VMware\VMware Tools\Guest SDK\vmStatsProvider\win32\vmGuestLib.dll", and "C:\WINDOWS\system32\wbem\vmStatsProvider.mof".

I also looked at the various tabs of the VMware Tools SysTray application, but didn't find anything of value there.

Oddly, it seems that the only servers that are uneffected by this cycle are my larger SQL servers.

Any insight would be welcome. Thanks in advance!

[Sreejesh_D]

Can you please tell on which Guest OSs you hit this issues? Please mention the service pack and release levels too.

:+: VCP,RHCE,EMCPA.

[pnewell]

Hi there,

They're all Windows. One of them is Server 2008 SP2 (32-bit). The rest are Windows 2003 Standard/Enterprise with SP2 (most 32-bit with a handful of 64-bit ones).

The only thing that seems to be true across the board are the servers which are running larger implementations of SQL (these entries happen either much less frequently or not at all). Of the SQL servers two are 64-bit Server 2003 Enterprise and one is running 32-bit Standard.

The guests operating properly are due to neither the 64-bit OSes nor Enterprise, however, as I have another 64-bit Enterprise Server running MOSS that suffers from the same behavior.

This is an odd one! 😕

[yogik]

hi,

Could you please tell me from what build or ESX version you did an upgrade. I am trying to reproduce this issue.

[pnewell]

Hi there,

Yes, I upgraded my one host to ESX 4.0, but after having started the update on that one I found out that 4.0 Update 1 had been released, so I did the other two with the 4.0 U1 ISO. The build on those hosts is 4.0.0, 208167.

I can't tell you what the other one had for a build, however, as I ran the installation for the Update that night as well, so all of my hosts are on the same build at this point.

My vCenter and vSphere client are at 4.0.0 build 208111, which is the most current one, to my knowledge.

Thanks!

[yogik]

pnewell

What I wanted to know was, what was the EXITSTING BUILD on the hosts before upgrading to ESX4.X builds.

I needed this info so that I can install the leagacy builds on the hosts and then upgrade to ESX4.0 U1 208167 build and look for the tools upgrade issue.

-Y

[pnewell]

Oh, my appologies; I didn't read it close enough... 😕

According to my boss's notes, we upgraded from ESX v3.5 U3, build 130756.

[renaudtouillet]

Hello,

After a migration from 3.5 to 4 we got the exact same messages on one of our VM (2003 SP1) every 15 minutes. Any updates?

0 2 258 vmStatsProvider N/A SGAR00A The "vmGuestLibrary" is successfully initialized for this Virtual Machine.

0 1 256 vmStatsProvider N/A SGAR00A The "vmStatsProvider" is successfully initialized for this Virtual Machine. WMI namespace: "root\cimv2".

Thanks!

Renaud

[pnewell]

I was out of the office last week, so sorry for the late follow-up...

I setup a support call with VMware and while we haven't come to a conclusion yet (been conversing via e-mail) it seems that this all might be coming about from our network montering system - namely the Average CPU Usage counter.

We changed that particular counter to run every 15 minutes and subsequently the VMs started entering the entries into their App Logs every 15 minutes as well.

Do you use some sort of network/server management system, Renaud? If so, how does it monitor your servers? SNMP? WMI?

[renaudtouillet]

Hi,

It's the weird thing... there's no monitoring at all. So we opened a case with VMware, we send them MS reports and we are waiting.

Thanks for the answer.

Renaud

[pnewell]

How about WMI scripts? Or anything that uses WMI, for that matter? Something in there somewhere is using WMI on your VMs because "root\cimv2" has to do with WMI. As to what, I'm not sure as I am not a WMI expert (or novice for that matter).

All I know is that if you search for WMI you find a lot about the various namespaces, one of which is CIMV2.

[polymis]

We have this too. For us it seems to be caused by our server monitoring software performing a status check, which makes a call to WMI. If anyone can figure out how to disable these messages please share...

[pnewell]

Hi all,

I just got it figured out yesterday - you need to do one of the following:

1. Disable WMI from being used by whatever external source is probing it, or

2. Uninstall the VMware Tools from the effected VMs, then reload them using a "Custom Installation" and UNCHECK the "WMI Performance Logging" option under the "Guest SDK" section.

I had been working with a VMware support engineer on this for the last month or so. We tried a bunch of different things, and the last thing he was able to suggest was to remove that portion of the VMware Tools. Seems to work okay on the couple of VMs I've tested it with thus far, but he also suggested that if they keep appearing after reloading the VMware Tools (sans WMI Perf Logging) you may need to search your server's registry and delete any references to vmStatsProvider, vmGuestLibrary, etc.

Might need to delete the files from %preogramfiles%\VMware\VMware Tools\Guest SDK\vmStatsProvider\ if they are still present after a reboot as well.

Hope it works for you all as well. Good Luck!

[pnewell]

Please see my final comment for the solution that worked for my situation.

[polymis]

Thanks for the update. Not sure if it's worth rebooting all of our VM's (once? twice?) for, but at least we have a solution when we do decide to pull the trigger.

[pnewell]

Actually, as it turns out, if your VMs already have the most-current version of the VMware Tools installed ("most-current" as relates to your vCenter, I suppose), you do NOT need to uninstall the tools (and reboot) first; simply select "VM > Guest > Install/Upgrade VMware Tools" (Interactive install, of course) and you will be given the opportunity to Modify the current install, which on the VMs I've tested have not required a reboot.

[polymis]

Great! I'll try it out.

thanks again

[autoxr]

I just ran into this issue and had to pull out the "WMI Performance Logging" part of the VM Tools (4.1 Update1).  Nothing that I know of changed on the server and it was working without any issues 2 weeks after the P2V.  Any idea of what may have caused this?

[max424ever]

THIS

Delete the files from %preogramfiles%\VMware\VMware Tools\Guest SDK\vmStatsProvider\