VMware Cloud Community
nabeelsayegh
Contributor
Contributor

Automate the disabling of VM Logging using PowerCLI 5.1

I am experiencing the log spew issue with ESXi 5.1 and I want to implement the documented workaround of disabling VM logging because it is blowing up my datastores. Problem is, I got a couple of thousand VM's to do this to and I am looking for a quicker way of doing it instead of opening up the settings on every VM and making the change manually.  Any help on how I can go about doing this using PowerCLI 5.1 (or other methods that will allows to rapidly make this change?)

Reply
0 Kudos
8 Replies
MKguy
Virtuoso
Virtuoso

I suppose you are referring to the issue described here? http://kb.vmware.com/kb/2036350

This issue was supposedly fixed in the VMware Tools patch ESXi510-201212402-BG released in December:

http://kb.vmware.com/kb/2035778

Luc also provides a great script to deal with VM logging (vmware.log) settings here:

http://www.lucd.info/2011/02/27/virtual-machine-logging/

-- http://alpacapowered.wordpress.com
Reply
0 Kudos
peetz
Leadership
Leadership

The log spew issue with the VMware Tools 5.1 will fill up the Windows event log inside the guest OS, but not the VM's log file on the ESXi host.

So this would not fill up any datastores (unless the VMs are thin provisioned and the growing Windows event log will consume a lot of disk space, but I don't think so ...)

So what log spew issue are we talking about?

Twitter: @VFrontDe, @ESXiPatches | https://esxi-patches.v-front.de | https://vibsdepot.v-front.de
Reply
0 Kudos
MKguy
Virtuoso
Virtuoso

The KB article actually mentions flooding the vmware.log file on the datastore too:

In the vmware.log file for the virtual machine, you see log spew of this error:

GuestRpc: Channel X reinitialized.

Though I have to agree that it seems a bit extreme if this ends up filling your datastores.

Hint: Use the log.keepOld and log.rotateSize VMX-parameters to limit the logging volume to some sensible degree (mentioned in the vSphere Hardening Guide too).

-- http://alpacapowered.wordpress.com
Reply
0 Kudos
RParker
Immortal
Immortal

MKguy wrote:

The KB article actually mentions flooding the vmware.log file on the datastore too:

In the vmware.log file for the virtual machine, you see log spew of this error:

GuestRpc: Channel X reinitialized.

Though I have to agree that it seems a bit extreme if this ends up filling your datastores.

Hint: Use the log.keepOld and log.rotateSize VMX-parameters to limit the logging volume to some sensible degree (mentioned in the vSphere Hardening Guide too).

Sounds like a knee jerk reaction.. logs are not filling up.. unless they are so low on disk space a few KB from a VM may push them over the edge.  Not sure what "blowing up" datastores means, is it performance related or space only?  Maybe the OP ASSUMES (incorrectly) log files creates IOPS and that's what they are looking at.

Either way, logs are not the problem.

Reply
0 Kudos
nabeelsayegh
Contributor
Contributor

I have an open ticket about this issue and I was told that this issue is not fixed until vmtools relase 5.1a. As of yesterday, there was no ETA on that release.  I am running build 914609 but the problem still exists not only in the Guest OS but on the ESXi vmware.log side of things as well.  We have never seen our vmware.log files go anywhere beyond 5-10MB in size between VM shutdowns.  Since upgrading to 5.1 (build 914609) our vmware.log sizes have gone in the upwards of 500MB in size.  In addition, the application event logs (in the guest OS) are getting flooded as well with misc errors like the following:

[ warning] [vmusr:vmusr] Channel restart failed [1]

The vmware.log is filling up with these errors (which post once per sec for every VM):

2012-12-23T21:21:06.552Z| vcpu-0| I120: GuestRpc: Channel 2 reinitialized.
2012-12-23T21:21:06.552Z| vcpu-0| I120: GuestRpc: Channel 2 reinitialized.
2012-12-23T21:21:06.552Z| vcpu-0| I120: GuestRpc: Channel 2 reinitialized.
2012-12-23T21:21:07.553Z| vcpu-1| I120: Guest: toolbox-dnd: Version: build-913578
2012-12-23T21:21:07.553Z| vcpu-1| I120: GuestRpc: Channel 2, conflict: guest application toolbox-dnd tried to register, but it is still registered on channel 3

As a point of clarification, the cluster/VM's that are experiencing this issue with are Windows Terminal servers.  We do not see the same behavior on other servers (to the extend we are seeing it on our app servers).  With this said, we absolutely are running our of space on our datastores if logging is being increased 100 fold on every VM and I have over 600 of them running on this cluster. By my calculations (on average), since the last power off/on we have done back on 1/6/2012...the size of our vmware.log file grows by 30MB per day, per VM.  30x600 = 18GB per day, and about100GB a week (assuming light weekend workload).  Normally, the logging configuration is not an issue and I am not looking to change it.  This is a VMware problem that needs to be fixed and in the mean time, I am willing to just shut it off until the fix is available.  Less rework for me to do...hence the request on how to turn off VM logging accross the VM's quickly.  Once the patch is in place, i can simply turn it back on.  (Note: It is a problem and creates a bunch of noise, but my concern is not about the guest OS application event logging, they are contained to 16MB in size by GP; the vmware.log filling up is what's killing me.)

Reply
0 Kudos
peetz
Leadership
Leadership

Thanks for clarifying this. So this really looks like an issue, and I understand your decision to turn off VM logging until it is fixed.

The PowerCLI script that you have been pointed to will do the job.

Please note: You do NOT need to turn off the VMs to make the change. You can run the script against powered-on VMs, and to make the change effective you can just vmotion them to another host in the cluster (instead of power-cycling them).

- Andreas

Twitter: @VFrontDe, @ESXiPatches | https://esxi-patches.v-front.de | https://vibsdepot.v-front.de
Reply
0 Kudos
nabeelsayegh
Contributor
Contributor

I tried to load the run the powershell script noted above, but that is an older post and the script seems to have issues running in a ESX/PowerCLI 5.1 environment. I (admittingly) do not know enough about powershell and/or PowerCLI to modify it to work correctly (but it certainly looks to do what i need).  Any superstars out there that can have a look at the post and see what might need to change to get it to work?  OR if there is an alternative method with the vsphere cli (although i kind of doubt it.)

Reply
0 Kudos
peetz
Leadership
Leadership

Does this thread help?

http://communities.vmware.com/thread/259379

Twitter: @VFrontDe, @ESXiPatches | https://esxi-patches.v-front.de | https://vibsdepot.v-front.de
Reply
0 Kudos