VPX_EVENT_ARG - 1 million events per day

[gbrown2]

I having a problem with vcenter DB.  It is on SQL 2005 and we are getting 1 million rows a day logged into table VPX_EVENT_ARG.

Running vcenter 5.1u1b (full install)

having logging level set at 1 (informational)

225 hosts

2800 vms.

I have a ticekt open with support, but so far their answer is - just purge the table. but that doesn't tell me what is causing the problem.

This has started happening 2 weeks ago

Thanks,

Glenn

[admin]

Hi Glenn,

Has this abnormal growth of of VPX_EVENT_ARG started from the very beginning or after some changes to the environment? Anything new...

I came across this blog post that you might want to look at: Dealing with VCenter 4.1 Database Tables Growth | VMware Support Insider - VMware Blogs (even though it's for 4.1, will be applicable for 5.1 as well). Also, could you share the SR number with VMware, would like to have a look at it.

-rupam

[gbrown2]

Thanks for the reply. this is after something changed, unfortunetly I do not know what changed, so I was hoping support could help me look at the logs to figure out what is generating the events and find what did change. I will have my DBAs go through the posted article to see if we can track down changes as well.

The SR is 13387009610

Just informational, I used the vcenter settings size calulator (under statistics) and for 225 hosts and 3000 vms it is saying my DB shoudl be ~23Gb, but ours is 45GB and growing, I'm assuming the calculator is doing some very basic assumptions, but double is quite off.

Thanks for the reply.

-Glenn

[admin]

Hi Glenn,

I think it would be a good idea to get the post checked out by your DBA. It's not a healthy sign because with purging, we will only be covering the underlying issue.

I did try to look into the logs however seems like it's not the complete vCenter Bundle - Can you do me a favour and get the complete vCenter bundle uploaded to the SR. I will give it another shot...

-rupam

[gbrown2]

Trying to generate a new log bundle, of course filled up my vcenter c:, arg... Here is a screen shot of the report from the SQL query if it helps while I regen the log files.

GB

[MKguy · Accepted solution]

Did you check in the vCenter Event view what kind of events are frequently occurring?

Are you perhaps using HP gen8 hosts with the HP agentless management (AMS) bundle (installed by default with the HP ISO/extensions)? There was a bug that has  been fixed recently by HP with the newest management bundle, which would generate root login events on every host every few seconds, polluting the events view and ultimately consuming a lot of space in the vCenter database. See here:

http://www.virten.net/2013/08/hp-proliant-gen8-agentless-management-floods-esxi-and-vcenter-logs/

http://h20565.www2.hp.com/portal/site/hpsc/template.PAGE/public/kb/docDisplay/?spf_p.tpst=kbDocDispl...

This bug is now fixed with the September release of the HP AMS-vib (hp-ams-500.9.4.0-25.434156):  http://vibsdepot.hp.com/hpq/sep2013/esxi-5x-bundles/

I also observed a similar problem from some 3rd party applications hooking into vCenter, filling logs with constant login messages.

having logging level set at 1 (informational)

This setting is irrelevant to the vCenter Event data stored in the SQL database. The logging level you can set in the vCenter settings only defines the verbosity of the local logfiles (vpxd.log vpxd-profiler.log etc.) which are stored on the filesystem of the vCenter server (C:\ProgramData\VMware\VMware VirtualCenter\Logs\).

[gbrown2]

So after working with a different TSE and doing some digging we found 3 things causing this huge growth.

1) Hp Proliant AMS issue -  fix is to downgrade ams or turn it off (known issue in various communities and above poster, that I just saw)

2) bad nic on 1 host (of 250), was creating 5 event every 3 seconds  -  fix disabled nic

3) local datastore on a single host had a RAID problem - was giving a vmfs datastore error, this created 900K event in 22hrs

The hardest part was finding these. The first 2 showed when you exported the events list from vcenter, but did not show which host was creating the errors, so that was clicking through 250 hosts to find it; The third could not be seen through vcenter and I pulled the vpx_events_arg table from sql into excel to get that one.  Hope this helps other and need to put in a feature request about exporting more info.