VMware Communities: Possible issue found? VirtualCenter performance graphs

Up to Discussions in VMware vCenter™ Server

This Question is Possibly Answered

1 "correct" answer available (10 pts) 2 "helpful" answers available (6 pts)

1 2 Previous Next

26 Replies Last post: May 8, 2008 6:10 AM by icor

Possible issue found? VirtualCenter performance graphs

Feb 5, 2008 1:51 PM

jterpstr 36 posts since
Jul 7, 2005

I'm starting to get tired of VirtualCenter's performance graphs not working properly. I've had numerous tickets open with VMware and they can never seem to find the solution to my problem. I truncate the data, and it all works for a while, then stops working.

I started to investigate packets that are going to our Oracle database server (we keep VC and the DB on separate servers) and I noticed the FIRST select statement that is run when a performance tab is clicked is a "select * from vpx_hist_stat", a table with 13.7Million rows. I've ran this a few times and it appears to be consistantly the first query after I select the tab.

Is this possible? Is VirtualCenter really doing this? If thats the case its no wonder VC performance keeps getting slower and slower until it eventually stops working.

I'm hoping someone from the VC team is watching this thread, as first level support doesn't seem to be getting me anywhere with this.

Reply Re: Possible issue found? VirtualCenter performance graphs Feb 6, 2008 6:34 PM

BenConrad 534 posts since
Mar 20, 2006

I just sniffed the connection between VC and the DB and don't see this statement. Where are you seeing this? In SQL Profiler or a Packet Sniffer? Paste the a few lines before and after, I'd be interested because I think graph performance sucks in general.

Ben

Reply Re: Possible issue found? VirtualCenter performance graphs Feb 7, 2008 8:06 AM

in response to: BenConrad

jterpstr 36 posts since
Jul 7, 2005

I am using Ethereal packet sniffer. Our Oracle database resides on a separate Oracle cluster so I can capture data coming in and out of the system. If I set up ethereal to watch the Oracle Listener port, then click on the performance tab of a VM its always the first statement I see. Yesterday I spent an hour on the phone with first level support showing them this. I can only hope this gets escalated to someone on the VC team to investigate further.

Attachments:

ethereal.JPG (191.4 K)

Reply Re: Possible issue found? VirtualCenter performance graphs Feb 7, 2008 9:29 AM

in response to: jterpstr

BenConrad 534 posts since
Mar 20, 2006

Wow, that's messed up. I on SQL I see 'select sm.id from vpx_sample where sm.sample.time = ......'

FYI, when I do a 'select count (*) from vpx_hist_stat' it usually takes about 60 seconds to return.

For the last few days I've been trimming my SQL database in order to bring it from 50 million to about 20 million rows (60 days of data).

Reply Re: Possible issue found? VirtualCenter performance graphs Feb 7, 2008 12:38 PM

in response to: BenConrad

juchestyle 1,328 posts since
Jun 20, 2005 Moderator

Physical or virtual?

Respectfully,

Matthew

Kaizen!

Reply Re: Possible issue found? VirtualCenter performance graphs Feb 7, 2008 1:29 PM

in response to: juchestyle

BenConrad 534 posts since
Mar 20, 2006

app is virtual, DB is physical.

Reply Re: Possible issue found? VirtualCenter performance graphs Feb 7, 2008 2:34 PM

in response to: juchestyle

jterpstr 36 posts since
Jul 7, 2005

Mine is physical/physical. I have a dual CPU application server talking to a quad CPU RAC environment all over GB networking.

Reply Re: Possible issue found? VirtualCenter performance graphs Feb 7, 2008 9:29 PM

GBromage 414 posts since
Jun 14, 2006

I have the same issue in a physical/physical environment. The DB is on Oracle for Linux.

My DBA has noted to me that there's an intermittent error with stats rollup query, which seems to be timing out a lot or taking a very long time.

My current working theory is that whilst the hour/day/week stats are being aggregated, the old rows aren't being removed from the table and thus the table is growing faster than my overdraft.

Please let me know what support line has to say.

I hope this information helps you. If it does, please consider awarding points with the 'Helpful' or 'Correct' buttons. If it doesn't help you, please ask for clarification!

Reply Re: Possible issue found? VirtualCenter performance graphs Feb 11, 2008 9:13 AM

in response to: GBromage

jterpstr 36 posts since
Jul 7, 2005

GBromage, thats interesting. I also have the same issue. I was provided a 3rd party sql script from VMware a while back to check whether stats rollup was working. It was working for the most part, however the yearly rollup is now marked as failed.

Reply Re: Possible issue found? VirtualCenter performance graphs Feb 11, 2008 10:45 AM

depping 2,786 posts since
Jan 17, 2005 VMware

Noticed the same at a couple of customer sites, it tends to get slow after a while and this might actually explain why...

Duncan
My virtualisation blog:
http://www.yellow-bricks.com

Reply Re: Possible issue found? VirtualCenter performance graphs Feb 11, 2008 12:41 PM

in response to: depping

tupelo.operations 76 posts since
Mar 5, 2007

I am fighting with this issue at this very moment. VMware tech support
has had me update the Virtual Center Server to 2.0.2 Patch 2 but this
didn’t help…still not getting performance data. I was then told to run
the MSSQL purge script referenced in Knowledge Base Article 1000125. I
ran the script and 2 hours later I am still not able to get performance
data. Since it is starting to look like I might have to whack the
current VCDB database…I am thinking it might be just as easy to go
ahead and upgrade to VC2.5. What do you guys think? We are not overly concerned with historic data....we just want to be able to get this data when we need it. I currently have the following:

6 ESX hosts at 3.0.2 61618

2 ESX hosts at 3.0.1 42829

1 ESX host at 3.0.2 62488

Those 2 at 3.0.1 won't be a problem will they?

Grant

Reply Re: Possible issue found? VirtualCenter performance graphs Feb 11, 2008 1:52 PM

in response to: tupelo.operations

BenConrad 534 posts since
Mar 20, 2006

I'd hold off on the VC upgrade until 3.5 patch 1 comes out.

Have you run the VCDB_table_cleanup_MSSQL.sql script? it takes a really long time to run but will clean up the history in your database. I ran it, deleted about 25 million rows (50%) have have see a mild improvement. I think I need to move from statistics level 2 -> 1 and then clean again. The way VMWare implemented graphing in VC is terribly inefficient.

Reply Re: Possible issue found? VirtualCenter performance graphs Feb 11, 2008 2:47 PM

in response to: BenConrad

tupelo.operations 76 posts since
Mar 5, 2007

I did run the sql cleanup script...ran it this morning as a matter of fact. It ran for 2 hours and deleted 27 million rows. I have heard back from vmware support and they want to do a webex sometime to take a look at things themselves. I am assuming you mean VC 2.5 patch 1 instead of 3.5 patch 1 right?

Reply Re: Possible issue found? VirtualCenter performance graphs Feb 11, 2008 3:09 PM

in response to: tupelo.operations

GBromage 414 posts since
Jun 14, 2006

Ben, jterpstr, depping and tupelo - are you running your DB on MS SQL Server or Oracle? If Orcale, on Windows or Linux?

I had wondered if it was an Oracle specific (or VMware's implementation of the scripts on Oracle) problem, but based on what you've said I'm not so sure now.

I hope this information helps you. If it does, please consider awarding points with the 'Helpful' or 'Correct' buttons. If it doesn't help you, please ask for clarification!

Reply Re: Possible issue found? VirtualCenter performance graphs Feb 11, 2008 4:30 PM

in response to: GBromage

BenConrad 534 posts since
Mar 20, 2006

I'm on SQL 2000

1 2 Previous Next

Actions

View as PDF

Up to Discussions in VMware vCenter™ Server