VMware Cloud Community
odonnellj
Enthusiast
Enthusiast

VPX_SAMPLE_TIME1 growing rapidly

I built a new VC 2.5 install and have been moving servers over gradually. It currently has 5 ESX 3.5 servers and 4 ESX 2.5.x servers running about 25 vm's. I got a call from our dba stating that the VC 2.5 database was growing a lot faster than they expected. It's about twice the size of our VC 2.0 database which has been running for over a year. The VC 2.5 has been running for 1 month.

VPX_SAMPLE_TIME1 is using 750mb out of the total 800mb currently used by the whole database.

Can someone tell me if this is normal, and what size growth should I expect. Or even better, how to reduce this. The VC 2.5 system will be hosting another 7 ESX servers and another 40+ vm's once the migration is finished and I'm concerned the database is growing out of control.

Thanks.

Reply
0 Kudos
15 Replies
odonnellj
Enthusiast
Enthusiast

Grew another 15% in 3 days. According to VC options, I should have about 100-200mb not hitting almost 1GB in a month.

Reply
0 Kudos
odonnellj
Enthusiast
Enthusiast

Well, just to let people know where this is....After two months of navel gazing and foot shuffling, VMware is admitting to a bug that my dba found for them.

This is Oracle DB related. You might want to check your db size, as were growing at around 50mb a day. It doesn't seem like a lot but the database is supposed to be pretty static and we sized it for that case. What is happening is the stats job is constantly adding new rows to a temporary table instead of reusing the free space.

load_stats_proc_oracle.sql script needs to have the append hint removed.

We're also finding problem with constraint errors in the database occasionally.

Version 2.5 of VC has been a disappointment so far for me.

Reply
0 Kudos
donovab
Contributor
Contributor

I'm glad someone else mentioned this because I've been at a client where we're using VC 2.5 with Oracle 10 and I'm having a significant amount of problems with "merge extents" errors in the VC logs. I'm seeing some troubling behavior as my ESX hosts fall in and out of VC with once a day regularity requiring me to run service mgmt-vmware restart to buy 24 hours of time. Perhaps I'll have to consier going to MS SQL for the database. Are you seeing anything similar to this behavior?

Reply
0 Kudos
odonnellj
Enthusiast
Enthusiast

The problems we're having are daily space growth and some jobs failing due to constraint violations although the job errrors are sproadic in the VC database. ESX hosts don't appear to have any issues. This is with Oracle 9i patched to the correct level. We've been using Oralce 9i with VC 2.0 for at least a year and no issues and before that Oralce 8 and VC 1.x for several years and no problems. This release they appear to have not fully understood how Oracle works.

e.g. one of the scheduled jobs fail due to constraint violation:

ORA-12012: error on auto execute of job 47

ORA-00001: unique constraint (VPX.VPX_SAMPLE_TIME2_U1) violated

ORA-06512: at "VPX.STATS_ROLLUP1_PROC", line 222

The job is #47 and performs the following actions:

begin stats_rollup1_proc; Purge_Stat1_proc; end;

Another time we checked:

SQL> select job,substr(what,1,50) what,failures,broken from user_jobs

JOB WHAT FAILURES B

-


-


-


-

47 begin stats_rollup1_proc; Purge_Stat1_proc; end; 0 N

48 begin stats_rollup2_proc; Purge_Stat2_proc; end; 3 N

49 begin stats_rollup3_proc; Purge_Stat3_proc; end; 0 N

Reply
0 Kudos
bpayne
Contributor
Contributor

Did VMWare provide a solution to this bug or a patch? We are seeing the same issues. We have 4 ESX hosts and 35 VMs (Very small environment). Our VPX_SAMPLE_TIME1 table is currently at 11Gb and our VPX_STAT_COUNTER table is at 310Mb.

Thanks.

Reply
0 Kudos
odonnellj
Enthusiast
Enthusiast

VMware support sent us new Oracle SQL scripts to load on our database a few days ago. It appears that the growth problems have beeb fixed. We were growing 50+ mb per day on those tables and that has stopped. My DBA had being doing periodic re-orgs to shrink those tables back to 0 and did so again before apply the new scripts and things look good now. It's still too early for me to call a complete victory though.

We do have one other problem in which a couple of jobs that are running into constraint violations.

eg.

ORA-12012: error on auto execute of job 49

RA-00001: unique constraint (VPX.VPX_SAMPLE_TIME4_U1) violated

ORA-06512: at "VPX.STATS_ROLLUP3_PROC", line 223

RA-06512: at line 1

It doesn't happen all the time but it sure looks like a bug on their end.

I would suggest you put in a ticket and referenece SR # 1102930301 if your contact is unaware of the problem.

Reply
0 Kudos
bpayne
Contributor
Contributor

Thanks. I was told by VMWare support yesterday that a new release of Vcenter is coming out that has a fix for this. I opened a ticket with them yesterday but the tech didn't seem aware of this issue. I will reference the SR you provided.

Thanks again for your help.

Reply
0 Kudos
odonnellj
Enthusiast
Enthusiast

According to the patch notes in VC 2.5 update 1, this bug has been fixed.

See: Size of the Oracle Database Table Increases Unnecessarily (KB 1004095)

http://kb.vmware.com/kb/1004095

Reply
0 Kudos
Guillir
Enthusiast
Enthusiast

I also have had constraint violated issues running VC25 with Oracle 10.2.0.1.0

ORA-12012: error on auto execute of job 123

ORA-00001: unique constraint (VPXADMIN.VPX_SAMPLE_TIME3_U1) violated

ORA-06512: at "VPXADMIN.STATS_ROLLUP2_PROC", line 223

Is this also an VC bug? Is this also fixed by VC 2.5 U1?

Reply
0 Kudos
bpayne
Contributor
Contributor

I have not noticed any constraint issues. Just the table growth. The U1 corrected this problem for us.

Reply
0 Kudos
odonnellj
Enthusiast
Enthusiast

I have an open ticket regarding the constraint issue. They took a copy of my database a few days ago to investigate it. I've been holding off upgrading to 2.5.1 until they're done looking in case they need anything else. We received the Oracle growth patch separately and have that applied for now.

Reply
0 Kudos
odonnellj
Enthusiast
Enthusiast

We received some Oracle scripts yesterday from Support to try and fix the constraint issues. My DBA applied them and thinks they will resolve the issue. We're going to keep a close on eye on things for a couple days now.

Reply
0 Kudos
Guillir
Enthusiast
Enthusiast

That's a good news. Let us know if those scritps resolve the issue. Could you give a status how it has worked until now? If it apparently have resolved that issue, could you share those scripts with us/me?

Reply
0 Kudos
odonnellj
Enthusiast
Enthusiast

The script fixed the constraint problems. I don't feel comfortable sending them out, in case it caused any problems. However, if you open a support ticket, you can reference my SR # 1110543731 which followed this problem. The replacement script sent was stats_3_rollup.sql. I would hope a patch is released as well.

Regards.

Reply
0 Kudos
Guillir
Enthusiast
Enthusiast

The enviroment wich I'm having that issue is just implemented, about 1 month. There are few production VMs on (more specific 3) and so I don't believe it can cause any hard problem. Nothing that a backup couldn't resolve.

But I can understand you. Feel free to not share those scripts.

Best Regards,

Reply
0 Kudos