psaucourt
Contributor
Contributor

Performance Statistics are not available after 24 hours

Hello,

When I use the following PowerCLI command, I never get more than 24 hours performance statistics :

get-stat -Entity hostname -Stat mem.usage.average -MaxSamples 900 -Start (get-date).adddays(-7) -Finish (get-date)

I have tried increasing or decreasing the .adddays(x) value but nothing change.
I have migrated from vSphere 4.1 to vSphere 4.1 Update 1 yesterday. As I know, it was working before the migration.
Someone have an idea ?
Thanks.
Pierre
0 Kudos
14 Replies
LucD
Leadership
Leadership

I have seen this happening before, after an upgrade the metrics from the previous version were not available anymore.

Did you check the statistics levels in vCenter ?

Is Historical Level 2 enabled and using a level less or equal to Historical Level 1 ?


Blog: lucd.info  Twitter: @LucD22  Co-author PowerCLI Reference

0 Kudos
psaucourt
Contributor
Contributor

I checked the Oracle database tables (VPX_HIST_STAT[1-4]) and views (VPXV_HIST_STAT_*) and all information is here. The vCenter Server is just unable to display performance statistics atfer 24 hours. I have the same behaviour using Powershell command (get-stat -Entity servername -Stat mem.usage.average -MaxSamples 2000 -Start (get-date).adddays(-3) -Finish (get-date)).

I finally opened a case with VMware.

Let wait and see what they will answer.

0 Kudos
LucD
Leadership
Leadership

I suspect that the current ID of the server doesn't correspond with the ID that is present in the DB tables.

Remember that an ID can change after a vCenter upgrade and in some cases even after a vCenter restart.

You can check the current ID with

Get-VMHost | Select Name,ID

Try to see if this ID corresponds with the one that is in the DB ?


Blog: lucd.info  Twitter: @LucD22  Co-author PowerCLI Reference

0 Kudos
psaucourt
Contributor
Contributor

Thank you for your answer, Luc.

I don't think it's related with the VMware Host ID because we are running for 2 weeks since I upgraded to vSphere 4.1 update 1 and we are not getting any statistics history after 24 hours. If it was, what you have proposed, we will get at least 2 weeks of history now. Are you OK with this ?

0 Kudos
avlieshout
VMware Employee
VMware Employee

I've had the same issue before on an SQL server installation. The problem was that the stats rollup jobs weren't running.

In that specific case we moved the database and forgot to move the rollup jobs.

There are three rollup jobs:

  • Past Day stats rollup
  • Past Week stats rollup
  • Past Month stats rollup

These jobs are responsible to convert the realtime stats (24h) into daily stats. Daily stats are converted to weekly stats, and so forth.

I don't know how things work exactly for Oracle, but there must be rollup jobs somewhere.

Check the existence and status of these jobs in your database.

Arnim van Lieshout Blogging: http://www.van-lieshout.com Twitter: http://www.twitter.com/avlieshout If you find this information useful, please award points for "correct" or "helpful".
0 Kudos
LucD
Leadership
Leadership

That is probably it.

See KB1004382 on how you can check these rollup jobs in Oracle


Blog: lucd.info  Twitter: @LucD22  Co-author PowerCLI Reference

0 Kudos
psaucourt
Contributor
Contributor

Thank you for your answer, avlieshout. We checked the rollup jobs yesterday and they are running fine. To be sure, we recreate it again. As I wrote before, the performance historic data are in the database, we are just unable to get it...

0 Kudos
psaucourt
Contributor
Contributor

Yes, we found this article but it's not applicable to our problem.

Thanks.

0 Kudos
LucD
Leadership
Leadership

Stupid question perhaps, but can you see the performance data from the vSphere client ?


Blog: lucd.info  Twitter: @LucD22  Co-author PowerCLI Reference

0 Kudos
psaucourt
Contributor
Contributor

Neither in vSphere Client, nor with Powershell CLI!

0 Kudos
avlieshout
VMware Employee
VMware Employee

Please make absolutely sure that the rollup jobs are doing there job. I still think that there's something wrong in that area.

Check the size of your VPX_HIST_STAT1.

This table contains the 5min. intervals. Ideally this table should only contain 288 rows for every Counter_ID, which corresponds to 24Hrs. The actual number of rows can grow a bit larger because the rollup job runs every 30 min. so max should be at 294 rows.

I assume that the 5min. statistics are kept for 1 day (default) and you didn't change that in the vcenter server settings.

If this table contains much more records, you Past Day Stats Rollup job is not doing it's work. It's supposed to compress 5min. intervals to 30min. intervals and store them in the VPX_HIST_STAT2 table and delete the correspinding 5min. intervals from te VPX_HIST_STAT1 table.

Just to be sure 😉

Arnim van Lieshout Blogging: http://www.van-lieshout.com Twitter: http://www.twitter.com/avlieshout If you find this information useful, please award points for "correct" or "helpful".
0 Kudos
LucD
Leadership
Leadership

I suspect you mixed up the table names.

The tables VPX_HIST_STATn contain links to other tables in the vCenter database and there can be thousands of rows in these tables.

The tables called VPX_SAMPLE_TIMEn are probably the ones you meant.

Indeed VPX_SAMPLE_TIME1 should have at a maximum around 294 rows.


Blog: lucd.info  Twitter: @LucD22  Co-author PowerCLI Reference

0 Kudos
psaucourt
Contributor
Contributor

The VMware Support finally gave me the correct solution. We applied the following KB : 1032755.

We have lost history data but now it works.

0 Kudos
avlieshout
VMware Employee
VMware Employee

Thank you for sharing.

Glad to hear that your issue is resolved.

At least nice to know that I was close that it had to do with the population of the tables.

Arnim van Lieshout Blogging: http://www.van-lieshout.com Twitter: http://www.twitter.com/avlieshout If you find this information useful, please award points for "correct" or "helpful".
0 Kudos