How Data Retention Works?

[vkaranam]

Hello Guys

How does the Data Retention works in vC Ops? In the document it says data retention in enterprise is 900 days. But in the vSphere UI we can keep up to 4 years which is default and minimum level. These two doesnt match?

Also there is different resolutions Weeks, Monthly, Quarterly & Annual. say for example if we keep weekly resolution as 12 weeks and monthly as 10, Quarterly as 9 and Annual as 5 years as shown in image how it works? I mean after 12 weeks the data will be removed but we said 5 years to keeep the data. how this is achieved.

Please help me

Thanks

VK

[vkaranam]

Hello Guys,

Any help on this is appreciated.

Thanks

VK

[admin]

Hi,

I am busy finding out 🙂

[admin]

Hi

vC Ops does Rollup stats. Thus 5min to Hourly, Hourly to Daily, Daily to Weekly, Weekly to Montly....you get the point.

So this Weekly roll up stats...how long do you want to keep it ? Default is 12 Weeks...so you can chnage it. Same for Montly, Q and Annualy.

If you make the retention time more...remember to add more disk to the Analytics VM.

[vkaranam]

Hey Hugo,

Thanks for the Answer. Roll up means before 5 mins to an hour vcops rolls up (Compress the stats data?) the data collected for that hour similarly it will roll up all the data collected in the Hours to daily. and daily to monthly,quarterly,yearly ? am i right?

Thanks

VK

[admin · Accepted solution]

Let me be more specific :

1. The colelctor collects the 20sec interval data every 5min

2. This is rolled up into a 5min avg and max value.

3. These 5min values is rolled up to hourly, daily, weekly...ect...for two reasons: Capacity analysis and Dynamic Thresholds calculations which is stored in two diffenent databases.

It does not compress the stats....it gets avg of the stats. (Analysis uses avg to get trends used for capacity forcasting...ect)

[vkaranam]

Hey Hugo,

This makes me more clear. if possible can you specify the database names where the capacity and DT are stored.

Thanks

VK

[admin]

Ideally you should not go a "Play" around with them. 🙂

For educational Purposes : /data on both VM's.

[vkaranam]

Thanks a lot hugo for your info

Thanks

VK

[vkaranam]

Hey Hugo,

The colelctor collects the 20sec interval data every 5min -- From this statement the collector is collecting 3 times per minute which is 15 times for 5 minutes. Doesn't it makes the vcenter Busy? Also will the colelctor collects the data for every metrics at an interval of 20 sec?

Thanks

VK

[admin]

ESXi collects stats every 20sec from the objects registered on the host (VM, datastores,ect) It keeps this info for I think 2 hours.

The vC Ops vCenter Adaptor "connect" every 5min to the vCenter and request this stats from the hosts.

Thus the 15 interval stats (3 per min * 5min = 15 intervals) is collected every 5min of all the objects and all metric's.

To preempt your next question...1000vm's, 270 Datastores, 7 vCenter's will create about 2.2mb of collection data every 5min.

[vkaranam]

Is there any Excel sheet to calaculate the collection size of data as the no of vm's, datastores and vcenters change. or is there any formula?

Thanks

VK

[admin]

No real formula...as this can change per vcenter/datacenters/clusters/datastores/vm's.

The Numbers I gave is a real world example that one of our clients.

[vkaranam]

Hey Hugo,

We are having a discussion and what to find few things.

EVen though the white paper says standard can hold 180 days and enterprise for 900 days but any version of vApp can only analyze 6 months of data from your environment. Is this some thing right?

Also when we keep the data retention as 4 years the vApp uses the 4 years of data only to generate the reports but not for analyzation? Is this right?

Thanks

VK

[admin]

Thought you might ask this 🙂

I wrote a blog on this today : http://www.virtualclouds.co.za/?p=317

Also note that the FSDB holds it data by default for unlimited.

These options will change in 5.6 .. whatch this space as I will write an article on this.

[admin]

For completeness :

There is only one FSDB. Both the vSphere UI and Custom UI uses this DB for DT Calculations.

There is two Capacity Analytic DB's but there are both used by the vSphere and Custom UI.

[vkaranam]

I got it. So the Data Retention applies for FSDB but not for Capacity analytics?

The 6 month period that iam talking applies to the capacity Analytics DB. ie., the capacity analytics analyzes you environment based on 6 month period.am i right?

Now, why they are creating two capacity Analytics DB's for the vsphere UI and Custom UI while both are accessing the both databases.

Thanks

VK

[jddias]

VK,

  The default 6 month retention period is for the metric data (as well as topology and change events) which is stored in the FSDB.  Analytics engines for performance and capacity are performed periodically against this data.  These are two seperate processes - performance is done on the Analytics VM and capacity done on the UI VM.  Each VM has a database for their respective analysis.

  The capacity analysis used to be a seperate product (Capacity IQ) but is now part of the vApp.  This is why the two are seperate.

[vkaranam]

Hey Jddias

So what ever might be the analysis it will be done depending on the past 6 maonths. rest of the period we mentioned will be used to generate historic reports from FSDB.

Thanks for the info,

Thanks

VK