Retrospectively enabling archiving

[DaveMcCloud]

As is often the way our pilot deployment went live.

We didn't have archiving for loginsight on day one and over time have been throwing more storage at the appliance.

Now we have an NFS server to enable archiving on.  I realise that archiving only copies buckets over as they fill and my understanding is it will not copy over my last year worth of buckets - is that correct?

My plan was to simply export the first X months worth of data using the export to NFS method.  However what I don't know is how to then remove that X months worth of data from the live loginsight instance.

Any ideas or am I simply understanding it wrong?

[Cederberg]

Hi. 

VMware Log insight saves events in what they call buckets, which is 0,5GB in size. When the bucket is full it will seal it self and become read only. If you have achiving enabled at the time the bucket will be marked for achiving and copied to the NFS server. Then the bucket will remain on the log insight storage until it is aged out either by the Log Insight running out of space or if you have set any retention periods with partitions.
https://docs.vmware.com/en/vRealize-Log-Insight/8.4/com.vmware.log-insight.getting-started.doc/GUID-...

There is also a good explanation on this blog but it's a bit old so the specifics might not be 100% acurate but it explains how it works.
https://sflanders.net/2015/07/01/log-insight-system-architecture-part-3-archiving/

From my understanding you can't remove events from an log insight appliance manualy. It will age out it's data with the first in first out principle if you are not using partitions to give your data different retention periods.
Regards
//Cederberg

[DaveFromWales]

.

[DaveMcCloud]

That all ties in with my understanding. What I'm not sure about is what happens with the 100s of buckets that are already full and sealed.  Will they be the first ones to age out? I'm hoping from what you say is changing the retention period will archive ALL of the oldest buckets.

[Cederberg]

From what i have read there is no way of archiving a sealed bucket. Thats because a bucket only gets marked for archiving when its beeing sealed it can't be done afterwards. The only difference for a bucket that is marked for achiving and one that is not are that the archiving one will be copied to nfs as soon as possible after it beeing sealed. Both remains on the appliance and is searchable until it ages out and gets deleted to make space for newer events. To be clear Archiving doesn't delete anything it copies it to NFS to be saved and imported later if the data should be needed.

The retention period of the Partitions will not help you with archiving. It will only partition the data from and then save them for different amount of time. Say that you have two Partitions. On that collects informational logs and one that takes the rest. If you set the retention period for the informational logs partition to two weeks it will age out any bucket in that partition after two weeks or if the log insight appliance runs out of diskspace. The other partition could have another retention period or not anyone defined and the buckets in that partition will age out according to that.

[Cederberg]

On this link there is a better explanation of Log Insight storage.

https://blogs.vmware.com/management/2020/05/vrealize-log-insight-index-partitions-and-variable-reten...

[DaveMcCloud]

that was my thinking.  I don't know if vmware have a method so might have to log a ticket