Log Insight Master Node disk is full while Other Worker nodes are free

[YahyaZahedi]

Hi

I planned and Installed vRealize Log Insight as Standalone, but after a few months its space became full and in order to keep more logs and prevent retention, I added a new worker node.
While statistics show the other node receives data but my oldest logs remove and as time goes more and more logs removed. I also added a new worker node to complete the cluster configuration of vRealize log insight. I like to know why my logs from the Master node become deleted? in case one of the nodes become full, logs must automatically transfer to the new worker node instead of removing logs from the master node!

The oldest logs that we have currently kept, for example, is 21/07/2020 04:00 AM, but after two hours the oldest backup becomes 21/07/2020 06:00 AM.

[RickVerstegen]

Log Insight has an internel storage managment system that allows the Log Data volume to get to 97% and it then starts to retire the old logs on a first in first out basis.

What node size did you deployed?

If the disk is small and log ingestion rate is so high that the free space (3 percent) is filled out within 1 minute, vRealize Log Insight runs out of disk.

When deploying Log Insight in a production environment, use a minimum of 3-nodes. This provides ingestion HA.

Consider adding storage capacity to the nodes.

[YahyaZahedi]

You right, but I have a cluster with 3 nodes, one master and two workers.

I have enough space in worker nodes, when one of the nodes becomes full the other nodes should take responsibility for storing new logs instead of retiring the old logs. but my logs while the whole cluster has enough space becomes retire

[Cederberg]

Hi.

I believe that the cluster only distributes the current incoming events/logs and not taking the available storage space in to consideration. So if I'm right the master node will continue to age out the oldest logs and the worker nodes will have a higher retention untill they are all at about the same storage usage level.