VMware Modern Apps Community
rnatarajan
Contributor
Contributor
Jump to solution

is there any possibility to schedule cron job in tanzu worker nodes to cleanup root FS?

even though I separated /var/lib/container from the /root. the worker nodes root filesystem always getting filled and disk pressure issues happen to Tanzu worker nodes. 

most of the time   /var/log/sgidlist is filled with huge files

 

 

 

Reply
0 Kudos
1 Solution

Accepted Solutions
jachintya20
VMware Employee
VMware Employee
Jump to solution

You can schedule a cron job in Tanzu worker nodes to clean up the root filesystem and address disk pressure issues. The goal is to regularly remove unnecessary files and reclaim disk space to avoid resource constraints.

Here are the possible steps you can take to set up a cron job for cleanup on Tanzu worker nodes:

  1. Create a Cleanup Script: First, you need to create a script that performs the cleanup tasks you want. This script should include commands to remove unnecessary files or folders, such as the large files in /var/log/sgidlist

    For example, a simple cleanup script could be:

    #!/bin/bash
    find /var/log/sgidlist -type f -mtime +7 -exec rm -f {} \;

    You can change it accordingly.
  2. Make the Script Executable:  chmod +x /path/to/cleanup_script.sh
  3. Test the Cleanup Script and Schedule the Cron Job using "crontab -e"
  4. Add a line at the end to specify the schedule for the cleanup script. For example, to run the script every day at midnight, add the following: 
    0 0 * * * /path/to/cleanup_script.sh

Keep in mind that modifying the cron job and running cleanup scripts directly on worker nodes might not be the ideal approach in a managed Kubernetes environment like Tanzu. In production environments, it's recommended to use a centralized logging and monitoring solution to handle log files more efficiently, and leverage Kubernetes-native features such as log rotation to manage logs.

Additionally, consider configuring log rotation and log retention policies for /var/log/sgidlist and other log directories to prevent them from filling up the disk space. Properly managing logs is essential to ensure the smooth operation of your Kubernetes cluster.

View solution in original post

Reply
0 Kudos
3 Replies
jachintya20
VMware Employee
VMware Employee
Jump to solution

You can schedule a cron job in Tanzu worker nodes to clean up the root filesystem and address disk pressure issues. The goal is to regularly remove unnecessary files and reclaim disk space to avoid resource constraints.

Here are the possible steps you can take to set up a cron job for cleanup on Tanzu worker nodes:

  1. Create a Cleanup Script: First, you need to create a script that performs the cleanup tasks you want. This script should include commands to remove unnecessary files or folders, such as the large files in /var/log/sgidlist

    For example, a simple cleanup script could be:

    #!/bin/bash
    find /var/log/sgidlist -type f -mtime +7 -exec rm -f {} \;

    You can change it accordingly.
  2. Make the Script Executable:  chmod +x /path/to/cleanup_script.sh
  3. Test the Cleanup Script and Schedule the Cron Job using "crontab -e"
  4. Add a line at the end to specify the schedule for the cleanup script. For example, to run the script every day at midnight, add the following: 
    0 0 * * * /path/to/cleanup_script.sh

Keep in mind that modifying the cron job and running cleanup scripts directly on worker nodes might not be the ideal approach in a managed Kubernetes environment like Tanzu. In production environments, it's recommended to use a centralized logging and monitoring solution to handle log files more efficiently, and leverage Kubernetes-native features such as log rotation to manage logs.

Additionally, consider configuring log rotation and log retention policies for /var/log/sgidlist and other log directories to prevent them from filling up the disk space. Properly managing logs is essential to ensure the smooth operation of your Kubernetes cluster.

Reply
0 Kudos
rnatarajan
Contributor
Contributor
Jump to solution

Hi Jachintya

 

Thanks for your reply. 

this cronjob may valid for only to that worker node, I want to create cron job for all the worker nodes when they create newly,

 

please let me know if we can achieve this by editing kubeadmconfigtemplate ?

 

regards 

Rajesh Natarajan

 

Reply
0 Kudos
jachintya20
VMware Employee
VMware Employee
Jump to solution

Yes you can achieve it by kubeadmconfigtemplate, but to to ensure that the defined kubeadmconfigtemplate is applied to all new worker nodes, you need to make sure it is referenced in a MachineDeployment.

With this, every time a new worker node is created through the MachineDeployment, the specified cron job script will be copied onto the node, and the script will run according to the cron schedule you defined.

Reply
0 Kudos