/usr/local/avamarclient folder seems to be full of huge log files. Are these safe to delete? 0 bytes left on /
How could I have prevented this?
I would first collect the log bundle from vdp-configure and then delete some older log files. Also revisit the retention policy or delete some older restore points to reclaim space. Note that deleting restore points would not reclaim the space immediately. It ideally happens during black-out window.
I have exactly the same issue: at the moment I'm playing with it in my test lap and have had to throw away one deployment and started again...
what I've notices is that it doesn't see to free up space when you delete or remove files.
did you get any resolution to this?
Hi,
It is during blackout window, VDP tries to clean-up the garbage(deleted one's). Try increasing your blackout window and check.
at the moment it will not start I get the message No space left on device . if a delete or move file I can see the space used using df -v
Filesystem Size Used Avail Use% Mounted on
/dev/sda2 7.9G 7.8G 0 100% /
so I never get space back on the device :-S
This is actually a known issue with the VDP appliance. If you create an SR with VMware support, we have a fix to resolve this issue.
If you are in a bind, look at /usr/local/avamarclient/var-proxy-? and see if there are log files you can clear up. By default we retain 14 days worth of logs, so you might be able to remove some of the older ones to free up the space.
Once the /root partition fills up, none of the services are going to start, so you will need to restart the appliance after you free up space. Long term fix for this issue will be forthcoming in a future release.
Thanks,
So are the recent logs important? Is it ok to run something like:
rm /usr/local/avamarclient/var-proxy-?/*.log
Also, is the fix too complicated to post here? I could open a support request I just hate doing that for some reason.
Check if this also helps..
The following moves the log directory to /space and creates a link.
Create a file.
touch /tmp/move.sh
Make executable
chmod +x /tmp/move.sh
Paste in the following (below the *s) and execute.
**********************************************************************
#!/bin/bash
# The purpose of this script is to relocate the /avamarclient directory to /space
#
# Instructions: Copy this script to the VDP linux console and run
#
# Version 1.0
#
echo "moving directory"
mv /usr/local/avamarclient /space
echo "creating link"
ln -s /space/avamarclient /usr/local/avamarclient
echo "Script Complete - Please logout, and reboot the appliance using Virtual Center - Restart Guest option"
In the folder /usr/local/avamarclient/var-proxy-?/ which files can safely be deleted to create space?
*.alg and *.log with the same name - oldest ones first?
e.g.
*-vmimagew.alg
*-vmimagew.log
*-vmimagew_avtar.log
for each of the oldest backups ?
Rich -
Yes, those are the safest to delete for the oldest backups.
Also, if you call VMware support, they have a script they can run that will modify the setup of the logging and prevent this from occurring in the future.
Greg
GSparks - Lol - i have a case open with them since friday - cant wait any longer to have no backups....
Thanks for the help - i'll delete the files myself.
Unfortunately it didn't give me space back on /dev/sda2
Dont suppose you know what other files the script deletes ? or have a copy of it? - i'm not getting very far with this....
It looks like although the files are being deleted it is not actually reclaiming the space as available.
I was being too cautious deleting only the oldest log files. Deleted all the logs and it regained enough space.
Restarted VDP and it is back up and running.
Message was edited by: RichElmer
Same exact issue here, of four 2TB vdp appliances one has / completely full. I am trying the workaround with with moving the logs to /space.
Hi,
I had the exact same issue.
Here is what I did (simillary to others):
# ssh to vdp and execute these commands:
find /usr/local/avamarclient/ -name "*.log" | xargs rm -f
dpnctl stop all
... be patient and go for a cofee
mv /usr/local/avamarclient /space
ln -s /space/avamarclient /usr/local/avamarclient
reboot
... be even more patient and go for a beer
Be sure to make a full reboot! Only restarting VPD services with dpnctl start all as suggested here, VMware KB: Connecting the vSphere Data Protection appliance to vCenter Server fails with the error: ... won't work!
---
Now, to be sure that VDP won't be stuck without space again, I put it in our Nagios monitoring system and enabeled disk usage for /, /var and /space:
# ssh to vdp and execute these commands:
# add repo
zypper addrepo -f http://ftp5.gwdg.de/pub/opensuse/discontinued/distribution/11.1/repo/oss/suse/ opensuse-11.1
# check if repo is installed
zypper lr
# install nrpe-client and plugins
zypper install nagios-nrpe-client nagios-plugins
# check if package has been installed
rpm -qa | grep nrpe
# make sure it boots upon system startup
chkconfig nagios-nrpe on
# edit /etc/nagios/nrpe.cfg
vim /etc/nagios/nrpe.cfg
# add/modify these lines:
---
allowed_hosts=<your Nagios server IP>
dont_blame_nrpe=1
command[check_disk]=/usr/lib/nagios/plugins/check_disk -E -w $ARG1$ -c $ARG2$ -p $ARG3$
command[check_procs]=/usr/lib/nagios/plugins/check_procs -w $ARG1$ -c $ARG2$ -s $ARG3$
command[check_swap]=/usr/lib/nagios/plugins/check_swap -w $ARG1$ -c $ARG2$
---
# start nrpe client
/etc/init.d/nrpe start
# allow nagios server to contact VDP nrpe client in the firewall
iptables -I INPUT -s <your Nagios server IP> -p tcp --dport 5666 -j ACCEPT
# go to your nagios server and make sure nrpe check works
/usr/lib/nagios/plugins/check_nrpe -H <your VDP IP> -c check_disk -a 15% 10% /
/usr/lib/nagios/plugins/check_nrpe -H <your VDP IP> -c check_disk -a 15% 10% /var
/usr/lib/nagios/plugins/check_nrpe -H <your VDP IP> -c check_disk -a 15% 10% /space
# make firewall rule permanent
vim /etc/firewall.base
# add these lines just before the "# DROP all other traffic and log it" line:
---
# 14. Allow Nagios server to talk to Nagios NRPE client
$IPT -A INPUT -s <your Nagios server IP> -p tcp --dport 5666 -j ACCEPT
---
Hi There,
The reason for unable to see the free space after removed was the logs already pointed to the destination in web app point of view and if you restart the web services of VDP you should be able to see the claimed space immediately please follow the command.
This problem can be occurred if you try to export bundle zip logs also because bundle logs sometimes more then 2.5 GB (Depend of the period of your VDP access)
File location:
# /usr/local/avamar-tomcat-7.0.54/webapps/ROOT/logBundle/
Actually this is the root location. Verify it by the following command to see the disk space of the current location with (.)
# df -h .
And try the same command from the root / as well, then you can make sure that the location is root actually.
Steps to resolve the issue:
#: find / -name "FinalLogBundle" -print
# cd /usr/local/avamar-tomcat-7.0.54/webapps/ROOT/logBundle/
# mkdir /space/home/bundle-zip
# mv FinalLogBundle1456933403769.zip /space/home/bundle-zip
YOU MUST RESTART THE WEBSERVICE TO RECLAIM THE SPACE VISIBLE
# emwebapp.sh --restart
Thank you for the post SugeeB, saved me tons of time this morning trying to find what was taking up my space on the appliance. A few weeks ago support had me attempt to collect logs and it failed. That failed attempt was taking up 40% of my space!