VMware Cloud Community
soundstripe
Contributor
Contributor

VDP 2TB image ran out of space on root

/usr/local/avamarclient folder seems to be full of huge log files. Are these safe to delete? 0 bytes left on /

How could I have prevented this?

17 Replies
snekkalapudi
VMware Employee
VMware Employee

I would first collect the log bundle from vdp-configure and then delete some older log files. Also revisit the retention policy or delete some older restore points to reclaim space. Note that deleting restore points would not reclaim the space immediately. It ideally happens during black-out window.

-Suresh
Reply
0 Kudos
David_Cognizant
Contributor
Contributor

I have exactly the same issue: at the moment I'm playing with it in my test lap and have had to throw away one deployment and started again...

what I've notices is that it doesn't see to free up space when you delete or remove files.

did you get any resolution to this?

Reply
0 Kudos
snekkalapudi
VMware Employee
VMware Employee

Hi,

It is during blackout window, VDP tries to clean-up the garbage(deleted one's). Try increasing your blackout window and check.

-Suresh
Reply
0 Kudos
David_Cognizant
Contributor
Contributor

at the moment it will not start I get the message No space left on device . if a delete or move file I can see the space used using df -v

Filesystem                Size        Used        Avail                Use%      Mounted on

/dev/sda2           7.9G             7.8G         0                    100%             /


so I never get space back on the device :-S

Reply
0 Kudos
GSparks
Enthusiast
Enthusiast

This is actually a known issue with the VDP appliance.  If you create an SR with VMware support, we have a fix to resolve this issue.

If you are in a bind, look at /usr/local/avamarclient/var-proxy-? and see if there are log files you can clear up.  By default we retain 14 days worth of logs, so you might be able to remove some of the older ones to free up the space.

Once the /root partition fills up, none of the services are going to start, so you will need to restart the appliance after you free up space.  Long term fix for this issue will be forthcoming in a future release.

Thanks,

-Greg Sparks
soundstripe
Contributor
Contributor

So are the recent logs important? Is it ok to run something like:

rm /usr/local/avamarclient/var-proxy-?/*.log

Also, is the fix too complicated to post here? I could open a support request I just hate doing that for some reason.

Reply
0 Kudos
snekkalapudi
VMware Employee
VMware Employee

Check if this also helps..

The following moves the log directory to /space and creates a link.

Create a file.

touch /tmp/move.sh

Make executable

chmod +x /tmp/move.sh

Paste in the following (below the *s) and execute.

**********************************************************************

#!/bin/bash

# The purpose of this script is to relocate the /avamarclient directory to /space

#

# Instructions: Copy this script to the VDP linux console and run

#

# Version 1.0

#

echo "moving directory"

mv /usr/local/avamarclient /space

echo "creating link"

ln -s /space/avamarclient /usr/local/avamarclient

echo "Script Complete - Please logout, and reboot the appliance using Virtual Center - Restart Guest option"

-Suresh
RichElmer
Contributor
Contributor

In the folder /usr/local/avamarclient/var-proxy-?/ which files can safely be deleted to create space?

*.alg and *.log with the same name - oldest ones first?

e.g.

*-vmimagew.alg

*-vmimagew.log

*-vmimagew_avtar.log

for each of the oldest backups ?

Reply
0 Kudos
GSparks
Enthusiast
Enthusiast

Rich -

Yes, those are the safest to delete for the oldest backups.

Also, if you call VMware support, they have a script they can run that will modify the setup of the logging and prevent this from occurring in the future.

Greg

-Greg Sparks
Reply
0 Kudos
RichElmer
Contributor
Contributor

GSparks - Lol - i have a case open with them since friday - cant wait any longer to have no backups....

Thanks for the help - i'll delete the files myself.

Smiley Happy

Reply
0 Kudos
RichElmer
Contributor
Contributor

Unfortunately it didn't give me space back on /dev/sda2

ScreenShot195.jpg

Reply
0 Kudos
RichElmer
Contributor
Contributor

Dont suppose you know what other files the script deletes ? or have a copy of it? - i'm not getting very far with this....

Reply
0 Kudos
RichElmer
Contributor
Contributor

It looks like although the files are being deleted it is not actually reclaiming the space as available.

I was being too cautious deleting only the oldest log files. Deleted all the logs and it regained enough space.

Restarted VDP and it is back up and running.

Message was edited by: RichElmer

Reply
0 Kudos
pitogo
Contributor
Contributor

Same exact issue here, of four 2TB vdp appliances one has / completely full.  I am trying the workaround with with moving the logs to /space.

Reply
0 Kudos
markokobal
Enthusiast
Enthusiast

Hi,

I had the exact same issue.

Here is what I did (simillary to others):

# ssh to vdp and execute these commands:

find /usr/local/avamarclient/ -name "*.log" | xargs rm -f

dpnctl stop all

... be patient and go for a cofee Smiley Wink

mv /usr/local/avamarclient /space

ln -s /space/avamarclient /usr/local/avamarclient

reboot

... be even more patient and go for a beer Smiley Wink

Be sure to make a full reboot! Only restarting VPD services with dpnctl start all as suggested here, VMware KB: Connecting the vSphere Data Protection appliance to vCenter Server fails with the error: ... won't work!

---

Now, to be sure that VDP won't be stuck without space again, I put it in our Nagios monitoring system and enabeled disk usage for /, /var and /space:

# ssh to vdp and execute these commands:

# add repo

zypper addrepo -f http://ftp5.gwdg.de/pub/opensuse/discontinued/distribution/11.1/repo/oss/suse/ opensuse-11.1

# check if repo is installed

zypper lr

# install nrpe-client and plugins

zypper install nagios-nrpe-client nagios-plugins

# check if package has been installed

rpm -qa | grep nrpe

# make sure it boots upon system startup

chkconfig nagios-nrpe on

# edit /etc/nagios/nrpe.cfg

vim /etc/nagios/nrpe.cfg

# add/modify these lines:

---

allowed_hosts=<your Nagios server IP>

dont_blame_nrpe=1

command[check_disk]=/usr/lib/nagios/plugins/check_disk -E -w $ARG1$ -c $ARG2$ -p $ARG3$

command[check_procs]=/usr/lib/nagios/plugins/check_procs -w $ARG1$ -c $ARG2$ -s $ARG3$

command[check_swap]=/usr/lib/nagios/plugins/check_swap -w $ARG1$ -c $ARG2$

---

# start nrpe client

/etc/init.d/nrpe start

# allow nagios server to contact VDP nrpe client in the firewall

iptables -I INPUT -s <your Nagios server IP> -p tcp --dport 5666 -j ACCEPT

# go to your nagios server and make sure nrpe check works

/usr/lib/nagios/plugins/check_nrpe -H <your VDP IP> -c check_disk -a 15% 10% /

/usr/lib/nagios/plugins/check_nrpe -H <your VDP IP> -c check_disk -a 15% 10% /var

/usr/lib/nagios/plugins/check_nrpe -H <your VDP IP> -c check_disk -a 15% 10% /space

# make firewall rule permanent

vim /etc/firewall.base

# add these lines just before the "#  DROP all other traffic and log it" line:

---

# 14. Allow Nagios server to talk to Nagios NRPE client

$IPT -A INPUT -s <your Nagios server IP> -p tcp --dport 5666 -j ACCEPT

---

-- Kind regards, Marko. VCP5
SugeeB
Contributor
Contributor

Hi There,

The reason for unable to see the free space after removed was the logs already pointed to the destination in web app point of view and if you restart the web services of VDP you should be able to see the claimed space immediately please follow the command.

This problem can be occurred if you try to export bundle zip logs also because bundle logs sometimes more then 2.5 GB (Depend of the period of your VDP access)

File location:

#  /usr/local/avamar-tomcat-7.0.54/webapps/ROOT/logBundle/

Actually this is the root location. Verify it by the following command to see the disk space of the current location with (.)

# df -h .

And try the same command from the root / as well, then you can make sure that the location is root actually.

Steps to resolve the issue:

#: find / -name "FinalLogBundle" -print

# cd   /usr/local/avamar-tomcat-7.0.54/webapps/ROOT/logBundle/

# mkdir /space/home/bundle-zip

# mv FinalLogBundle1456933403769.zip /space/home/bundle-zip

YOU MUST RESTART THE WEBSERVICE TO RECLAIM THE SPACE VISIBLE

# emwebapp.sh --restart

zdundore144
Contributor
Contributor

Thank you for the post SugeeB, saved me tons of time this morning trying to find what was taking up my space on the appliance.  A few weeks ago support had me attempt to collect logs and it failed.  That failed attempt was taking up 40% of my space!

Reply
0 Kudos