Issues with GhettoVCB

[Stevester]

Good Evening Everyone,

I am having issues with ghettoVCB running on ESX 3.5. At first i thought it was due to editing the script and input text file on a Windows system, however i am experiencing some strange symptoms when editing on ESX 3.5 Server. Basically if i run the script from this command - ./ghettoVCB.sh vmlist - the script runs perfectly. However if i run the script from a cron job it behaves strangely. I have the VM_BACKUP_ROTATION_COUNT = 1. It runs on my esx 3.5 server in a directory i created at /vmscripts. I have an NFS mount to a storevault appliance to where the backups occur. Here is my cron job - 45 22 * * 1,2,3,4,5 /vmscripts/ghettoVCB.sh /vmscripts/vmlist

Here is what is happening: When the cron job runs a 2nd time, a 2nd folder of each VM is created only with the vmx file and none of the vmdk files are created. It appears that everything is hanging. A snapshot of the VM with the folder and its vmx is created. I generally have to manually delete the snapshot and manually run the script.

Can someone help?

Stevester

[lamw]

If the script runs fine manually on the ESX host, then its not the script, its your cronjob.

=========================================================================

William Lam

VMware vExpert 2009

VMware ESX/ESXi scripts and resources at:

VMware Code Central - Scripts/Sample code for Developers and Administrators

If you find this information useful, please award points for "correct" or "helpful".

[Stevester]

Here is the crontab entry:

0 0 * * 1,2,3,4,5 /vmscripts/ghettoVCB.sh /vmscripts/vmlist

It ran and created two folders of two VMs with only the vmx files. What could be issue with this cronjob

[lamw]

As I've said, if you can manually run this script manually 2-3,4x, then there's nothing wrong the script. You can even do the following, by adding the entry into a file and executing the file manually and see if it errors out and I'm pretty sure it won't.

cat my_script
./ghettoVCB.sh vmlist
./ghettoVCB.sh vmlist
./ghettoVCB.sh vmlist

sh my_script

The issue is with your cronjob and it's not with the actual entry but by default cron does not run under the same set of environmental variables as root and it does not have all the paths configured by default .... why is this important? because vmkfstools is used to copy the VMDKs and I'm pretty positive I don't use the full path which will cause an error and hence you'll get some of the script working but the entire script. A quick fix is to edit the script and provide the full path OR define your path variable within the crontab.

I say again, this has nothing to do with the script but with your cron entry, I recommend doing a search online and checking out how to setup a cron entry along with the various options

=========================================================================

William Lam

VMware vExpert 2009

VMware ESX/ESXi scripts and resources at:

VMware Code Central - Scripts/Sample code for Developers and Administrators

If you find this information useful, please award points for "correct" or "helpful".

[Stevester]

Sir I am not stating that the script is defective. As I have stated, I am asking for your advice on what could be the issue with the cron job.

Thank You

[Stevester]

When you say "full path", is the path being discussed the path to the backup location? In addition, I run this same cron job on a Centos 5.3 machine which has Vmware Server 2.01 on a VM with the serverVCB scritpt and have no issues.

Thanks

Steve

[lamw]

When I refer to the full path, I mean any of the commands issued in the script that does not contain the full path to the binary, one example is vmkfstools where its located on ESX by default in /usr/sbin/vmkfstools and on ESXi, it's /sbin/vmkfstools

You can either edit the script to specify the full path or configure your cron job to have the right path, take a look at the following links for some information on cronjobs:

http://unixgeeks.org/security/newbie/unix/cron-1.html

http://www.adminschoice.com/docs/crontab.htm

Secondly, your comment to VMware Server, you're comparing apples to oranges, there are no correlation that something works in Server and does not in ESXi, and I'm not the author of serverVCB. The code is definitely a copy of ghettoVCB with slight modification to support VMware Server and I'm not aware of what is done in that script or how the cronjob was setup for that system.

Again, your best bet is to look at setting up the right paths and see if you're still having issues.

=========================================================================

William Lam

VMware vExpert 2009

VMware ESX/ESXi scripts and resources at:

VMware Code Central - Scripts/Sample code for Developers and Administrators

If you find this information useful, please award points for "correct" or "helpful".

[Stevester]

Considering the cronjob, are you speaking of the environmental variable paths in the crontab file or path to the ghettoVCB.sh and vmlist that you specify?

Here is the entry:

0 0 * * 1,2,3,4,5 /vmscripts/ghettoVCB.sh /vmscripts/vmlist

This seems to be correct.

[lamw]

Referring the environmental path, also who is actually executing the cron job? is it root?

=========================================================================

William Lam

VMware vExpert 2009

VMware ESX/ESXi scripts and resources at:

VMware Code Central - Scripts/Sample code for Developers and Administrators

If you find this information useful, please award points for "correct" or "helpful".

[Stevester]

My environmental variables appear fine. Root is running the job.

SHELL=/bin/bash

PATH=/sbin:/bin:/usr/sbin:/usr/bin

MAILTO=root

HOME=/

1. run-parts

01 * * * * root run-parts /etc/cron.hourly

02 4 * * * root run-parts /etc/cron.daily

22 4 * * 0 root run-parts /etc/cron.weekly

42 4 1 * * root run-parts /etc/cron.monthly

I will have to think of some way to get ./ghettoVCB.sh vmlist executed or use another solution such as vmbackups

Thanks for your assistance

Stevester

[lamw]

I would say the easiest fix is to edit the script and update the path to where vmkfstools is located at, since the copy of the .vmx file is working sucessfully, I think thats the only path it's unable to locate.

=========================================================================

William Lam

VMware vExpert 2009

VMware ESX/ESXi scripts and resources at:

VMware Code Central - Scripts/Sample code for Developers and Administrators

If you find this information useful, please award points for "correct" or "helpful".

[Stevester]

I placed this variable vmkfstools=/usr/sbin/vmkfstools in the ghettoVCB function and now the script doesn't even run under cron.

[lamw]

I don't think you're setting it right. How are you actually calling the variable? I would also recommend you changing the name of it, so you don't get confused maybe VMKFSTOOLS_CMD=/usr/sbin/vmkfstools

=========================================================================

William Lam

VMware vExpert 2009

VMware ESX/ESXi scripts and resources at:

VMware Code Central - Scripts/Sample code for Developers and Administrators

If you find this information useful, please award points for "correct" or "helpful".

[Stevester]

Here is your ghettoVCB() function. I added the change below in 2nd line. I assume this would be substituted in every instance in your function.

ghettoVCB() {

VM_INPUT=$1

vmkfstools=/usr/sbin/vmkfstools #made change here

START_TIME=`date`

S_TIME=`date +%s`

#dump out all virtual machines allowing for spaces now

${VMWARE_CMD} vmsvc/getallvms | sed 's/[[:blank:]]\{3,\}/ /g' | awk -F' ' '{print "\""$1"\";\""$2"\";\""$3"\""}' | sed 's/\] /\]\";\"/g' | sed '1,1d' > /tmp/vms_list

IFS=$'\n'

for VM_NAME in `cat "$" | sed '/<sup>$/d' | sed -e 's/</sup>[[:blank:]]//;s/[[:blank:]]$//'`;

do

VM_ID=`grep -E "\"$\"" /tmp/vms_list | awk -F ";" '{print $1}' | sed 's/"//g'` #ensure default value if one is not selected or variable is null if ; then VM_BACKUP_DIR_NAMING_CONVENTION="$(date +%F)" fi VMFS_VOLUME=`grep -E "\"$\"" /tmp/vms_list | awk -F ";" '{print $3}' | sed 's/\[//;s/\]//;s/"//g'`

VMX_CONF=`grep -E "\"$\"" /tmp/vms_list | awk -F ";" '{print $4}' | sed 's/\[//;s/\]//;s/"//g'` VMX_PATH="/vmfs/volumes/$/$" VMX_DIR=`dirname "$"` #checks to see if we can pull out the VM_ID if ; then echo "Error: failed to extract VM_ID for $!"

#devel mode

elif ; then

echo "##########################################"

echo "Virtual Machine: $VM_NAME"

echo "VM_ID: $VM_ID"

echo "VMX_PATH: $VMX_PATH"

echo "VMX_DIR: $VMX_DIR"

echo "VMX_CONF: $VMX_CONF"

echo "VMFS_VOLUME: $VMFS_VOLUME"

echo -e "##########################################\n"

#checks to see if the VM has any snapshots to start with

elif ls "$" | grep -q delta > /dev/null 2>&1; then echo "Snapshot found for $, backup will not take place" #checks to see if the VM has an RDM elif $ vmsvc/device.getdevices $ | grep "RawDiskMapping" > /dev/null 2>&1; then echo "RDM was found for $, backup will not take place" elif [ && [; then #nfs case and backup to root path of your NFS mount if ; then BACKUP_DIR="/vmfs/volumes/$/$/$" if [[ -z $ ]] || [[ -z $ ]] || [[ -z $ ]]; then echo "Variable BACKUP_DIR was not set properly, please ensure all required variables for non-persistent NFS backup option has been defined" exit 1 fi #non-nfs (SAN,LOCAL) else BACKUP_DIR="$/$" if [[ -z $ ]]; then echo "Variable VM_BACKUP_DIR was not defined" exit 1 fi fi #initial root VM backup directory if ; then mkdir -p "$

1. directory name of the individual Virtual Machine backup followed by naming convention followed by count

" fi VM_BACKUP_DIR="$/$-$" mkdir -p "$" cp "$" "$" #get all VMDKs listed in .vmx file VMDKS_FOUND=`grep -iE '(scsi|ide)' "$" | grep -i fileName | awk -F " " '{print $1}'` #loop through each disk and verify that it's currently present and create array of valid VMDKS for DISK in $; do #extract the SCSI ID and use it to check for valid vmdk disk SCSI_ID=`echo ${DISK%%.*}` grep -i "$.present" "$" | grep -i "true" > /dev/null 2>&1 #if valid, then we use the vmdk file if ; then DISK=`grep -i $.fileName "$" | awk -F "\"" '{print $2}'` VMDKS="$:$" fi done ORGINAL_VM_POWER_STATE=$($ vmsvc/power.getstate $ | tail -1) #section that will power down a VM prior to taking a snapshot and backup and power it back on if ; then START_ITERATION=0 echo "Powering off initiated for $, backup will not begin until VM is off..." $ vmsvc/power.shutdown $ > /dev/null 2>&1 while $ vmsvc/power.getstate $ | grep -i "Powered on" > /dev/null 2>&1; do #enable hard power off code if ; then START_ITERATION=$((START_ITERATION + 1)) if ; then echo "Hard power off occured for $, waited for $((ITER_TO_WAIT_SHUTDOWN*3)) seconds" $ vmsvc/power.off $ > /dev/null 2>&1 #this is needed for ESXi, even the hard power off did not take affect right away sleep 5 break fi fi echo "VM is still on - Iteration: $ - waiting 3secs" sleep 3 done echo "VM is off" fi #powered on VMs only if [[ ! $ -eq 1 ]] && [; then echo "################ Taking backup snapshot for $ ... ################" $ vmsvc/snapshot.create $ vcb_snap VCB_BACKUP_$_`date +%F` > /dev/null 2>&1 else echo "################## Starting backup for $ ... #####################" fi OLD_IFS="$" IFS=":" for j in $; do VMDK="$" #added this section to handle VMDK(s) stored in different datastore than the VM echo $ | grep "^/vmfs/volumes" > /dev/null 2>&1 if ; then SOURCE_VMDK="$" DS_UUID="$(echo ${VMDK#/vmfs/volumes/*})" DS_UUID="$(echo ${DS_UUID%//})" VMDK_DISK="$(echo ${VMDK##/*/})" mkdir -p "$/$" DESTIONATION_VMDK="$/$/$" else SOURCE_VMDK="$/$" DESTIONATION_VMDK="$/$"

fi

if ; then

vmkfstools -i "$" -d 2gbsparse "$" else vmkfstools -i "$" "$"

fi

done

IFS="$"

#powered on VMs only w/snapshots

if [[ ! $ -eq 1 ]] && [; then $ vmsvc/snapshot.remove $ > /dev/null 2>&1 #do not continue until all snapshots have been committed echo "Removing snapshot from $ ..." while ls "$" | grep -q delta; do sleep 3 done fi if [[ $ -eq 1 ]] && [; then

#power on vm that was powered off prior to backup

echo "Powering back on $" $ vmsvc/power.on $ > /dev/null 2>&1 fi checkVMBackupRotation "$" "$" VMDKS="" echo -e "#################### Completed backup for $! ####################\n"

else

echo "Error: failed to lookup $!"

fi

done

unset IFS

if [[ $ -eq 1 ]] && [[ $ -eq 1 ]] ; then

$ hostsvc/datastore/destroy $

fi

echo

END_TIME=`date`

E_TIME=`date +%s`

echo "Start time: $"

echo "End time: $"

DURATION=`echo $((E_TIME - S_TIME))`

#calculate overall completion time

if ; then

echo "Duration : $ Seconds" else echo "Duration : `awk 'BEGIN{ printf "%.2f\n", '$'/60}'` Minutes"

fi

echo -e "\nCompleted backing up specified Virtual Machines!\n"

}

[lamw]

A) Don't post the entire script, it's unreadable within the forum due to mangling and not using proper tags.

B) You're not understanding how variables are defined/used in a shell script and hence it's not working.

defining a variable:

HI_IM_A_VARIABLE=/usr/sbin/vmkfstools

utilizing a variable:

echo ${HI_IM_A_VARIABLE}

You need to specify '$' to actually make use of the variable substitution and you need to make a modification everytime vmkfstools is called and in the original script, the call itself is to the vmkfstools binary.

=========================================================================

William Lam

VMware vExpert 2009

VMware ESX/ESXi scripts and resources at:

VMware Code Central - Scripts/Sample code for Developers and Administrators

If you find this information useful, please award points for "correct" or "helpful".

[Stevester]

This still is not working!!!!!!!!!!!!!! I have substituted the path /usr/sbin/vmkfstools in two locations in the script. I removed the variable substitution did earlier. I have done research on the internet and have seen others having the same issue. Does anyone else have a solution to this?

[Stevester]

Got it working!!!!! It appears my cron service was off. Mr Lam: After doing research I saw that others were having the same issue as I was and it was suggested by you to do the same. I would suggest that an update be placed on your site so others can avoid this same issue I had. It would save others valuable time. I am just looking out for others.

Stevester

[lamw]

Good to hear!

What was the actual fixed that made it work? I'll be more than welcome to document this in the FAQ section regarding issues/setting up cronjobs.

=========================================================================

William Lam

VMware vExpert 2009

VMware ESX/ESXi scripts and resources at:

VMware Code Central - Scripts/Sample code for Developers and Administrators

If you find this information useful, please award points for "correct" or "helpful".

[Stevester]

Hold on. There is still issues with your script.

In the 3.5 verison, I merely substitued the absolute path in two locations. However, the script doesn't backup all the VM's. I have three VMs and it hangs on backing the 2nd VM. The snapshot doesn't get removed on the 2nd VM.

In your new verison, there is a syntx error stating there is a syntax error in the print function. A { symbol is missing.

I posted some new threads concerning this issue

Stevester

[lamw]

First off, if you're going to edit the script ... then the issue lies with your edits and that will NOT be supported because I do not know what edits you've made. I'm willing to push an update that will contain the full path of vmkfstools.

I highly recommend you re-downloading the script and see if it's having that issue. If it's not, then we can continue troubleshooting your cron job issue.

=========================================================================

William Lam

VMware vExpert 2009

VMware ESX/ESXi scripts and resources at:

VMware Code Central - Scripts/Sample code for Developers and Administrators

If you find this information useful, please award points for "correct" or "helpful".