VMware Communities > VMTN > VMware Infrastructure™ > VI: ESX 3.5 > Documents

ghettoVCB.sh - Free alternative for backing up VM's for ESX(i) 3.5 and 4.0+

VERSION 34 Published

Created on: Nov 17, 2008 7:04 PM by lamw - Last Modified:  Jun 1, 2009 8:34 PM by lamw

Description:


This script performs backups of virtual machines residing on ESX(i) 3.5/4.0+ servers using methodology similar to VMware's VCB tool. The script takes snapshots of live virtual machines, backs up the master VMDK(s) and then upon completion, deletes the snapshot until the next backup. The only caveat is that it utilizes resources available to the Service Console of the ESX server running the backups as opposed to following the traditional method of offloading virtual machine backups through a VCB proxy.

This script has been tested on ESX 3.5u3 and ESXi u3 and supports the following backup mediums: LOCAL STORAGE, SAN and NFS. The script is non-interactive and can be setup to run via crontab. Currently, this script accepts a text file that lists the display names of virtual machine(s) that are to be backed up. Specification of the destination path is done by editing the top portion of the script itself. It is important to note that backup destinations specified in the VM_BACKUP_DIR variable must be the full path to storage (LOCAL, NFS, and SAN) that is being presented to the ESX(i) server as a datastore (i.e. datastores that are located within /vmfs/volumes/).

Additionally, for ESX environments that don't have persistent NFS datastores designated for backups, the script offers the ability to automatically connect the ESX server to a NFS exported folder and then upon backup completion, disconnect it from the ESX server. The connection is established by creating an NFS datastore link which enables monolithic (or thick) VMDK backups as opposed to using the usual *nix mount command which necessitates breaking VMDK files into the 2gbsparse format for backup. Enabling this mode is self-explanatory and will evidently be so when editing the script (Note: VM_BACKUP_DIR variable is ignored if ENABLE_NON_PERSISTENT_NFS=1 ).

In its current configuration, the script will allow up to 3 unique backups of the Virtual Machine before it will overwrite the previous backups; this however, can be modified to fit procedures if need be. Please be diligent in running the script in a test or staging environment before using it on production live Virtual Machines; this script functions well within our environment but there is a chance that it may not fit well into other environments.

Features
  • Support for multiple VMDK disk(s) backup per VM
  • Only valid VMDK(s) presented to the VM will be backed up
  • Online back up of VM(s)
  • Ability to shutdown guestOS and initiate backup process and power on VM afterwards with the option of hardpower timeout
  • Allow spaces in VM(s) backup list (not recommended and not a best practice)
  • Ensure that snapshot removal process completes prior to to continuing onto the next VM backup
  • VM(s) that intially contain snapshots and RDMs (raw device mapppings) will not be backed up and will be ignored
  • Ability to specify the number of backup rotations for VM
  • Output back up VMDK(s) in either THICK (default behavior) or 2GB SPARSE format
  • Non-persistent NFS backup
  • Fully support VMDK(s) stored across multiple datastores

Detail instructions on how to configure/schedule ghettoVCB (Raj Perumal's The World Runs on Technology Blog):

Part 1
Part 2
Part 3

If you have any questions feel free to post on this thread and for scripts/resources please visit: http://engineering.ucsb.edu/~duonglt/vmware/



Script Usage Chart


Host Version Script to use
ESX 3.5/4.0+ and ESXi 3.5+ ghettoVCB.sh
ESXi 4.0 ghettoVCB4i.sh


Requirements:

VMs running on ESX(i) 3.5/4.0+
SSH console access to ESX/ESXi host

The only variable required to ge the basic functionality is VM_BACKUP_VOLUME, the rest are optional for the user.

Ensure you do not edit past this section:

# DO NOT MODIFY PAST THIS LINE #


Setup:


1) Download ghettoVCB.sh or ghettoVCB4i.sh to either your ESX or ESXi system based on the supported script usage chart

2) Change the permissions on the script to ensure it can be executed:

chmod +x ghettoVCB.sh


Usage:


[root@himalaya scripts]# ./ghettoVCB.sh

Usage: ghettoVCB.sh [VM_FILE_INPUT]


The input to this script is a file that contains the display name of the virtual machine(s) separated by a newline. When creating this file on a non-Linux/UNIX system, you may introduce ^M character which can cause the script to miss-behave. To ensure this does not occur, plesae create the file on the ESX/ESXi host.

Here is a sample of what the file would look like:

vi-admin@vima-primp-industries ~$ cat vm_backup_list
grimlock
slag
ALima
I HATE SPACES
AAmbrosio
WILLIAM-LC-24
WILLIAM-LC-25

Sample Execution


Here is a sample run on ESX 3.5u3:
[root@himalaya scripts]# ./ghettoVCB-enhance.sh example_virtual_machine_backup_list
Powering off initiated for Quentin, backup will not begin until VM is off...
VM is still on - Iteration: 1 - waiting 3secs
VM is still on - Iteration: 2 - waiting 3secs
VM is off
################## Starting backup for Quentin ... #####################
Destination disk format: VMFS thick
Cloning disk '/vmfs/volumes/himalaya-local-SAS.VMStorage/Quentin/Quentin_1.vmdk'...
Clone: 100% done.
Destination disk format: VMFS thick
Cloning disk '/vmfs/volumes/himalaya-local-SAS.VMStorage/Quentin/Quentin.vmdk'...
Clone: 100% done.
Powering back on Quentin
#################### Completed backup for Quentin! ####################


Start time: Tue Jan  6 16:09:35 PST 2009
End   time: Tue Jan  6 16:12:12 PST 2009
Duration  : 2.62 Minutes

Completed backing up specified Virtual Machines!


Here is a sample run on ESXi 3.5u3 (recommend changing the ITERATION value to higher number, ESXi for some reason may take longer):
~ # ./ghettoVCB-enhance.sh backup
Powering off initiated for UCSB-ENGINEERING, backup will not begin until VM is off...
VM is still on - Iteration: 1 - waiting 3secs
VM is still on - Iteration: 2 - waiting 3secs
VM is still on - Iteration: 3 - waiting 3secs
VM is still on - Iteration: 4 - waiting 3secs
Hard power off occured for UCSB-ENGINEERING, waited for 12 seconds
VM is off
################## Starting backup for UCSB-ENGINEERING ... #####################
Destination disk format: VMFS thick
Cloning disk '/vmfs/volumes/dlgCore-FC-LUN200.Templates/UCSB-ENGINEERING/UCSB-ENGINEERING.vmdk'...
Clone: 100% done.
Powering back on UCSB-ENGINEERING
#################### Completed backup for UCSB-ENGINEERING! ####################


Start time: Sat Jan 10 00:57:37 UTC 2009
End   time: Sat Jan 10 01:01:55 UTC 2009
Duration  : 4.30 Minutes

Completed backing up specified Virtual Machines!



Change log


UPDATE (05/31/2009: An issue was reported by a user that ghettoVCB4i does not support backup of ide drives which is now supported in vSphere and a fix has been implemented to support this feature.

UPDATE (05/27/2009): There have been some significant changes on ESXi 4.0's unsupported Busybox console v1.2.1 to v1.9.1 that caused ghettoVCB to halt and display erroneous messages/errors. An individual version of ghettoVCB has been created specifically for ESXi 4.0 which is available for download at the bottom of this document ( ghettoVCB4i.sh ). Some of the changes included in the modified version is to enforce/validated IFS variable, not specifying the full path to vim-cmd and declaring header with /bin/ash instead of /bin/bash

UPDATE (02/22/2009): An issue was discovered when VMDK(s) are stored across multiple datastores. Due to the automated naming convention by VMware, VMDK disk names are only unique within the same datastore and not across datastores, the remediation for this is to create a directory in the destination capturing the datastore's unique UUID.

(e.g)

Source:
TEST_VM

DATASTORE-A/TEST_VM.vmdk
DATASTORE-B/TEST_VM.vmdk

Destination:
TEST_VM

TEST_VM.vmdk
DATASTORE-B_UUID/TEST_VM.vmdk

UPDATE (01/26/2009): An issue was noted by a few users using the backup script on ESXi that when looking for all VMDK(s), the parameter has been modified from fileName to filename and this failed due to the grep not being in-case-sensitive, this was remiediate by adding -i flag.

UPDATE (01/06/2009): An issue was discovered by user NoSa on a corner case that could cause a disk_lib() warning message when executing a backup. The issue arises when a user removes a VM disk using the VIC with either option: "Remove from virtual machine" or "Remove from virtual machine and delete file from disk" which could cause some unexpected behavior.

Case "Remove from virtual machine": If you remove the VM disk, the VMDK will continue to exist on the filesystem but in the .vmx file, it will still contain the present flag set to false, the backup script will still backup the VMDK

Case "Remove from virtual machine and delete file from disk": If you remove the VM disk, it will delete the VMDK off the filesystem but a reference will still exist in the .vmx file, in which the script will try to backup the deleted VMDK and causes the warning message

This is now fixed and will only look for valid presented disks to the VM and backup only those VMDK(s).

  • Added a new flag at the top of the script "POWER_VM_DOWN_BEFORE_BACKUP", if set to 1, will try to power down the guestOS prior to taking a backup, which will not include taking a snapshot as the primary VMDK will not be locked. Once the backup has finished, it will power the VM back on and continue on to the next VM. By default this option is disabled

  • Added a new flag at the top of the script "ENABLE_HARD_POWER_OFF" which allows you to force a hard power off of the guestOS shutdown has not powered down within the specified time interval using the variable "ITER_TO_WAIT_SHUTDOWN". This allows you to set the number of 3 second iterations to wait for a guestOS shutdown, and if it does not shutdown within the time interval, then a forced hard power off will occur. For example, the default value is set to "4" in the script. This means it will wait up to 12 seconds total prior to the hard power off, of which are 3 second interval checks. This functionality is optional to the user, by default this is option is disabled along with powering off VMs prior to backups.

UPDATE (12/04/2008): Small update, In the NON-PERSISTENT NFS section, the flag to check whether a NFS Datastore has been mounted or not is checking for the wrong exit status value. Should be "0" and not "1"

UPDATE (11/29/2008): Updated a small bit of code with the suggestions from JoSte and aremmes to ensure maximum efficiency. The script name has also been changed from ghettoVCBni.sh to just ghettoVCB.sh

UPDATE (11/26/2008): A new fix has been implemented to allow the name of the Virtual Machines to contain spaces, which will be provided via the input file. This is not a recommended best practice in terms of naming conventions in general, underscores and dashes should be used to add separation in your naming schemes.

INFO (11/24/2008): The text input file should contain the displayName of your Virtual Machine separated by a newline and should look similar to the following (the file can be named anything, ensure it's created on ESX or ESXi else you'll have some issues with the Windows "^M" character:

vm1
vm2
vm3


UPDATE (11/22/2008): The script has been updated with the following fixes and additional features:

Fixes:

  • An issue that involved snapshot commits during virtual machine backups has been addressed. Performance degradation may occur if snapshot remove operations persist and accumulate in number during the backup process. The script now ensures that the snapshot removal process completes on the current virtual machine prior to continuing onto the next virtual machine specified in the backup list.

Additional features:

  • The script now detects virtual machines that contain existing snapshots and RDMs (raw device mappings). If any of these two attributes exist, the script will skip the virtual machine in question and continue onto the next virtual machine specified in the backup list.

  • Backup rotation is now possible. To specify the number of backups to keep, please modify the VM_BACKUP_ROTATION_COUNT variable at the top of the script. For example, a value of 3 will tell the script to retain three prior backups of the specified virtual machine. Old virtual machine backups that fall beyond the third prior backup will be deleted. If this value is left blank (NULL), the script will default to 1, that is, every new backup will overwrite the previous backup if it exists. The default format of the rotated backup directory is: <VMDISPLAYNAME>-YYYY-MM-DD. The suffix, YYYY-MM-DD, can be modified via the VM_BACKUP_DIR_NAMING_CONVENTION variable at the top of the script to fit your backup procedures.

  • The script now provides the option to output VMDK files as THICK (default behavior) or 2GB SPARSE format on the backup datastore. A self-explanatory variable at the top the script can be modified to enable 2GB SPARSE format output.

  • The backup directory for the non-persistent NFS backup feature can now be specified. Please modify the NFS_VM_BACKUP_DIR variable to specify the backup directory name. This directory will be created if it does not exist. This variable MUST be defined or else the script will exit.

UPDATE (11/17/2008): Thanks to JoSte, a small duplication of code is being executed which caused multiple snapshots to be taken if a Virtual Machine has more than 1 VMDK, the code has been updated and optimized to only snapshot the Virtual Machine once and backing up all VMDK(s). Please download the latest script which has been uploaded to fix this.
Attachments:
Average User Rating
(8 ratings)




Dec 17, 2008 10:16 AM Shaveht  says:

How can i edit it to run on VIMA?

Dec 19, 2008 8:59 AM lamw  says:

Forgot to mention, you can also find more scripts/resources located at:

http://engineering.ucsb.edu/~duonglt/vmware/

Dec 19, 2008 9:08 AM lamw  says: in response to: Shaveht

Hi Shaveht,

This script can not be locally executed on VMware VIMA, but can be remotely kicked off from VIMA or any other UNIX/Linux, even a Windows host using plink.

The process is as follows:

You store the "ghettoVCB.sh" on shared storage (FC/iSCSI SAN / NFS), including the list of Virtual Machines you would like to backup. You can then create a cron entry that will kick off the backup process say every Friday at 12am. To ensure this does not require a user to enter a password, you'll need to utilize public/private SSH Key Authentication. This would then ssh into the ESX/ESXi host(s) and execute this script passing in the list which is also stored on the shared storage and the backup process would be kicked off.

This allows you to centralize your backup process without having to create separate cron entries on each of the ESX and ESXi hosts and storing a copy on each system. This does add an additional system to the process and if you know this VIMA host will always be accessible to the ESX/ESXi cluster, you should be fine.

An entry to the cron for the command would look like the following:

ssh root@A.B.C.D "/vmfs/volumes/SOME_DATASTORE/ghettoVCB.sh /vmfs/volumes/SOME_ADMIN_DATASTORE/virtual_machine_backup_list"


Again, you would just create a cron entry for the set of ESX/ESXi hosts and ensure that the datastore that contains the backup script and backup list are presented and visiable to all hosts. This also includes the datastore that is used to backup all the VMs are also visible/presented to all hosts

Hopefully this cleared up some of your questions and good luck

Dec 29, 2008 12:35 AM Shaveht  says: in response to: lamw

Hi lamw,
Thanks for the explanation.
in effect i was confused about how to use it.

thanks again.

Jan 9, 2009 6:43 PM dan82  says:

Hi,

I have 7 blade servers with approx 50 VM's, is it possible to run this script from just one esxi host and have it back up all the VM's that are running on it locally and also the VM's running on the other blades?

Jan 10, 2009 10:53 PM lamw  says: in response to: dan82

Take a look at the my previous comment. I would not suggest setting this up on an ESXi host, probably an administrative linux/windows or even VIMA host.


=========================================================================
--William
VMware ESX/ESXi scripts and resources at: http://engineering.ucsb.edu/~duonglt/vmware/

Jan 14, 2009 3:51 PM tuxfan2183  says: in response to: lamw

Nice script and Thanks for the same. The backup runs fine after few issues being faced in the start. I have the following queries.

1. Is there a way I can backup the vm's without shutting them down?
2. What is the ideal restore procedure? (assuming Its going to be a disk-to-disk restoration.)

Thanks
tux

Jan 14, 2009 4:05 PM lamw  says: in response to: tuxfan2183

Hi tuxfan2183,

1) The power down feature was recently added, if you want to backup your VM(s) while they're live, just ensure the flag "POWER_VM_DOWN_BEFORE_BACKUP" is set to 0=no, 1=yes. By default this option is set to 0 which is to backup while the VM is running

2) There are multitude of options, depending where and how the backups are being stored and what your backup scenario requires. You can either mount the backup datastore and register the VM either by browsing the VI Client datastore or manually using CLI tool like "vmware-cmd -s register" or "vim-cmd" for ESXi. If you do not want to directly mount your backup but rather restore back to your source ESX/ESXi host, you can just manually run "vmkfstools" copy from your backup datastore to your source datastore and then register your VM. Again, this will be environment specific, but those are your options and this can be easily scripted or done manually as this is a backup process which requires additional care.

Hope this answered your questions.


=========================================================================
--William
VMware ESX/ESXi scripts and resources at: http://engineering.ucsb.edu/~duonglt/vmware/

Jan 14, 2009 4:10 PM tuxfan2183  says: in response to: lamw

Thanks william. You are the best

Jan 21, 2009 6:57 PM nappyrat  says:

does this only back up 1 vmdk file per virtual machine? I ran it and it backed up teh vmdk of a vm i have but not another vmdk that is in the same directory

Jan 21, 2009 7:43 PM lamw  says: in response to: nappyrat

No, it should backup all "valid" VMDK(s) that is associated with a VM. Yon can verify this by going into the VM's "edit virtual machine settings" and ensure that all disks are listed within the hardware tab.

When you use the VIC to delete a disk and selecting the option "Remove from virtual machine", this will only remove the disk from being accessible to the VM, but the actual disk will still be located on the datastore and within the .vmx file, the disk will have a present value of "false". If you want to truly delete the VM's disk, you need to select the other option, so be sure before doing so.

Let me know if you have any other questions


=========================================================================
--William
VMware ESX/ESXi scripts and resources at: http://engineering.ucsb.edu/~duonglt/vmware/

Jan 21, 2009 8:09 PM tuxfan2183  says: in response to: lamw

There are certain things which I as a user thought could add more value to this script. When taking backup to compress and store the backup in a compressed format in the remote location. makes sense?

Jan 21, 2009 8:12 PM lamw  says: in response to: tuxfan2183

That is a good suggestion, but unfortunately there are many options in terms of compression format/type/etc and I would leave that to the end user to implement if required.

=========================================================================
--William
VMware ESX/ESXi scripts and resources at: http://engineering.ucsb.edu/~duonglt/vmware/

Jan 21, 2009 8:27 PM alubel  says: in response to: tuxfan2183

Why dont you try to setup a solaris 10 or opensolaris box and use a compressed ZFS file system via NFS to ESXi host?

Its extremely easy - and no expensive hardware required.

Jan 21, 2009 8:52 PM nappyrat  says: in response to: lamw

i found the problem. here is the 2nd disk in the .vmx file. I can almost guaruntee you this wasn't modified by hand because I don't think the person who created it has no idea how to.

scsi0:1.filename = "datastore_1.vmdk"

the line in your script that finds the disks is

VMDKS=`grep scsi "${VMX_PATH}" | grep fileName | awk -F "\"" '{print $2}'`

it's a case issue. maybe that should be changed to a "grep -i fileName"

Jan 21, 2009 9:47 PM lamw  says: in response to: nappyrat

Yea it should be -i, but from the various .vmx file I've viewed in the past for both 3.0 and 3.5 it's always been "fileName". What version of ESX are you running btw? This is the first I've heard of this issue


=========================================================================
--William
VMware ESX/ESXi scripts and resources at: http://engineering.ucsb.edu/~duonglt/vmware/

Jan 22, 2009 3:31 AM nappyrat  says: in response to: lamw

esx server 3i 3.5.0 build 123629

i just reproduced this by adding a second hard disk with VIC to the virtual machine and it puts it in the vmx with filename not fileName

Jan 22, 2009 6:54 AM Thomas BRETON  says: in response to: nappyrat

Hello,
I'm trying your script.
Thanks for your work.
Just a question relative to performance.
In fact i was using vcbMounter on a local datastore and on an nfs server.
Backup time for the same machine:
with vcbmount on local datastore: 7 min
with vcbmount on NFS: 13 min

With your script on local datastore: 5 min
With your script on NFS: 180 min

Have you got an explanation for this or any clue?
Regards
Tom

Jan 22, 2009 7:09 AM nappyrat  says: in response to: Thomas BRETON

my first run of this script was really slow over nfs too, around 25GB per hour. i blamed that on my cheap nas device.

i don't know what vcbmounter is, but does it copy with a straight copy or does it do a clone disk like this script does?

Jan 22, 2009 8:34 AM lamw  says: in response to: Thomas BRETON

I'm not familiar with the vcbMounter tool, YMMV with different environments and I've seen various postings about the different performance with NFS and it all depends on your configuration and the type of NFS server you're running. It should not differ with over 100+min so I'm not exactly sure what type of VM you're backing up/etc. Does this occur on all VM(s) or just this one?

Also and FYI, the VMware VCB tools that come installed by default on the ESX host are only available with the version of ESX w/Service Console, this is not part of the ESXi releaese.

=========================================================================
--William
VMware ESX/ESXi scripts and resources at: http://engineering.ucsb.edu/~duonglt/vmware/

Jan 22, 2009 8:35 AM lamw  says: in response to: nappyrat

Interesting, perhaps this is only with ESXi. I'll double check with my ESXi 3.5u3 and see if this is also the case, if so I'm surprised this is the first time it's been caught. I know there's quite a few people using this script on unlicensed version of ESXi. Thanks for the info

=========================================================================
--William
VMware ESX/ESXi scripts and resources at: http://engineering.ucsb.edu/~duonglt/vmware/

Jan 22, 2009 11:04 AM apignoloni  says:

when I run the command sh ./ghetto.sh vm_list from path / vmfs/volumes/datastore1/ of esxi 3.5 returns the following error:
. / ghetto.sh:. / ghetto.sh: 76: Syntax error: word unexpected (expecting "do").
What could be the problem?
Thank you.

Jan 22, 2009 11:27 AM lamw  says: in response to: apignoloni

I'm not exactly sure what the issue could be. I suggest re-downloading the script and ensure the script has been set to execute. If you download to a Windows system, you might have some ^M characters if you tried to edit prior to executing on ESX/ESXi host. Make sure you edit it on the host themselves to ensure you don't run into this issue.

You can always run:

sh -x ghetto.sh vm_list


and see what might be causing the issue.


=========================================================================
--William
VMware ESX/ESXi scripts and resources at: http://engineering.ucsb.edu/~duonglt/vmware/

Jan 22, 2009 11:49 AM apignoloni  says:

Thanks for your answer, the script has been set to execute and the file has been downloaded and modified with ubuntu desktop.
This is the result of the command:
/vmfs/volumes/4975d95d-45fc3458-7ff4-001b212e4341 # sh -x ghetto.sh vm_list
+
: not foundghetto.sh: 6:
+ VM_BACKUP_VOLUME=/vmfs/volumes/datastore2/backupvm/
+
: not foundghetto.sh: 9:
+ ENABLE_2GB_SPARSE=0
+
: not foundghetto.sh: 12:
+ VM_BACKUP_ROTATION_COUNT=3
+
: not foundghetto.sh: 15:
+ date +%F
+ VM_BACKUP_DIR_NAMING_CONVENTION=2009-01-22
: not foundghetto.sh: 17:
+
: not foundghetto.sh: 18:
+ POWER_VM_DOWN_BEFORE_BACKUP=0
+
: not foundghetto.sh: 23:
+ ENABLE_HARD_POWER_OFF=0
+
: not foundghetto.sh: 26:
+ ITER_TO_WAIT_SHUTDOWN=4
+
: not foundghetto.sh: 31:
+
: not foundghetto.sh: 36:
+ ENABLE_NON_PERSISTENT_NFS=0
+
: not foundghetto.sh: 38:
+ UNMOUNT_NFS=0
+
: not foundghetto.sh: 41:
+ NFS_SERVER=172.30.0.195
+
: not foundghetto.sh: 44:
+ NFS_MOUNT=/nfsshare
+
: not foundghetto.sh: 47:
+ NFS_LOCAL_NAME=nfs_storage_backup
+
: not foundghetto.sh: 50:
+ NFS_VM_BACKUP_DIR=mybackups
: not foundghetto.sh: 52:
+
: not foundghetto.sh: 53:
+
: not foundghetto.sh: 55:
+
: not foundghetto.sh: 57:
+ DEVEL_MODE=0
+
: not foundghetto.sh: 59:
+ basename ghetto.sh
+ SCRIPT_PATH=ghetto.sh
VM_FILE_INPUT\n ghetto.sh

VM_FILE_INPUT

+ exit
: not foundghetto.sh: 63: exit
+ }
: not foundghetto.sh: 64: }
+
: not foundghetto.sh: 65:
+ local BACKUP_DIR_PATH=vm_list
+ local BACKUP_VM_NAMING_CONVENTION=
+ ls -tr vm_list
: No such file or directory
+ LIST_BACKUPS=
+
: not foundghetto.sh: 70:
ghetto.sh: ghetto.sh: 76: Syntax error: word unexpected (expecting "do")
/vmfs/volumes/4975d95d-45fc3458-7ff4-001b212e4341 # sh -x ./ghetto.sh vm_list
+
: not found: ./ghetto.sh: 6:
+ VM_BACKUP_VOLUME=/vmfs/volumes/datastore2/backupvm/
+
: not found: ./ghetto.sh: 9:
+ ENABLE_2GB_SPARSE=0
+
: not found: ./ghetto.sh: 12:
+ VM_BACKUP_ROTATION_COUNT=3
+
: not found: ./ghetto.sh: 15:
+ date +%F
+ VM_BACKUP_DIR_NAMING_CONVENTION=2009-01-22
: not found: ./ghetto.sh: 17:
+
: not found: ./ghetto.sh: 18:
+ POWER_VM_DOWN_BEFORE_BACKUP=0
+
: not found: ./ghetto.sh: 23:
+ ENABLE_HARD_POWER_OFF=0
+
: not found: ./ghetto.sh: 26:
+ ITER_TO_WAIT_SHUTDOWN=4
+
: not found: ./ghetto.sh: 31:
+
: not found: ./ghetto.sh: 36:
+ ENABLE_NON_PERSISTENT_NFS=0
+
: not found: ./ghetto.sh: 38:
+ UNMOUNT_NFS=0
+
: not found: ./ghetto.sh: 41:
+ NFS_SERVER=172.30.0.195
+
: not found: ./ghetto.sh: 44:
+ NFS_MOUNT=/nfsshare
+
: not found: ./ghetto.sh: 47:
+ NFS_LOCAL_NAME=nfs_storage_backup
+
: not found: ./ghetto.sh: 50:
+ NFS_VM_BACKUP_DIR=mybackups
: not found: ./ghetto.sh: 52:
+
: not found: ./ghetto.sh: 53:
+
: not found: ./ghetto.sh: 55:
+
: not found: ./ghetto.sh: 57:
+ DEVEL_MODE=0
+
: not found: ./ghetto.sh: 59:
+ basename ./ghetto.sh
+ SCRIPT_PATH=ghetto.sh
VM_FILE_INPUT\n ghetto.sh

VM_FILE_INPUT

+ exit
: not found: ./ghetto.sh: 63: exit
+ }
: not found: ./ghetto.sh: 64: }
+
: not found: ./ghetto.sh: 65:
+ local BACKUP_DIR_PATH=vm_list
+ local BACKUP_VM_NAMING_CONVENTION=
+ ls -tr vm_list
: No such file or directory
+ LIST_BACKUPS=
+
: not found: ./ghetto.sh: 70:
./ghetto.sh: ./ghetto.sh: 76: Syntax error: word unexpected (expecting "do")
/vmfs/volumes/4975d95d-45fc3458-7ff4-001b212e4341 #

Jan 22, 2009 11:56 AM lamw  says: in response to: apignoloni

It's kind of hard to tell what's going on with the output. What system are you running this on? Which version of the script did you download? Can you also list the contents of your vm_list

Can you also try the following:

Change

VM_BACKUP_VOLUME=/vmfs/volumes/datastore2/backupvm/

to

VM_BACKUP_VOLUME=/vmfs/volumes/datastore2/backupvm

If that still yields the same results, can you also find the variable DEVEL_MODE and change that from 0 to 1 and see if the script progresses farther or continues to error out.


=========================================================================
--William
VMware ESX/ESXi scripts and resources at: http://engineering.ucsb.edu/~duonglt/vmware/

Jan 22, 2009 12:01 PM nappyrat  says: in response to: apignoloni

also, you didn't remove the first line with #!/bin/bash did you? or you're not using some wierd shell are you?

Jan 22, 2009 12:15 PM apignoloni  says:

I downloaded version ghettoVCB-enhance.sh. The contents of the list is as follows:

windows xp

ubuntu server 32bit

this change VM_BACKUP_VOLUME=/vmfs/volumes/datastore2/backupvm had no effect.

Aalso the change of value DEVEL_MOdE variable had no effect.

Jan 22, 2009 12:20 PM lamw  says: in response to: apignoloni

Are you running this on ESX or ESXi? I also noticed you're executing this script on one of your datastores, it should not matter but can you try to move the script to another location, perhaps /root. I just downloaded both scripts and they executed fine using the default and just modifying the backup path, the issue seems to be in your environment but I'm not sure what might be causing this issue.

Could you also run the following on the shell:

echo $SHELL


vmware -v 


=========================================================================
--William
VMware ESX/ESXi scripts and resources at: http://engineering.ucsb.edu/~duonglt/vmware/

Jan 22, 2009 12:24 PM apignoloni  says:

I have not touched the first line of the script and the shell I use is integrated into the system vmware ESXi 3.5.I try to move the script in the root.

Jan 22, 2009 12:32 PM apignoloni  says:

The command echo $SHELL returns:/bin/ash
The command vmware -v returns:VMware ESX Server 3i 3.5.0 build-123629
Moving the file in the root me back the message: Operation not permitted

Jan 22, 2009 12:38 PM lamw  says: in response to: apignoloni

Okay so looks like you're running the the latest ESXi 3.5u3, which is the same release I just finished testing as well and I don't see any issues. Since you're running on ESXi, there will not be a directory /root, just /.

It's going to get pretty difficult to troubleshoot exactly what the issue is, for some reason the system isn't liking the script. If you understand what the script is doing or have some scripting experience, you can setup some basic echo statements and see exactly where the script stops execution. From the looks of it, it seems to get through all the user defined variables but halts right before the actual code starts. If you can narrow down on the junk of code that it fails on, perhaps I can assist you further.

Also, your two VM(s) have the following displayName?

windows xp
ubuntu server 32bit

Actually locate the following line and change that from

# DO NOT MODIFY PAST THIS LINE #


to

exit

# DO NOT MODIFY PAST THIS LINE #


I'm wondering if the issue is after the variable declaration or after.

=========================================================================
--William
VMware ESX/ESXi scripts and resources at: http://engineering.ucsb.edu/~duonglt/vmware/

Jan 22, 2009 12:53 PM Thomas BRETON  says: in response to: lamw

Hello,
I don't know really the difference between vcbmount and snapshots.
I know that vcbmount work on vmdk files to compact them.
The NFS server is a FreeNAS on a 2x3Ghz xeon with one disk of 300Gb 15K U320.
I'm using ESX 3.5.

I don't think it's related to nfs configuration or server performance because for the same vm and for the same nfs server the vcbmount take 13 min and your script take 180 min
And yes it's the same problem for all machines

I'm still investigating on this
Regards
Tom

Jan 22, 2009 12:58 PM RENATOG123  says:

Can you tell me what variables need to be changed (other than NFS_SERVER and VM_BACKUP_DIR) in order for this script to work?
Also, how can I create a script that will initiate a snapshot of each VM, store them in the NFS (for daily backups) and another script that will do a full backup of each VM?
We are using ESXi 3.5u3
Thanks

Jan 22, 2009 1:00 PM lamw  says: in response to: RENATOG123

Please take a look at the documentation for your questions, it should answer majority of them.


=========================================================================
--William
VMware ESX/ESXi scripts and resources at: http://engineering.ucsb.edu/~duonglt/vmware/

Jan 22, 2009 1:00 PM apignoloni  says:

I am not an expert in scripting and general Linux environment, I am a beginner.
Yes, my two VM(s) have that name.
Also by entering the command exit the error is always in the same line in the script

Jan 22, 2009 1:11 PM lamw  says: in response to: apignoloni

The "exit' you inserted, allowed me to see where the issue is occuring. It's telling to the script to basically exit once it hits that line. This means the error is caused during the variable declaration section before it even starts to execute.

This brings me to believe the file may have some additional characters that may not be part of the original script/etc, perhaps characters from a Windows system. Did you download this from your Ubuntu box and scp the script to your ESXi host?

Can you run the following:

wc -l ghettoVCB-enhance.sh


wc -c ghettoVCB-enhance.sh


Also can you paste the variables you've modified from the default script and please format the output using the following tags {quote} ...your output... {quote}, it makes it easier for me to read and try to troubleshoot


=========================================================================
--William
VMware ESX/ESXi scripts and resources at: http://engineering.ucsb.edu/~duonglt/vmware/

Jan 22, 2009 1:11 PM RENATOG123  says: in response to: lamw

I'm sorry, can you provide a link for the documentation?
Thanks

Jan 22, 2009 1:17 PM lamw  says: in response to: RENATOG123

If you're posting in this section of the forum, you're already looking at the documentation. Please go over the instructions carefully, I do ask that new users read through the documentation before posting a question as you can answer most of your questions this way. You'll see that both your use cases can be solved by using this script and may require slight modifications, again this script is intended to help backup VM(s) that may reside on ESX/ESXi 3.5+. This script will not cover all processes/requirements for a backup or recovery scenario, I leave it to the discretion of the end user to understand what this script is doing and making modifications that fit your operating environment and needs.

You're welcome to post follow-up questions if you're still unsure of how the script operates or question about the functionality of the script. Hope this helps

=========================================================================
--William
VMware ESX/ESXi scripts and resources at: http://engineering.ucsb.edu/~duonglt/vmware/

Jan 22, 2009 2:04 PM apignoloni  says:

Problem solved: the file ghettoVCB-enhance.sh downloaded from the link higher contains the end of each line the notorious ^M, while the other file ghettoVCB.sh is clean
Thanks to all.

Jan 22, 2009 2:47 PM nappyrat  says:

William,

I was wondering if you could recommend teh best way for recovering a vm from a backup like this. Like say your esxi server blew up one day and you put a brand new one in it's place and hooked it up to the nas you had bacccked up your vms to.

most of hte instructions out there involve manually creating a vm in vic like the one you used to have, deleting it's hard drive, cloning the backup disk into your vm dir with vmkfstools -i and then adding it to your virtual machine.

since all the vm information is stored in a .vmx it seems like it should be easier to restore a vm, is it just that there aren't vm creation tools at the command line that can be used?

Jan 22, 2009 3:16 PM lamw  says: in response to: nappyrat

There are many scenarios in terms of recovery and this really depends on your requirements/environment. If you look at the simplest case where you have a new ESX/ESXi host and you present the NFS datastore which contains the backup VM(s) to the host. All you really need to do is just browse the datastore using the VIC and click into each of your VM(s) and right click and add to inventory. You should see your VM registered to the host and you can power them VM on. This assumes that want your recovered VM(s) running on your backup datastore, it's probably best to definitely copy the VM to your new host, register and power on. Going off of that, yes you'll need to copy the .vmx and copy the VMDK(s) associated with that VM and you should be using vmfkstools -i as you've suggested. Again once you understand this process manually, it's relatively easy to automate and script this process. Again, this process will vary from user to user and based on your requirements, you'll come up with a solution hat fits your environment.


=========================================================================
--William
VMware ESX/ESXi scripts and resources at: http://engineering.ucsb.edu/~duonglt/vmware/

Jan 22, 2009 3:28 PM nappyrat  says: in response to: lamw

gotcha, thanks again for your work!

Jan 23, 2009 6:38 AM apignoloni  says:

Good morning,
I created the script backupvm in /usr/local/bin running:
#!/bin/sh
ssh root@192.168.100.165 "sh /vmfs/volumes/datastore1/ghettoVCB.sh /vmfs/volumes/datastore1/vm_list"
#invio promemoria
echo backup vm posmailbe del `date ‘+%d-%B-%Y’`|mail -s “backup vm posmailbe” \ aaaaaa@iol.it

I also assigned the permission of execution.

I changed the file cron as follows:

americo@ubuntuserver32:/usr/local/bin$ sudo crontab -l

  1. m h dom mon dow command
15 15 * * * /usr/local/bin/backup_vm

but at the established nothing happens. The commands are formally correct?
Thank you

Jan 23, 2009 7:31 AM lamw  says: in response to: apignoloni

When setting up a cron entry, make sure the paths to all the commands use are the fullpath.

Please take a look at this thread to ensure they are: http://communities.vmware.com/message/1151164#1151164

I believe the vmkfstools is not being passed in my script as the full path, so you'll probably have to fix at least that command.


=========================================================================
--William
VMware ESX/ESXi scripts and resources at: http://engineering.ucsb.edu/~duonglt/vmware/

Jan 26, 2009 10:37 AM ollivetti  says: in response to: lamw

Hello, I have a question: Is it possible to backup additional added disks too? We have the OS on disk1 and the files on disk2. Does this backupscript support this. I could not find anything and tying it only made a backup of disk1.
Another thing I would like to ask is if you know any troubles using this script wiht ESX Version 3.02.

Thank you very much - also for the hard work writing this script!
Regards

Jan 26, 2009 10:38 AM lamw  says:

Yes, please download the latest script ghettoVCB-enhance.sh which will backup all valid VMDK(s) presented to the VM. This was added awhile back but I guess I must have forgotten to update the doc. I'll make sure I do that tonight and also remove the old instance of the script.


=========================================================================
--William
VMware ESX/ESXi scripts and resources at: http://engineering.ucsb.edu/~duonglt/vmware/

Jan 26, 2009 11:13 AM ollivetti  says: in response to: lamw

I just read in the comments about the grep -i fileName change...It now works ...
That's really great work you've done! Respect!
Thank you

Jan 26, 2009 11:48 PM ollivetti  says: in response to: lamw

Sorry one more stupid question - does it also work on ESX3.02? I see the note "You*'re not running ESX 3.5+ ....
Or are there known issues using the script with 3.02.
Thank you!

Jan 27, 2009 7:24 AM lamw  says: in response to: ollivetti

Hi ollivetti,

Sorry, this script only supports ESX 3.5+, the requirement is stated in the title and also noted in the body of the document. This was intentional since majority of the users were also using this on ESXi and the only common toolset between ESXi and ESX is vimsh. You probably can slightly modify it to support older version of ESX, as vimsh does exists going back to 3.0, but I have not tested that far back. You can uncomment the section of the code that blocks it from executing if you're not running either ESXi or ESX 3.5+.


=========================================================================
--William
VMware ESX/ESXi scripts and resources at: http://engineering.ucsb.edu/~duonglt/vmware/

Jan 27, 2009 1:21 PM RENATOG123  says: in response to: lamw

lamw,
I was finally able to get the NFS setup working with WSFU (in case anyone's pulling their hairs out, make that when you configure User Name Mapping, you map the LOCAL ADMIN account (not a domain admin) to "root")

I tested with a live running VIMA and it did a full backup of the VM.

I want to modify the script to do regular snapshots, not clone snapshots. Where/What do I have to change on the ghettoVCB-enhanced.sh script?
Thanks

Jan 27, 2009 1:24 PM lamw  says: in response to: RENATOG123

I'm not sure what you mean by regular snapshots, I'm aware of only one type and the snapshot can be taken with/without the state of the memory. Please look into script on where the snapshot is being taken, I believe it's documented in the script if you go through it.


=========================================================================
--William
VMware ESX/ESXi scripts and resources at: http://engineering.ucsb.edu/~duonglt/vmware/

Jan 28, 2009 5:56 AM RENATOG123  says: in response to: lamw

What would be the procedure to restore the ghettoVCB backups?

Jan 28, 2009 6:13 AM nappyrat  says: in response to: RENATOG123

the procedure i use is

If you just need to restore a vm to a state it was during the backup, and you still have your existing vm:

1. delete the hard drive of the vm from within vic, choosing to remove tne files from the disk as well.
2. clone the disk from backup to the directory of your vm with
vmkfstools -i /path/to/backup/disk.vmdk /path/to/vm/
3. add the hard drive to your vm choosing to use an existing virtual disk, select the /path/to/vm/disk.vmdk that you just made

If you need to restore the vm to another host or simply deleted it all together

1. make a directory /vmfs/volumes/whatever/vm-name
2. copy the .vmx file from backup to this dir
3. clone the disk from backup to the directory of the vm
vmkfstools -i /path/to/backup/disk.vmdk /vmfs/volumes/whatever/vm-name/
4. Browse the datastore in VIC, find the .vmx file, right click and select add to inventory.

not sure if theres a better way of doing it but it works for me.

Jan 28, 2009 6:35 AM RENATOG123  says: in response to: nappyrat

How were you able to create a directory? When I try "mkdir whatever" I get: Operation not permitted.
I am logged in as "root". Am I missing something?

Thanks

Jan 29, 2009 1:38 AM ollivetti  says: in response to: lamw

Hi, first of all thank you again for your response! I see you have quite a lot of questions to answer. One thing I would like to ask because I have problems is: I am using 3.02 and don't have vmware-vim-cmd so I use vmware-cmd or shall I use vimsh -n -e to execute commands. Unfortunately the execution stops here: echo "Error: failed to extract VM_ID for ${VM_NAME}!" Does it mean that the script is looking for der VirtualMachine ID in /tmp/vms_list.
Regards

Jan 29, 2009 8:27 AM lamw  says: in response to: ollivetti

Please take a look at the vimsh documentation to understand how it works.

http://engineering.ucsb.edu/~duonglt/vmware/#vmware_vimsh

=========================================================================
--William
VMware ESX/ESXi scripts and resources at: http://engineering.ucsb.edu/~duonglt/vmware/

Jan 30, 2009 9:07 AM ibhp  says: in response to: lamw

Hello

Thank you for this script! Really good work. The only thing that does not seem to work for me is backing up more than one vmdk in a vm. I tried several times with no luck. Do I have to do something special to get it to work?

Thank you for an answer.

Jan 30, 2009 9:14 AM lamw  says: in response to: ibhp

Are you running ESXi? If so, you might need to change the line "grep fileName" to' grep -i fileName".

Multi-VMDK(s) are supported, so long they're valid and presented to the guestOS.


=========================================================================
--William
VMware ESX/ESXi scripts and resources at: http://engineering.ucsb.edu/~duonglt/vmware/

Jan 30, 2009 11:01 AM ibhp  says: in response to: lamw

Yes, I am running ESXi. If I run the command:

VMDKS_FOUND=`grep scsi "${VMX_PATH}" | grep -i fileName | awk -F " " '{print $1}'`

on a cmd-sh, it gives me both scsi-vm-hdd... so thats all ok. But when I run your script, only the first (50G) HDD is backed-up, the second HD (300GB) not.

Do you have any ideas what could cause this problem?
Thank you very much!

Jan 30, 2009 11:29 AM nappyrat  says: in response to: ibhp

you have to modify the script to have grep -i as well, it only uses grep without the -i which only returns matches to the exact case fileName. esxi servers sometimes have it as filename (lower case n)

Jan 30, 2009 12:34 PM ibhp  says: in response to: nappyrat

Thank you for the response.

Of course I also modified the script... But still got only one vm-hdd backuped. I tried to troubleshoot in the script and what I found is the following. After the line:
grep "${SCSI_ID}.present" "${VMX_PATH}" | grep "true" > /dev/null 2>&1
it gives me the Value "1" for the second vm-hdd, which probably means it is not present??? But it is my data-disk which IS present in the vm... what do I have to modify?

Thank you.

Jan 30, 2009 1:10 PM lamw  says: in response to: ibhp

if you get a value of 1 for that particular line, it means that within the .vmx file for the ${SCSI_ID}.present = "true". This means that the VMDK was either removed from the VM but not deleted from the filesystem or it was removed and deleted from the filesystem. In either case, the reference is left in the .vmx file and the flag is just changed from true to false.

To verify this, you can look at your .vmx file, and if the code is returning a 1, it means that disk is not present to the VM which means it should not be showing up in the VI Client when you go into it's properties unless the .vmx file has been manually modified without the ESX/ESXi knowing about it.

=========================================================================
--William
VMware ESX/ESXi scripts and resources at: http://engineering.ucsb.edu/~duonglt/vmware/

Jan 30, 2009 4:43 PM ibhp  says: in response to: lamw

That seems to be strange... if you look at the content of my vmx-file:
/vmfs/volumes/497b9254-df6992a6-9fc0-0090270f57c5/Win2K3 TS # cat Win2K3\ TS.vmx
config.version = "8"
virtualHW.version = "4"
floppy0.present = "true"
nvram = "Win2K3 TS.nvram"
deploymentPlatform = "windows"
virtualHW.productCompatibility = "hosted"
tools.upgrade.policy = "useGlobal"
powerType.powerOff = "default"
powerType.powerOn = "default"
powerType.suspend = "default"
powerType.reset = "default"

displayName = "Win2K3 TS"
extendedConfigFile = "Win2K3 TS.vmxf"

numvcpus = "2"
scsi0.present = "true"
scsi0.sharedBus = "none"
scsi0.virtualDev = "lsilogic"
memsize = "1024"
scsi0:0.present = "true"
scsi0:0.fileName = "Win2K3 TS.vmdk"
scsi0:0.deviceType = "scsi-hardDisk"
ide0:0.present = "true"
ide0:0.clientDevice = "FALSE"
ide0:0.deviceType = "cdrom-image"
ide0:0.startConnected = "TRUE"
floppy0.startConnected = "false"
floppy0.clientDevice = "true"
ethernet0.present = "true"
ethernet0.networkName = "VM Network"
ethernet0.addressType = "generated"
guestOSAltName = "Microsoft Windows Server 2003, Standard Edition (32-bit)"
guestOS = "winnetstandard"
uuid.location = "56 4d 61 2a b1 4c 7f ca-b4 81 00 31 27 13 e1 0d"
uuid.bios = "56 4d 61 2a b1 4c 7f ca-b4 81 00 31 27 13 e1 0d"
toolScripts.afterPowerOn = "true"
toolScripts.afterResume = "true"
toolScripts.beforeSuspend = "true"
toolScripts.beforePowerOff = "true"

ide0:0.fileName = "/vmfs/volumes/497b9254-df6992a6-9fc0-0090270f57c5/Windows Server 2003/Windows Server 2003.Enterprise.German.ISO"
floppy0.fileName = "/dev/fd0"

scsi0:0.redo = ""
ethernet0.generatedAddress = "00:0c:29:13:e1:0d"
tools.syncTime = "FALSE"
sched.swap.derivedName = "/vmfs/volumes/497b9254-df6992a6-9fc0-0090270f57c5/Win2K3 TS/Win2K3 TS-af7ea147.vswp"
ethernet0.generatedAddressOffset = "0"
hostCPUID.0 = "0000000a756e65476c65746e49656e69"
guestCPUID.0 = "0000000a756e65476c65746e49656e69"
userCPUID.0 = "0000000a756e65476c65746e49656e69"
hostCPUID.1 = "00010676000208000008e39dbfebfbff"
guestCPUID.1 = "0001067800010800000802010febbbff"
userCPUID.1 = "0001067600020800000802010febbbff"
hostCPUID.80000001 = "00000000000000000000000120100000"
guestCPUID.80000001 = "00000000000000000000000000100000"
userCPUID.80000001 = "00000000000000000000000120100000"
evcCompatibilityMode = "FALSE"
scsi0:1.present = "TRUE"
scsi0:1.deviceType = "scsi-hardDisk"
scsi0:1.filename = "Win2K3 TS_1.vmdk"
scsi0:1.mode = "persistent"
scsi0:1.redo = ""
****************************************
Here you can see that scsi0:0 as well as scsi0:1 is present and true.... And I am working with this device every day, so it IS present...

Do you have any idea? Thank you very much for your help, I am really out of any guess....

Jan 30, 2009 4:49 PM lamw  says: in response to: ibhp

Look at what the script is doing .... it's grepping for "true" ... looks like your .vmx has "TRUE", see the problem.

I'm curious on why some of the user's .vmx files are slightly different from the one's I've worked with. I'm not sure if this is due to any specific version or user's are manually modifying their files. I've always seen the right hand side use lower case values, but I have seen it few times in older releases where it's all caps. I'll try to find some time this weekend to fix those two issues, it should be using grep -i.

That should fix your issue


=========================================================================
--William
VMware ESX/ESXi scripts and resources at: http://engineering.ucsb.edu/~duonglt/vmware/

Jan 31, 2009 3:03 AM ibhp  says: in response to: lamw

Thank you, it's now working like a charme!

But does somebody know how to start this script automatically with some kind of "cronjob" in ESXi? I tried the idea with "autobackup.sh" and included the "cron" directory in this script, but it does not seem to work... (Always get the following error:

diff: Can't stat '/tmp/auto-backup.237536.dir/var/spool/cron/crontabs/root': No such file or directory
which means this temporary directory/file has not been created by the script...

Thank you!

Jan 31, 2009 3:52 AM nappyrat  says: in response to: ibhp

I did it by adding the following lines to /etc/rc.local

/bin/echo "0 19 5 * * /vmfs/volumes/vmback/ghettoVCB.sh" >> /var/spool/cron/crontabs/root
/bin/busybox crond -b

then run auto-backup.sh it will backup the changes to the rc.local file

basically this makes it so on boot the ghettovcb entry is added to the crontab and crond is restarted.

Jan 31, 2009 8:24 AM lamw  says: in response to: nappyrat

I believe these changes do not persist do a reboot? I remember talking to someone about this and on reboot these changes are reverted to their original configurations, including the cron entry. Is that correct?


=========================================================================
--William
VMware ESX/ESXi scripts and resources at: http://engineering.ucsb.edu/~duonglt/vmware/

Jan 31, 2009 8:32 AM nappyrat  says: in response to: lamw

the auto-backup.sh script which runs like every hour or so i think backs up a number of files in /etc to be persistant upon reboot. this is why things like modifying inetd.conf to enable ssh still work after reboot. /etc/rc.local is another one that changes will be saved. that is why i put the code in there to add the crontab entry on boot.

Jan 31, 2009 2:07 PM lamw  says: in response to: nappyrat

Ah good to know, I'm aware of the backup but I thought someone had mentioned earlier in another forum thread that files are not persisted through a reboot. That is good to hear, I'll try to find some time to update the doc stating that fact. Thanks


=========================================================================
--William
VMware ESX/ESXi scripts and resources at: http://engineering.ucsb.edu/~duonglt/vmware/

Feb 1, 2009 2:42 AM ibhp  says: in response to: lamw

Hello

Yes, I can confirm the Backups are persistent when you write the stuff in rc.local. All runs without problems.

Thank you for all the good tipps!

Feb 10, 2009 1:50 PM cybersayo  says: in response to: ibhp

First, thank for the great script lamw!

Everything works well when I do ghetto from SAN to local hard drive, but when I try gheto to NFS I get
"DiskLib_Check() failed for source disk The system cannot find the file specified (25)" after cloning and snapshot remove lasts about 2 mins with lot of "...missing *-delta.vmdk" in ESX's /var/log/messages. There is no difference with Windows or Linux NFS server - backed vmdk looks fine but I don't know is this only warning or something to worry about. It's same story with all my ESXs.

Feb 10, 2009 1:59 PM lamw  says: in response to: cybersayo

Usually when you get those warnings, it means it was not able to find the source VMDK for whatever reason. I know some user's have had issues in the past where their VMDK(s) were stored in other directories or if they had multiple VMDK(s), earlier versions of the script did not handle that use case.

The latest script should be validating that certain VMDK(s) exists prior to trying to take a backup, though if you've manually edited the .VMX file, there could be some inconsistencies in the .vmx itself. You can always try the script with a -x flag which gives detail verbosity of what's happening while the script execute and see if you can anything that's off. If it works for SAN to Local, then it should also work for NFS unless you some miss-configurations on your NFS Server (you can always manually take a snapshot and manually run vmkfstools copy manually to make sure it's working with your NFS Server)

(e.g.)

sh -x ghettoVCB.sh my_back_vm_list.txt


If you're still not sure, you can direct that output to a file and attach to your reply and I can try to take a look.

Feb 11, 2009 12:41 AM cybersayo  says: in response to: lamw

Thanks, I'll try.

Another thing; currently I do three kind of backups from san: daily to local hard drive (raid1), dayly to nfs (raid1) and weekly to tape. Since there is not enough space on local hard drives to do rotation, ghetto returns error every second day (and delete files then, so backup is ok tommorow). Rotation count is 0.

What can I do to make this backup daily?

Feb 11, 2009 8:46 AM lamw  says: in response to: cybersayo

Well nothing is stopping the script do run daily backups, that is up to the end user's and the setup of their cron schedule. If you're looking for more elaborate backup processes such as transfer from SAN to local, then to NFS, etc, etc. You'll need to script that process, this script is intended to help provide a backup solution for your VM(s), it's left open for any modifications/specific requirements one may have for their environment.

=========================================================================
--William
VMware ESX/ESXi scripts and resources at: http://engineering.ucsb.edu/~duonglt/vmware/

Feb 12, 2009 1:07 PM ddigioia  says: in response to: lamw

William,

Thanks for sharing your great scripts and knowledge.

Everyone,

Thanks for the great forum. I've learned a lot here.

We are using ESXi with an NFS datastore on Openfiler 2.3.

I've modified William's script to make ESXi "thin" disk format backups, as it was not too easy to add backups or clones of VMs back to the VM inventory when they were in 2gbsparse disk format, and I did not want to lose the disk space "thick" backups would have required.

The backups and restore/clone seem to work fine with thin format, but I feel it seems too easy. Am I missing something?

Is there any reason why I should not use William's ghettoVCB.sh script to do thin backups rather than thick or 2gbsparse?

(I realize the 2gbsparse format is more portable to other filesystems, other versions of VMware, etc.)

Thanks,

-David

Feb 13, 2009 5:54 AM jaxislnad  says:

First off, this is really some great work. Thank you for posting this.

But, unfortunately I am having some issues. I downloaded the file yesterday 2-12-2009, onto my ubuntu machine. I opened it with gedit and it had the ^M throughout the file. I dont know if Ubuntu did that or the file up there has them but either way I fixed it with dostounix. So I set it up and ran it on a VM with 1 vmdk file and it worked like a charm. HDD to HDD on a 10 GB VM takes consistently less than 5 mins, which is great.

My next step was I added a vmdk with the VIC, I used the VM and added a partition and put files on the new hard drive, works fine. When I run the backup it still only backups the main HDD, or the first vmdk. I checked the vmx file and they are both set to "true". I checked your script and it is grep -i. So I am totally at a loss at what could be preventing the script from backing up the second vmdk.

Now my next question would be I have 4 HDD in the server. Disk 1 has a vmdk, Disk 2 has a vmdk, both for the same VM. Can I backup Disk 1 vmdk to Disk 3, and Disk 2 vmdk to Disk 4. All 4 drives are 1 TB drives. So both vmdk files are more than 1 TB which is the size of the other disk so I need each vmdk to go to a seperate location on the same server. I hope this makes sense if not let me know.

Also my last question is how does this backup run while the machine is on, can I still get a restorable copy of my files if I do the backup when the machine is powered on? Is it best to power down then do the backup?


********EDIT********02-13-2009 09:12
The 2nd vmdk that doesnt get backed up is stored on another drive, I added a 3rd disk to the same directory as the vmx and it backs up both, so for some reason it cannot find the 2nd vmdk on another drive.
Im also getting this when it tries to clone the disk:
DiskLib_Check() failed for source disk The system cannot find the file specified (25).

********EDIT********02-13-2009 09:45
I have attached the portion of my file that I believe to show the issue. As you can see the _2.vmdk and regular vmdk that are in the same directory as the vmx file backup fine. These files are in /vmfs/volumes/Main

Now the _1.vmdk is on /vmfs/volumes/Third, which it calls out by the long string of numbers which also match the string of numbers in the vmx file.

What looks wrong to me is it is starting out in the /Main directory then tries to go find it but cant. I could use some guidance on this one.

+ echo ################## Starting backup for MyVM ... #####################
+ OLD_IFS=

+ IFS=:
+ VMDK=MyVM_2.vmdk
+ 0 -eq 1
+ vmkfstools -i /vmfs/volumes/Main/MyVM/MyVM_2.vmdk /vmfs/volumes/Secondary/MyVMbkup/MyVM/MyVM-2009-02-13/MyVM_2.vmdk
+ VMDK=/vmfs/volumes/4896efad-39367fad-4058-00151762f7d0/MyVM/MyVM_1.vmdk
+ 0 -eq 1
+ vmkfstools -i /vmfs/volumes/Main/MyVM//vmfs/volumes/4896efad-39367fad-4058-00151762f7d0/MyVM/MyVM_1.vmdk /vmfs/volumes/Secondary/MyVMbkup/MyVM/MyVM-2009-02-13//vmfs/volumes/4896efad-39367fad-4058-00151762f7d0/MyVM/MyVM_1.vmdk
DiskLib_Check() failed for source disk The system cannot find the file specified (25).
+ VMDK=MyVM.vmdk
+ 0 -eq 1
+ vmkfstools -i /vmfs/volumes/Main/MyVM/MyVM.vmdk /vmfs/volumes/Secondary/MyVMbkup/MyVM/MyVM-2009-02-13/MyVM.vmdk
+ IFS=

I hope you can help me out, I do appreciate you taking a look.

Thanks again for the great work.

-Ryan

Feb 13, 2009 8:27 AM lamw  says: in response to: jaxislnad

I noticed another user had a similar issue too, I may have to look at the case when the VMDK(s) are stored in another datastore other than the root VM directory. I'm curious when it tries to backup the other drives, is the location correct? In terms of how it's able to backup VM(s) while it's still running is with a snapshot. When you take a snapshot, the main VMDK is changed to readonly and a new delta file is created which will contain all changes after the snapshot has been taken. This then releases a file lock on the main disk which allows you to backup the disk. Once the snapshot has been removed (committed), all changes will merge back with the primary disk. Then on the next backup, it'll do the same and those changes will be captured.

I'll take a look at the issue sometime next week ... need to spend sometime with the gf this weekend =)


=========================================================================
--William
VMware ESX/ESXi scripts and resources at: http://engineering.ucsb.edu/~duonglt/vmware/

Feb 13, 2009 8:31 AM jaxislnad  says: in response to: lamw

The location is correct that it comes up with. It matches the same string thats in the vmx file so it almost knows its there it just cannot copy it.

Also is there a way to backup one vmdk to one location and a second vmdk to another, even though they are in the same VM?

I thought it was using the snapshot feature, but I wasnt sure. I appreciate the update, and have a good weekend! :-)

Feb 13, 2009 8:37 AM lamw  says: in response to: jaxislnad

hm ... that's interesting. If the snapshot was taken successfully, then both those disks should be unlocked. You can always manually do what the script is doing to just verify that it should work. In terms of your second question, you would need to modify the script to duplicate it to two destinations. You can always just re-run the script with a second destination as a work around.


=========================================================================
--William
VMware ESX/ESXi scripts and resources at: http://engineering.ucsb.edu/~duonglt/vmware/

Feb 14, 2009 2:01 PM jsmithsd  says: in response to: lamw

Has anybody come up with a reliable method of monitoring this script for success/failure? Here's an idea off the top of my head for Windows users who use Kaseya/LPI for monitoring. I wouldn't have the expertise to write something like this but maybe somebody else would see value in this.

  • Have ghettoVCB produce a log file. Does it already do this?
  • Modify the .bat file in your scheduled task to download the log file to the Windows machine via Plink.

From there you could:
  • Launch a .vbs that would parse the log and write custom events to the event viewer. Monitor events with your monitoring tool of choice.
  • Parse the log file directly if your monitoring tools are capable of this (Kaseya is).

Feb 16, 2009 8:01 AM jaxislnad  says: in response to: lamw

How do I run it with 2 destinations, wont it copy the first vmdk each time. So I would only end up with one of the 2 vmdks in 2 places?

Im not sure where in the script I should try manually.

Thanks for the help.

Feb 16, 2009 9:49 AM lamw  says: in response to: jaxislnad

You'll have to add that logic into the script if you're looking to split your destination copy.


=========================================================================
--William
VMware ESX/ESXi scripts and resources at: http://engineering.ucsb.edu/~duonglt/vmware/

Feb 17, 2009 2:16 AM JoeyDE  says:

Many thanks for your great script!!

To everyone:

Everything works great except the output to a log file. I have a scheduled task on a windows 2003 server machine to run batch file which executes ghettoVCB.sh via plink.exe. Thanks to Raj Perumal for his blog guide.

The command looks like this:
plink.exe root@192.168.100.200 -pw mypassword "nohup ./vmfs/volumes/vmfs-storage/ghettoVCB.sh /vmfs/volumes/vmfs-storage/vmlist > /vmfs/volumes/vmfs-storage/backuplog.txt &"

After a successful backup the backuplog.txt contains not a single line and the file size is 0 btyes.

When I execute the command in a putty session the output to the backuplog.txt works great.

Any ideas whats wrong?

Feb 18, 2009 9:59 AM cyberfire  says: in response to: JoeyDE

I have a problem.....
It all works find but VM_BACKUP_ROTATION_COUNT=1 does not do any thing.
I have now
.15_SpiceWorks-2009-02-17
.15_SpiceWorks-2009-02-16
as backups and if I run it now I get
.15_SpiceWorks-2009-02-18

any ideas?

Feb 18, 2009 10:04 AM lamw  says: in response to: cyberfire

Looks like something was modified in the script because the rotation count is kept track by a special extension "--X" ... I don't see that in your output, did you change something in the script?

=========================================================================
--William
VMware ESX/ESXi scripts and resources at: http://engineering.ucsb.edu/~duonglt/vmware/

Feb 18, 2009 10:11 AM cyberfire  says: in response to: lamw

Config part
VM_BACKUP_VOLUME=/vmfs/volumes/489326a8-7adbcdf8-a9f4-00188bf6e6f5

POWER_VM_DOWN_BEFORE_BACKUP=0

Code part
LIST_BACKUPS=$(ls -Atr ${BACKUP_DIR_PATH})
This is due to that my VM names are .15_Name_XXXX

Feb 18, 2009 10:13 AM cyberfire  says: in response to: cyberfire

BTW, wnat can I comment out in order to run all but the actual cloning of the folder?

Feb 18, 2009 10:19 AM cyberfire  says: in response to: cyberfire

OK, I recopied the original code.
this is my output.

~ # /ghettoVCB.sh /backup_vms

                                1. Taking backup snapshot for .15_SpiceWorks ... ################
Destination disk format: VMFS thick
Cloning disk '/vmfs/volumes/Virtual_OS_V2/SpiceWorks/XP_Base_1.vmdk'...
Clone: 100% done.
Removing snapshot from .15_SpiceWorks ...
bash: /vmfs/volumes/489326a8-7adbcdf8-a9f4-00188bf6e6f5/.15_SpiceWorks/.15_SpiceWorks-2009-02-17: bad number
/ghettoVCB.sh: /ghettoVCB.sh: 322: Syntax error: /vmfs/volumes/489326a8-7adbcdf8-a9f4-00188bf6e6f5/.15_SpiceWorks/.15_SpiceWorks-2009-02-17+1

Feb 18, 2009 10:21 AM lamw  says: in response to: cyberfire

Ah, then I think you can just remove the -A param, as that ignores "." Any reason why you're naming convention is in that form? I would probably change that since that usually interpreted as hidden directories (just a thought).


=========================================================================
--William
VMware ESX/ESXi scripts and resources at: http://engineering.ucsb.edu/~duonglt/vmware/

Feb 18, 2009 10:24 AM lamw  says: in response to: cyberfire

I'm not sure what you mean by actual cloning of the folder? If you're asking about the rotation count, you'll need to set that = 1, it does not make sense to set it to 0.

You should give the script a read through, the logic should not be too difficult to follow and you're welcome to ask further questions. If you can manually run through this process, then the script will make more sense.


=========================================================================
--William
VMware ESX/ESXi scripts and resources at: http://engineering.ucsb.edu/~duonglt/vmware/

Feb 18, 2009 10:26 AM lamw  says: in response to: cyberfire

Again, this is probably due to your naming convention, so it's unable to rename the directory since it's looking for a "--X' extension. Please take a look at the previous reply for a fix or modify the way you VM directory.


=========================================================================
--William
VMware ESX/ESXi scripts and resources at: http://engineering.ucsb.edu/~duonglt/vmware/

Feb 18, 2009 10:31 AM cyberfire  says: in response to: lamw

Just changed the name of the VM. No .15_ as prefix.
Running the code.....
Lets see.

Feb 18, 2009 10:47 AM cyberfire  says: in response to: cyberfire

What I ment about cloning the folder... was that I would like to coment the part that actualy does the backup until I get the rotation count working

BTW, VM_BACKUP_DIR_NAMING_CONVENTION="$(date +%F)" is what is set for the naming.

Feb 18, 2009 10:54 AM cyberfire  says: in response to: cyberfire

is Date: 11/01/2008 the last version of the file? I see updates later than that

Feb 18, 2009 10:57 AM lamw  says: in response to: cyberfire

That is actually the date I started the project. The latest script will always be available in the download section

To your other comemnt, that is the naming convention that is appended to the backup directory, this can be anything but I thought having the date as part of the backup directory might be useful. This can be changed to any string you would like.


=========================================================================
--William
VMware ESX/ESXi scripts and resources at: http://engineering.ucsb.edu/~duonglt/vmware/

Feb 18, 2009 11:15 AM cyberfire  says: in response to: lamw

mmmm
now its creating
SpiceWorks-2009-02-18--1
I will rename to 02-17--1 and run again.

Feb 18, 2009 12:11 PM cyberfire  says: in response to: cyberfire

YES!!!!!!!!!!!!!!!!!!!!!!
Thanks!!!!!!!!
Now I need to just find a way to get the files of the ESXi server.

Regards.
George.

Feb 18, 2009 12:13 PM lamw  says: in response to: cyberfire

Glad to hear. Try setting up an NFS Server, it's pretty simple and this will allow you to get the VM(s) off the host to other backup or tape storage.


=========================================================================
--William
VMware ESX/ESXi scripts and resources at: http://engineering.ucsb.edu/~duonglt/vmware/

Feb 19, 2009 7:08 AM ScottChapman  says: in response to: lamw

When I am running the script I am getting the following error:
DiskLib_Check() failed for source disk The system cannot find the file specified (25).

I turned on tracing on the script and discovered it is happening as a result of this command:
vmkfstools -i /vmfs/volumes/datastore2/OpenFiler//vmfs/volumes/4947b0ac-5b9a4e28-9b21-0011092a9380/OpenFiler/OpenFiler_2.vmdk /vmfs/volumes/Disk1/vmbackups/OpenFiler/OpenFiler-2009-02-19//vmfs/volumes/4947b0ac-5b9a4e28-9b21-0011092a9380/OpenFiler/OpenFiler_2.vmdk

Any ideas what is causing this?

Feb 19, 2009 7:12 AM santalaus  says: in response to: ScottChapman

Realy nice Work, but I found an Izzue ;)
When you have multiple vmdks on different datastores in one vmx thd filename contains the fullpath to the vmdk
So the Script could not save this vmdk because it is not in the VMX_DIR Path.

I hope you understand this.

Feb 19, 2009 7:16 AM jaxislnad  says: in response to: ScottChapman

Same issue Im having. Im trying to modify the script but no success so far.

Feb 19, 2009 7:17 AM ScottChapman  says: in response to: santalaus

Nope, sorry you lost me.

I do have a couple of different VMs on different datastores, but I got this error when I tried to back up just one of them.

Are you saying that all my VMs need to be on the same datastore for the script to work?

Or am I just not understanding this...

Feb 19, 2009 7:20 AM jaxislnad  says: in response to: ScottChapman

My situation is my VM's have 2 vmdk files. They are both on different datastores. When I run the script only the vmdk that is on the same datastore as the vmx gets backed up. The vmdk on the other datastore does not and it gives me the same error.

Feb 19, 2009 7:25 AM ScottChapman  says: in response to: jaxislnad

Oh wait, I think I (might) get it!

So, yes the VM I am backing up also has a couple of disks which do reside on different data store.

Fortunately I do not actually need to back those up (they are storage for openfiler which I am spanning across a software RAID).

I really just need to backup the main disk associated with the VM.

So I am guessing I can just ignore this error.

If you do happen to extend the script to support this, you might consider allowing the user to specify which disks to back up and which to ignore...

Feb 19, 2009 7:27 AM santalaus  says: in response to: jaxislnad

Jepp, this is because the filename of the vmdk on the diffrent Datastore contains the fullPath.

scsi0:0.fileName = "Fileserver.vmdk"
scsi0:1.filename = "/vmfs/volumes/4982de6f-6ea7343b-c3c9-00221986180a/Fileserver/Fileserver_1.vmdk"

So you could not easily use the VMDK Name as source or target for copy.
You have to check first the path an the make 2 arrys VMDKS and VMDKFULLPATH

Feb 19, 2009 8:24 AM lamw  says: in response to: santalaus

All, this has been acknowledged by a few as an issue, the current workaround is to only backup VM(s) that reside on the same datastore.

I've been occupied with other projects recently and when I get some time I'll take a look and provide a fix. If anyone has implemented a a working solution, feel free to post or send me a PM.


=========================================================================
--William
VMware ESX/ESXi scripts and resources at: http://engineering.ucsb.edu/~duonglt/vmware/

Feb 19, 2009 12:53 PM docsys  says: in response to: lamw

Anyone having issues downloading the ghettoVCB.sh from this page? I get it and it doesn't run. It shows a "Bad Interpreter" File Not Found error. It seems to be corrupt or have some control characters in it (^M). I can't see them in vim or vi and I only downloaded to a windows xp system, didnt edit it at all on windows.

It seems the ghettoVCB.sh on http://engineering.ucsb.edu/~duonglt/vmware/ and here on VMware site are different revisions, but the one from the ucsb site is working OK.

TIA.

Feb 19, 2009 1:01 PM lamw  says: in response to: docsys

Sorry about that, I must have forgotten to update the engineering account. They should be the same, I've uploaded the current version and can you try the download on the VMTN doc again?


=========================================================================
--William
VMware ESX/ESXi scripts and resources at: http://engineering.ucsb.edu/~duonglt/vmware/

Feb 19, 2009 1:18 PM ScottChapman  says: in response to: lamw

If you do have a problem with the trailing "^M" line termination you can do the following to get a good version of the script:

sed 's/.$//g' ghettoVCB.sh > ghettoVCBFixed.sh

Just make sure to turn the execution bit on...

Feb 19, 2009 1:36 PM docsys  says: in response to: ScottChapman

Alright it seems the file from the vmware site still has the ^M somehow, tried the sed command and it was OK.
The file from the ucsb site is updated and Good still.
Thanks.

Feb 20, 2009 10:19 AM Gregori  says:

How to change the ghettoVCB.sh to delete the old backup before execute new backup?

Thanks.

Feb 20, 2009 12:45 PM lamw  says:

I'm looking at the issue and basically it boils down to the SOURCE and DESTINATION path, the VMDKS array will contain either the local VMDK disk names with respect to the VM's folder which you'll need to append the VMX_PATH variable or if the disk is stored in another datastore, it'll contain the full path to the VMDK. A quick fix would to check the variable that goes through each for loop iteration to see if it starts with /vmfs/volumes, if so then you know it's a disk that's located in another datastore and setup the variable accordlying. Running through some print statements, I noticed an issue with the DESTINATION which also needs similar logic.

Though right now the issue from my basic test is the following: I have one Windows VM with 1 disk call it Quentin.vmdk which is the unique descriptor describing one of Quentin's disk that is stored in Datastore A. I know add a second disk and store i in Datastore B, I commit the changes and then look at the disk name from both the VIC and on the filesystem and what do I notice?

The name of the .vmdk is the same as disk 1, this will be an issue when backing up the disks to your destination as these filenames are no unique and will most likely over write one another. I then added a third disk two the VM and stored it in Datastore B and the .vmdk now has a unique name appended with _1.

The observation, at least on ESX w/SC (haven't tested on ESXi) if you have a VM with 1 disk and you add another and you're going to store it on another datastore the name will not be unique as it'll default to the VM's display name. This will cause a problem when backing up ... I'm trying to think of what changes would need to occur but it seems that it'll most likely requiring the modification of the .vmdk name's to the destination which may or may not be something end users want.

I'm still pondering over the issue right now over lunch ... if anyone has any suggestions I'm open, but this may be a feature that may not be supported due to the way the disk's are not uniquely labeled across datastores.


=========================================================================
--William
VMware ESX/ESXi scripts and resources at: http://engineering.ucsb.edu/~duonglt/vmware/

Feb 20, 2009 2:47 PM ScottChapman  says: in response to: lamw

I think what would make sense would be to create folders for the datastores names and put the files there.

This would also make it a lot easier to restore since one can easily tell what when where...

Make sense?

Feb 20, 2009 2:52 PM ScottChapman  says: in response to: ScottChapman

Or more technically, retain the path relative to /vmfs/volumes

Feb 20, 2009 2:55 PM lamw  says: in response to: ScottChapman

Unfortunately that does nothing for the current situation other than additional directories and a disconnect between what's listed in the .vmx file and the additional work a user would have to remove the disk reference and re-map on a backup/recovery. The solution would be too specific as some user's may want to mount the backup datastore directory whereas others would want to manually copy from the recovery datastore but not directly register the VM(s) from backup.

The more I think about this, the only decent solution is to automatically rename and update the backup VMDK(s) as they're being backed up and set them to be referenced locally within the VM's directory. This then allows a user to either mount the VM directly if he/she would need to recovery or restore from backup and have all VMDK(s) referenced properly. The only difference with VM(s) that were initially stored across multiple datastores will not be preserved (the ability to recovery may be more important than how the disks were initially stored). For further use cases, that'll be something end-users will need to customize for their enviornment.


=========================================================================
--William
VMware ESX/ESXi scripts and resources at: http://engineering.ucsb.edu/~duonglt/vmware/

Feb 20, 2009 2:58 PM lamw  says: in response to: ScottChapman

Even if you retain the path ... what's to say a user can't bring up the original datastores, they're recovering to another ESX/ESXi host with another datastore or local. This really goes hand in hand with recovery which is going to be unique for each environment. This script will not be able to handle all those use cases, hence it's up to the end users to customize as needed for their enviornment. We're trying to generalize the process so that such customization is available if necessary.


=========================================================================
--William
VMware ESX/ESXi scripts and resources at: http://engineering.ucsb.edu/~duonglt/vmware/

Feb 20, 2009 3:34 PM lamw  says:

How about the following as a compromise:

To preserve the functionality and familiarity of the script with user's that have already implemented a recovery/restore solution, I will continue to backup all VMDK(s) that are stored locally with the VM to the destination folder as you're aware of. The added fix would then support VMDK(s) that are not stored in the same datastore as the VM but in backing those VMDK(s) up to take into consideration of non-unique VMDK names provided by VMware, I would create an additional directory which will be the unique UUID of the original VMDK destination and then it would be up to the end user's to provide a solution on recovery/restoration.

This will preserve the original .vmx file as I have currently in the script and if a user chooses to restore a VM to it's initial configuration, the only added step would be to vmkfstools copy any VMDK(s) that were not stored with the VM to it's correct datastore. A user would also be able to directly mount the backup datastore but then would be required to manually update the location of the VMDK(s) that were not stored with the original VM ( this would have to be done even if the current script supported VMDK(s) across multiple datastores as your recovery .vmx file will not contain the correct paths )

Example:


  • DS1/VM_TEST
    • DS1/VM_TEST.vmdk
    • DS1/VM_TEST_1.vmdk
  • DS2/VM_TEST/VM_TEST.vmdk

Backup would be:

  • VM_TEST
    • VM_TEST.vmdk
    • VM_TEST_1.vmdk
    • DS2/VM_TEST.vmdk

I think this makes the most sense and preserves the original script's behavior and I can not stress this enough, that it's the responsibility of the end user's to design/automate a process that fits their operating environment. This script is to aide in backing up your VM(s) but as you know there will be different sets of requirements for environment and the specifics will need to be customized by the end users.


=========================================================================
--William
VMware ESX/ESXi scripts and resources at: http://engineering.ucsb.edu/~duonglt/vmware/

Feb 22, 2009 8:43 PM lamw  says:

Script has been updated, please let me for those that have multiple VMDK(s) across multiple datastores if this works. Thanks


=========================================================================
--William
VMware ESX/ESXi scripts and resources at: http://engineering.ucsb.edu/~duonglt/vmware/

Feb 23, 2009 4:12 AM santalaus  says: in response to: lamw

It seemed to work good.
Now I have an backup of the vmdk which where on the second datastore

Feb 25, 2009 4:09 AM bpsmanagement  says: in response to: lamw

First of all.. thank you William and all others who have contributed to this. The script is sweet. Much faster than other scripts that use other backup methods. I am however getting speeds that are unusually slow for file transfer on an internal LAN.
I am running ESXi embedded on my host and I have a Allegro NFS Demo running on a physical server 2008 box (attempted to configure Unix on server 2008 using native Windows tools with no luck; authentication issues). The export is located on my local server 2008 storage. It's a 1.5TB Barracuda. Both boxes have Intel Gigabit Nics and are connected VIA a Linksys Gigabit switch. I tested my transfer speeds using wireshark at 15.8MBit/sec while running this script. I tested windows transfer from my guest VM to my Windows 2008 server at ~320MB/s. I know it's not my network configuration because of the results I get with Windows. Is allegro just slow as mollases? Server 2008 and Allegro Hate eachother? I know Server 2008 hates Unix (unless it's solaris 10), but thats another story.. What numbers should we expect/shoot for? What are other seeing backing up to NFS storage?

Feeling down? Need a good laugh? 100GB of data split between two VM's..

Start time: Wed Feb 25 00:16:28 UTC 2009
End time: Wed Feb 25 11:56:51 UTC 2009
Duration : 700.38 Minutes

Completed backing up specified Virtual Machines!

Feb 25, 2009 11:08 AM cyberfire  says: in response to: bpsmanagement

Quick question.
How do I restore form the Backup?
I currently store the Backup on the same Datastore.
Do I just add to Inventory and run from there? or do I move the files to the original Folder?
What happens with multiple vmdks?

Feb 25, 2009 11:55 AM cyberfire  says: in response to: cyberfire

mmm
I just did a backup of a server that has its OS on one DS and the data on a second DS.
The only vmdk that was backedup was the data disk. the the actual OS vmdk was not.

Feb 25, 2009 11:58 AM lamw  says: in response to: cyberfire

Please make sure you're using the latest version.

Restore can be either direct mount from your backup datastore and just register or replicate from your backup datastore using vmkfstools. I would recommend that latter, that way you don't use your backup for use but replicate it to the host for recovery/restore. The recovery/restoration process is something that is more manual and can be scripted but the requirement will be based around your requirements and should be implemented based on that.


=========================================================================
--William
VMware ESX/ESXi scripts and resources at: http://engineering.ucsb.edu/~duonglt/vmware/

Feb 25, 2009 12:41 PM bpsmanagement  says: in response to: bpsmanagement

no comment?

Feb 25, 2009 12:46 PM lamw  says: in response to: bpsmanagement

not really, I'm not familiar with NFS off of Windows, there are many configurations that could cause the speeds you're seeing. If you look in the forums, you'll find various comments on good/bad speeds. If you're running an NFS Server on VM, you may see speed decreases than on a physical. Your export options can also affect the speeds (e.g. sync vs async). There have been many individuals getting very good speeds and many that get less than up to par/expected. Your best bet is to ask the community for their configurations but can also be due to the hardware you're using... unfortunately when you're discussing performance, ymmv depending on your setup.

The script has nothing to do with your performance if that's what you're after.

=========================================================================
--William
VMware ESX/ESXi scripts and resources at: http://engineering.ucsb.edu/~duonglt/vmware/

Feb 27, 2009 12:23 AM ibhp  says: in response to: lamw

hello all
I am backing up about 4 different vm-images each night with the script. But somethimes the snapshots are not removed after backing up and so the next time the backup of this vm will fail.

  • would it be possible to backup a vm also if a snapshot exists? What's the reason you check for existing snapshots?
  • How is it possible to send a mail after each backup with the content of the ghettovcb-script?

Thank you,

ibhp

Feb 27, 2009 8:12 AM lamw  says: in response to: ibhp

VM(s) w/Snapshots are not supported because snapshots in general should not be kept for long periods of time. They're meant to be used for quick testing and all changes should be merged back into the main VMDK as soon as possible. If you're looking to send email, a quick search online you'll find many solutions available including using default utilities like (mail or mutt) if you're on a UNIX/Linux system.


=========================================================================
--William
VMware ESX/ESXi scripts and resources at: http://engineering.ucsb.edu/~duonglt/vmware/

Feb 28, 2009 12:38 PM ibhp  says: in response to: lamw

Thank you for your response. But somethimes the snapshots created by ghettovcb are not removed correctly and so the next backups are not performed anymore.

Is there a way to remove every snapshot automatically (in the script) before the new backups are performed? That would ensure that the backups run even if there are "old" snapshots.

Feb 28, 2009 1:33 PM lamw  says: in response to: ibhp

Interesting, I haven't heard of any issues of snapshots not being removed. Are there any previous snapshots on the VM when running the backup? If so, that may cause some issues.

Currently with the method of using the vimsh there is not an option to remove all snapshots, you also don't want to automate that process of committing older snapshots without checking the state of those snapshots. Committing snapshots should be a manual process to ensure you don't revert or make any changes to the system that the Application/System owner is not aware of. Again, the script enforces the rule that if it finds any prior snapshots it will not continue. You should not be using snapshots for long durations and should commit the changes as soon as possible especially if you're looking to backup the VM(s).


=========================================================================
--William
VMware ESX/ESXi scripts and resources at: http://engineering.ucsb.edu/~duonglt/vmware/

Feb 28, 2009 1:45 PM ibhp  says: in response to: lamw

I have a directory with my virtual disc and a snapshot created from ghettoVCB (not removed after the backup). When I try to:
${VMWARE_CMD} vmsvc/snapshot.remove ${VM_ID}

then I get a segmentation fault. So that seems to be the problem the snapshots are not correctly removed after the script terminates.

Any idea what could be the reason?

Mar 2, 2009 1:42 AM shechtl  says: in response to: ibhp

Hello lamw,

since your script is running now for a few weeks, I got two days ago this error:

on esxi1:

Clone: 100% done.
Powering back on ps1
bash: /vmfs/volumes/iscsi1backupdsa1/backups//ps1/ps1-2009-02-26: bad number
/vmfs/volumes/iso/backup/ghettoVCB-enhance-iscsi1.sh: /vmfs/volumes/iso/backup/ghettoVCB-enhance-iscsi1.sh: 323: Syntax error: /vmfs/volumes/iscsi1backupdsa1/backups//ps1/ps1-2009-02-26+1

on esxi2:

Clone: 100% done.
Powering back on exchange3
bash: /vmfs/volumes/iscsi1backupdsa1/backups//exchange3/exchange3-2009-02-26: bad number
/vmfs/volumes/iso/backup/ghettoVCB-enhance-iscsi1.sh: /vmfs/volumes/iso/backup/ghettoVCB-enhance-iscsi1.sh: 323: Syntax error: /vmfs/volumes/iscsi1backupdsa1/backups//exchange3/exchange3-2009-02-26+1
Plink error: Remote command failed
(Exit code 2)

I can see that your script was not removing the backup files: Backup rotation is set to: 2 and there are 3 directories now in the folder.

It looks like that only the last vm of my machinelist had this problem. I did not change anything in your script an machinelist since weeks.

Hope you have a answer.

Mar 2, 2009 3:50 AM hremul  says: in response to: shechtl

Sorry, but where i can't find ghettoVCB-enhance.sh script?

Mar 2, 2009 8:34 AM lamw  says: in response to: shechtl

Please download the current version of the script and manually try again and see if you run into the same issues, if not, then it could be some issues when remoting in with plink.


=========================================================================
--William
VMware ESX/ESXi scripts and resources at: http://engineering.ucsb.edu/~duonglt/vmware/

Mar 2, 2009 8:35 AM lamw  says: in response to: hremul

The current download is the most up to date version of the script, please look over the documentation if you have further questions.

=========================================================================
--William
VMware ESX/ESXi scripts and resources at: http://engineering.ucsb.edu/~duonglt/vmware/

Mar 3, 2009 12:18 PM shechtl  says: in response to: lamw

Hello lamw

I updated to the latest version, but I got the same problem.
See:

                                    1. Starting backup for dc2 ... #####################
Failed to clone disk : The file already exists (39).
Destination disk format: VMFS thick
Cloning disk '/vmfs/volumes/iscsi2dsa4/dc2/dc2.vmdk'...
Powering back on dc2
bash: /vmfs/volumes/iscsi1backupdsa1/backups/dc2/dc2-2009-03-02: bad number
/vmfs/volumes/iso/backup/ghettoVCB-iscsi1.sh: /vmfs/volumes/iso/backup/ghettoVCB-iscsi1.sh: 337: Syntax error: /vmfs/volumes/iscsi1backupdsa1/backups/dc2/dc2-2009-03-02+1

I am sure that i deleted all backuped subfolders which I dont needed so there should be not more then two subfolders be in. Rotation is set to 2

I started your script direct per putty and only the machine dc2 and I got the complete same error.
In the backup folder there are only to subfolders dc2-2009-02-28--2 and dc2-2009-03-01--1

Any idea what this problem could be ?

Mar 3, 2009 12:35 PM lamw  says: in response to: shechtl

I think you might have had some bad backups that were not rotated out properly which is causing some issues. Can you please move any previous backups whether they're good/bad out of that directory or purge them completely. Try a fresh backup


=========================================================================
--William
VMware ESX/ESXi scripts and resources at: http://engineering.ucsb.edu/~duonglt/vmware/

Mar 4, 2009 12:01 AM shechtl  says: in response to: lamw

I deleted the complete folder with the backups and tried 3 backups only with this problematic machnine and they were running. So I delted also the other folders with
problematic machine. I have to wait now for a few days to see if everything is working again.

Mar 4, 2009 9:27 AM Yulman  says: in response to: jsmithsd

Just like you I wanted to log to a file.

This is what I came up with

http://communities.vmware.com/docs/DOC-9653

I did this at my own risk...I'm not a shell programmer so this should be tested before used in a production environment.

Hope this helps

Mar 4, 2009 9:34 AM lamw  says: in response to: Yulman

Thanks for the post, the logging feature will be available in an upcoming release. Stay tuned


=========================================================================
--William
VMware ESX/ESXi scripts and resources at: http://engineering.ucsb.edu/~duonglt/vmware/

Mar 5, 2009 8:21 PM hremul  says: in response to: lamw

First of all, i wish to thank for great script.

I have one suggestion. It is possible to add functionality to backup power off machines which have snapshots?
In some kind applications (like development) night downtime does not matter. But snapshots does. Thank you.

Mar 5, 2009 8:25 PM lamw  says: in response to: hremul

Hi hremul,

As mentioned in the previous posts, snapshots should not be kept for long durations. They can grow and left uncheck could cause issues on your datastores being filled up. If you're looking to backup VM(s) that have snapshots, best practice is to commit those changes and merge the delta files back to the VM then backup the virtual machine. There some complexity with the way the script handles removing snapshots if you're going to try to backup a VM with snapshots and partially it's a limitation with the vimsh and keeping track of the snapshots. It's best that you don't backup VM(s) with snapshots, but if you do, you'll need to make modifications to the script and that is an unsupported feature.


=========================================================================
--William Lam
VMwae vExpert 2009
VMware ESX/ESXi scripts and resources at: http://engineering.ucsb.edu/~duonglt/vmware/

Mar 10, 2009 2:42 AM jbrandligt  says:

Hi lamw,

Thanks for a great script! I have one question though. Is it possible to change the script in such way that it can copy the backups to a datastore on a remote esxi host using something like scp?

Mar 10, 2009 10:07 PM lamw  says: in response to: jbrandligt

You could append some scp commands and using paired SSH keys to remotely copy the backups to external storage or another host. There have been some threads regarding the implementation of scp on the unsupported console and the performance/speed in copying large files. You might want to take a look at a free tool by Veeam fastscp, I have not used it personally but I've heard many good things about their product. That might be a better approach if you need to copy files in/out of ESXi.

You also have the option to leverage vifs.pl from the VI Perl Toolkit default utilities, but again these are customizations that'll need to be added to your custom version of the script.


=========================================================================
William Lam
VMware vExpert 2009
VMware ESX/ESXi scripts and resources at: http://engineering.ucsb.edu/~duonglt/vmware/

http://engineering.ucsb.edu/~duonglt/vmware/vexpert_silver_icon.jpg

Mar 11, 2009 4:24 AM jbrandligt  says: in response to: lamw

Thanks for your reply. Fastscp is a nice tool, but it is only a file copy-tool. It does not handle running vm's because it does not take snapshots.

There is also veeam-backup 3.0 now with support for esxi. The draw back of this tool is that it does not backup to a target esxi host, only local storage or san or iscsi storage. Veeam-backup makes use of the console of esx to make incremental backups of vm's and esxi of course does not have an esx-like console, so that is why it cannot use esxi as a target for its backups.

And then there is also an other tool, vm-explorer, which does do backups to other esxi hosts, but this tool does not run as a service. It is intended to be run as a user only.... For me that is not an option for a backup tool. I could run the tool with a service wrapper, to run it as a service, but i cannot test this scenario with the trial license because there is this annoying popup window, you have to click away every time you start the tool.

So it looks like i have to customize your script. Wish me luck ;)

Jeroen

Mar 25, 2009 8:36 AM dotcom  says: in response to: lamw

Compressing the backup is more than a good suggestion... here is why...

The VM filesystem is stored as a sparse file. When you copy that data to the network storage, you are un-sparsing it, unless you encapsulate the data into a compressed file, using compression that knows how to properly restore the sparse data.

So my 20g vm, that is really only using 270m of disk space in it's sparsely allocated file, will grow instantly to 20g using your backup system. If you restore it, or use the backup to clone in, you are no longer sparse... Instead you suddenly have a big file full of zeros. And those zeros take up network bandwidth.

The backup is not accurate if it fails to preserve the state of the stored data, and this includes the sparseness.

Sparse files are strange beasts, and programs like cp do not handle them correctly most of the time, so the moment you "forget" to handle the file correctly, it inflates to a non-sparse state. In a sparce file, an unallocated block of 0's is not stored on disk, rather the index points to null, which represents an unallocated data block assumed to be zeros. If you cat the file, the filesystem produces those 0's, but they don't really exist on disk. If you copy the file without an awareness of the sparse allocation, you end up with a huge file full of unneeded zeros... and this takes up network bandwidth as well as disk io. When you unsparse a file, there is a heavy performance cost and storage cost.

While some might say "use a compressed filesystem on the NFS server"... That does not deal with the whole issue. On a compressed filesystem you are still storing unneeded 0's, and when you move the file to a non-compressed filesystem, all that space is consumed, along with the disk io needed to write it all out. So you go from a vm that takes twenty seconds to move from one server to another using vmware's tools, to one that takes 3 hours to move. And if you depend on your network bandwidth to do work, now you have a whole lot less of it... not to mention the time lost. I in a day can move hundreds of vm's that only take 20 seconds to transit across the network... but that number falls to 8 when they take 3 hours each.

If you have 3 vm's to back up, it's a minor issue, especially if you do the backup when you sleep. But if you store the backup with the intent to clone it during work hours, it's is a big hit on productivity.

The backups need to preserve the sparseness, a sparse aware compression tool needs to be used. There are not a lot of compression tools on your ESXi system, there is one. So the suggestion that there are many options is wrong. And the suggestion that this is not an issue is misguided. And using a compressed filesystem to store the data on is also not addressing the main issue.

If you don't know what a sparse file is, you can read about it at:
http://administratosphere.wordpress.com/2008/05/23/sparse-files-what-why-and-how/

This does need to be addressed for this tool to be useful the the larger audience.

It is a great tool by the way, don't let my sparse rant take away from the fact that this tool is wonderful. My point is that compression in the utility is not as optional as you would like to think. Compression and correct sparse handling are critical.

Apr 1, 2009 10:47 PM mylesw  says: in response to: lamw

Hi there and thank you for a wonderful script and contribution to the community!

I have two ESXi host machines that are colocated at a data center. There is a 3rd Linux backup computer with NFS on it, and I want to be able to backup the VMs running on these hosts on a regular basis. I have a 'public' WAN network (the VLAN provided by the data center that is public to the world) and a private LAN that is not private. All machines have dual NICs - one for the public WAN and one for the private LAN.

I would like to have it so that my NFS server runs the backups of all VMs using the LAN for this. I'm also sensitive to security concerns, particularly brute force attacks on SSH which are showing up routinely. As a result, all public servers on the WAN have their SSH ports changed from default ports to non-default ports, and Fail2Ban scripts installed to detect brute force login attacks on these boxes. This has been very successful so far.

I cannot install a firewall in this environment because the cost of doing so is prohibitive since I have about 20 WAN IP addresses on the VLAN. I may do this in the future when I work out how to set this up with pfSense but for now, its prohibitive as an option. Consequently every server that is WAN facing has their own local IPTables firewalls in place.

I understand from reading the posts and documentation on the ghettoVCB scripts that it appears to run from within a ESXi host which is made available through a SSH port enable on the box. As much as I would like to be able to do this, its not possible due to security concerns. I can't have default port 22 open like this. So is there a way that I can install it on my NFS server for the LAN only, and not have any of the ESXi boxes set with SSH at all?

Or if not, is there anyway to apply some 'best practice' security approach to enabling SSH on the ESXi hosts that allow for the use of non-default ports, and some form of script that could be run to thwart brute force attacks on the boxes?

Thanks again for your wonderful work, and I look forward to any comments or suggestions.

Myles

Apr 3, 2009 3:43 PM lamw  says: in response to: mylesw

Hi Myles,

Thanks for the comments.

I wanted to get a few things cleared up and this may help you decide on how you go forward with implementing a backup solution. ghettoVCB can run on both ESX/ESXi as you know and specifically on ESXi, you'll need to enable the unsupported SSH console. This is not required for the backup to work but you'll need a way to execute the script from a remote system and currently SSH is the only method to communicate with the host to run the script.

You should also note that the unsupported SSH Console is unsupported by VMware, if you have a support contract this could potentially void the agreement. Best practice is not to enable the console, but there are certain things that is limited with ESXi if you were to just rely on RCLI or VIMA to manage and sometimes an admin may need to login to the console. It's best that you secure your environment and ensure that your hosts are not directly accessible to the public WAN network and only hosts within your environment can communicate to your ESXi host(s).

Hopefully this clears up a few things and good luck on your implementation.


=========================================================================
William Lam
VMware vExpert 2009
VMware ESX/ESXi scripts and resources at: http://engineering.ucsb.edu/~duonglt/vmware/
http://twitter.com/lamw

http://engineering.ucsb.edu/~duonglt/vmware/vexpert_silver_icon.jpg

Apr 25, 2009 3:51 PM GruberK  says:

Hi,
I am using ESXi 3.5 Update4 with the ghettoVCB.sh script downloaded from this website.
I trying to backup 3 machines: Ubuntu 8.10, Windows 2003 STD, Windows 2008 STD, on all 3 machines are the latest VMWare tools installed to a NFS share which is mounted.

I have tried to backup all 3 machines with putty connected to the ESXi server.
su root ./ghettoVCB.sh virtual_machine_backup_list

The problem now is that I would like to backup the machines while they are running (Online)
POWER_VM_DOWN_BEFORE_BACKUP=0
ENABLE_HARD_POWER_OFF=0
2 of the 3 machines are shut down immediately (not gracefully, its like power off) when I start the backup script.

Can anyone tell me where the problem can be?
Thanks in advance for any suggestions!

Apr 25, 2009 3:56 PM lamw  says: in response to: GruberK

Your stated configurations and your expectations slightly contradict each other? Are you trying to perform the backup while the VM(s) are running? If so, the VM(s) should not be shutting down, unless there is some other issue with the VM(s) or host. If you're trying to backup the VM(s) and have them power down prior to backup, then you'll need to enable the appropriate flags. If you enable both flags to be 1 and you do not have VMware Tools installed on the guest, then the host will hard power off the VM after a certain duration.

Please clearly describe your scenario and setup.


=========================================================================
William Lam
VMware vExpert 2009
VMware ESX/ESXi scripts and resources at: http://engineering.ucsb.edu/~duonglt/vmware/
vGhetto Script Repository
http://twitter.com/lamw

http://engineering.ucsb.edu/~duonglt/vmware/vexpert_silver_icon.jpg

If you find this information useful, please award points for "correct" or "helpful".

Apr 25, 2009 5:38 PM GruberK  says: in response to: lamw

Thanks for your quick answer, the 2 machines where the problem with the hard shutdown occurs have the current VMWare Tools installed, the two machines are also Powered On and running before starting the backu script.

Apr 28, 2009 8:45 PM tbsky  says:

hi:
thanks for your great script!

i use ghettoVCB.sh under ESX 3.5i u4 with live snapshot(no shutdown before snapshot).
i wonder if it is a quiesced snapshot? i can not find information about the command "svmvc/snapshot.create".
i hope MS-SQL server with vmwaretools would be happy with the snapshot.
can you give me some advice?
thanks for your help!!

Regards,
sky_li

Apr 30, 2009 8:49 AM gbras  says:

Hi,
I'm confused about the versions of this script. The version I downloaded today from the link in the "attachments" section of this page (or from "http://engineering.ucsb.edu/~duonglt/vmware/") is 11/01/2008:
#!/bin/bash

  1. Author: william2003atgmaildotcom
  2. Date: 11/01/2008

the "CHANGELOG" section lists a version of the script (11/22/2008) that detects existing snapshots and RDMs: where can download it?

Thanks
Guido

Apr 30, 2009 10:21 PM tbsky  says: in response to: gbras

hi:
the author didn't modify the change log. but they are the latest version and can detect existing snapshot. i have tried that.

the attachment and the script at author's website is different files. but after filtering all the white spaces, they are the same.

May 1, 2009 7:20 AM lamw  says: in response to: tbsky

Hi sky_li,

If you're running hight transactional databases, it's generally not recommended to use snapshots for backups due to transactions occurring within a DB. There is something called VSS which is available within VMware Tools that makes the guest aware that a snapshot is going to occur and it forces all data to be flushed from memory to disk, I've not used this personally and I believe it may only be available when VMware VCB is called, Duncan has an article at http://www.yellow-bricks.com/2008/07/26/vss-snapshots/

I would probably stay away from taking VMware snapshots and rely on database snapshots.

In terms of vimsh wrapper for creating snapshots, you won't find much information on any of the commands as these are not officially supported by VMware even though it provides a slew of advanced functionality.

You can always run the following for some basic info:

[root@himalaya scripts]# vmware-vim-cmd vmsvc/snapshot.create
Insufficient arguments.
Usage: snapshot.create vmid [snapshotName] [snapshotDescription] [includeMemory]

Creates a snapshot for the vm.



Where the includeMemory accepts a boolean


=========================================================================
William Lam
VMware vExpert 2009
VMware ESX/ESXi scripts and resources at: http://engineering.ucsb.edu/~duonglt/vmware/
vGhetto Script Repository
http://twitter.com/lamw

http://engineering.ucsb.edu/~duonglt/vmware/vexpert_silver_icon.jpg

If you find this information useful, please award points for "correct" or "helpful".

May 1, 2009 7:45 AM lamw  says: in response to: gbras

Hi Guido,

As tbsky stated, the latest script is currently available for download on this page. The date within the script may be a little miss-leading, it's actually the date that I started develoment of the script. The chanelog on the other hand is accurate in terms of the changes that have taken place since the release of the script and again the latest download will always be available on the VMTN document download page.

I'll ensure that both the script on the VMTN and on our website is the same, sorry for any confusion


=========================================================================
William Lam
VMware vExpert 2009
VMware ESX/ESXi scripts and resources at: http://engineering.ucsb.edu/~duonglt/vmware/
vGhetto Script Repository
http://twitter.com/lamw

http://engineering.ucsb.edu/~duonglt/vmware/vexpert_silver_icon.jpg

If you find this information useful, please award points for "correct" or "helpful".

May 1, 2009 7:50 AM lamw  says: in response to: gbras

Hi Guido,

As tbsky stated, the latest script is currently available for download on this page. The date within the script may be a little miss-leading, it's actually the date that I started develoment of the script. The chanelog on the other hand is accurate in terms of the changes that have taken place since the release of the script and again the latest download will always be available on the VMTN document download page.

I'll ensure that both the script on the VMTN and on our website is the same, sorry for any confusion


=========================================================================
William Lam
VMware vExpert 2009
VMware ESX/ESXi scripts and resources at: http://engineering.ucsb.edu/~duonglt/vmware/
vGhetto Script Repository
http://twitter.com/lamw

http://engineering.ucsb.edu/~duonglt/vmware/vexpert_silver_icon.jpg

If you find this information useful, please award points for "correct" or "helpful".

May 2, 2009 5:20 AM ibhp  says:

hello lamw

I ve got a server to backup which has only space for one backup at a time. Now your
script does first the backup and then rotates/deletes the old backups. This means, in
some time during the backup there are to backups on the disc...

Would it be possible to just move the "rotate"-function to the beginning of your script
and do the backup after rotating/deleting the old backups? Or can you see any problems?

Thank you,

ibhp

May 2, 2009 6:24 AM tbsky  says: in response to: lamw

hi lamw:

i am just curios. "quiesced windows snapshot" seems become complete at 3.5 update 2 with sync+vss driver. but the link you give me tell us stay away from exchange,ad,sql. then i don't know what this "quiesce" function for.

also there seems some version of snapshot command line have "quiesce" option, like the link mentioned:
http://www.esxguide.com/esx/index.php?option=com_mambowiki&page=createsnapshot&page_heading=&show_pageheading=&allowanonymoususers=

but it seems the new command line version didn't have the "quiesce" parameter, only "memory" option is left. i wonder if the snapshot is quiesced automatically now, or the option is because "quiesced" may bring problems.

someone said that if you do a "quiesced snapshot", ms-sql log will tell you.i haven't any real ms-sql server running in the vm now. maybe i can check that in the future.

May 2, 2009 4:29 PM lamw  says: in response to: ibhp

The reason I put the rotate function after the backup is to ensure we have a good backup prior to rotation, especially in the case of just rotating once. I would say please ensure that you have sufficient space for all backups


=========================================================================
William Lam
VMware vExpert 2009
VMware ESX/ESXi scripts and resources at: http://engineering.ucsb.edu/~duonglt/vmware/
vGhetto Script Repository
http://twitter.com/lamw

http://engineering.ucsb.edu/~duonglt/vmware/vexpert_silver_icon.jpg

If you find this information useful, please award points for "correct" or "helpful".

May 2, 2009 4:34 PM lamw  says: in response to: tbsky

Yea I'm not too familiar with the expected behavior of VSS other than asking the VM to flush its data from memory to disk. I just know from reading on the VMTN forums from other experts it's not recommended to using VMware snapshots on VM(s) like databases and if you do, you'll need to look into VSS which is only available per VMware VCB (I think). You may need to run a few tests to see the behavior and I would actually recommend posting a question regarding this on the main VMTN forum and I'm sure an expert familiar with the process can provide further insight.


=========================================================================
William Lam
VMware vExpert 2009
VMware ESX/ESXi scripts and resources at: http://engineering.ucsb.edu/~duonglt/vmware/
vGhetto Script Repository
http://twitter.com/lamw

http://engineering.ucsb.edu/~duonglt/vmware/vexpert_silver_icon.jpg

If you find this information useful, please award points for "correct" or "helpful".

May 11, 2009 1:51 AM pietcur  says:

Hi lamw,

First of all, thx for this great script! I have a question though and hope that the following might also be possible with your script: can you specify a vmdk within a vm that you want to backup? I would like to backup only one vmdk of a vm that is on the same datastore. So basically I want to exclude all the other vmdk's that are attached to this vm and stored on a different datastore. Is this possible?

Thank you,

Pietcur

May 11, 2009 8:51 AM lamw  says: in response to: pietcur

Specific VMDK disk(s) backups are not supported, only full backups. You're more than welcome to modify the script to fit your needs.


=========================================================================
William Lam
VMware vExpert 2009
VMware ESX/ESXi scripts and resources at: http://engineering.ucsb.edu/~duonglt/vmware/
vGhetto Script Repository
http://twitter.com/lamw

http://engineering.ucsb.edu/~duonglt/vmware/vexpert_silver_icon.jpg

If you find this information useful, please award points for "correct" or "helpful".

May 12, 2009 6:34 PM tbsky  says: in response to: lamw

hi:
i am considering put all the windows pagefile and linux swap file to a separate vmdk file. so i can skip backing up them in the future. would you consider implement this feature? or would you consider receive patch?

thanks for your help!!

May 13, 2009 2:22 PM mhanby  says:

Thanks for the script.

Shouldn't the script bypass the creation of a snapshot if the guest is in a powered down state? Seems like extra work to take a snapshot and then merge the snapshot back into the base when the machine is already static.

Also, I got the following error while backing up a guest that was powered off:

/vmfs/volumes/datastore1/scripts # ./ghettoVCB.sh single_vm

################ Taking backup snapshot for Windows XP 32bit Base ... ################
DiskLib_Check() failed for source disk Insufficient permission to access file (38).
Destination disk format: VMFS thick
Cloning disk '/vmfs/volumes/Development/Windows XP 32bit Base/winxp-32bit-base.vmdk'...
Clone: 16% done.

The clone appears to have worked, however out of all the guests that I backed up, that was the only one that echoed the DiskLib_Check() message.

May 13, 2009 2:31 PM mhanby  says: in response to: mhanby

I notice that the script also doesn't appear to take into consideration whether or not a vmdk is thinly provisioned. (as previously mentioned by ddigioia)

I could alter the command:
vmkfstools -i "${SOURCE_VMDK}" "${DESTIONATION_VMDK}"
to
vmkfstools -i "${SOURCE_VMDK}" -d thin "${DESTIONATION_VMDK}"

But:
a) is there a way for the script to identify if a vmdk is thin, and if so add '-d thin'
b) is there any danger in storing thin clones? maybe why the script doesn't already do the previous

May 13, 2009 3:36 PM lamw  says: in response to: mhanby

You know I think this was something I had missed when dealing with the option of allowing a user to configure the shutdown of a guest, I think I might have forgotten to take into consideration the current state of the VM. You're right, this may be a bug. Majority of the user's I've spoke to that have used the script are running live backups and would not run into the issue, but it's a good point and I'll ensure the next update of the script is fixed.

Thanks for bringing it to my attention


=========================================================================
William Lam
VMware vExpert 2009
VMware ESX/ESXi scripts and resources at: http://engineering.ucsb.edu/~duonglt/vmware/
vGhetto Script Repository
http://twitter.com/lamw

http://engineering.ucsb.edu/~duonglt/vmware/vexpert_silver_icon.jpg

If you find this information useful, please award points for "correct" or "helpful".

May 13, 2009 3:45 PM lamw  says: in response to: mhanby

At the time, we were not looking at using thin provision backups, it does not hurt and provides optimized storage consumption (hence probably why vSphere officially supports it now)

Your modifications will work and thinly provision disks upon backup, also a note when you're backing up to NFS datastore, it'll automatically thin provision the disk for you AFAIK

a) You can look at the descriptor file and you'll find a line that says: ddb.thinProvisioned = "1"
b) In terms of backup, there is are no dangers, better storage utilization. In terms of running thin provision VM(s), you just need to be aware of your storage utilization and with ESX 3.5, there are not sufficient monitoring on the datastores to notify you as VM(s) start to fill up, with vSphere there are better notifications/reports.

In an upcoming release of the script all 3 backup types will be supported zeroedthick, thin and 2gb sparse

Thanks for the comments


=========================================================================
William Lam
VMware vExpert 2009
VMware ESX/ESXi scripts and resources at: http://engineering.ucsb.edu/~duonglt/vmware/
vGhetto Script Repository
http://twitter.com/lamw

http://engineering.ucsb.edu/~duonglt/vmware/vexpert_silver_icon.jpg

If you find this information useful, please award points for "correct" or "helpful".

May 14, 2009 6:14 PM tbsky  says: in response to: lamw

hi:
please consider add "monolithic sparse" also. with that you get a small,single vmdk file.
backup tools like rdiff-backup can compare two versions for a file and only backup the difference.
with "monolithic sparse" single vmdk file, backup tools can easily backup many versions of vms with little harddisk space.

May 18, 2009 12:16 PM ibhp  says:

Hello community

I have a win2k3 server running on a ESXi-host with 2 virtual harddrives attached. Now one time the backup runs correct (backing up both virtual harddrives) and the other time it backs-up only the first virtual harddrive (every second time only one virtual hdd is backed-up). This looks like this:

Failure:
Destination disk format: VMFS thick
Cloning disk '/vmfs/volumes/First_750GB_HD/Win2K3 TS/Win2K3 TS_1.vmdk'...
Destination disk format: VMFS thick
Cloning disk '/vmfs/volumes/First_750GB_HD/Win2K3 TS/Win2K3 TS.vmdk'...
Clone: 100% done.
*******************************

Success:
Destination disk format: VMFS thick
Cloning disk '/vmfs/volumes/First_750GB_HD/Win2K3 TS/Win2K3 TS_1.vmdk'...
Clone: 100% done.
Destination disk format: VMFS thick
Cloning disk '/vmfs/volumes/First_750GB_HD/Win2K3 TS/Win2K3 TS.vmdk'...
Clone: 100% done.
*******************************

Any idea what this could be caused from?
Thanks for any help!

May 21, 2009 7:25 AM lamw  says:

Hi everyone,

We're proud to announce the release of ghettoVCBg2 the next gen of ghettoVCB. You can find more information located at: http://communities.vmware.com/docs/DOC-9843

Hope you guys like it!

=========================================================================
William Lam
VMware vExpert 2009
VMware ESX/ESXi scripts and resources at: http://engineering.ucsb.edu/~duonglt/vmware/
vGhetto Script Repository
http://twitter.com/lamw

http://engineering.ucsb.edu/~duonglt/vmware/vexpert_silver_icon.jpg

If you find this information useful, please award points for "correct" or "helpful".

May 26, 2009 5:48 AM freez267  says: in response to: lamw

I've just upgraded my ESXi from 3.5 to 4.0 and the old ghetto script doesn't work.
Infact if I connect with putty and launch the script:

./ghettoVCB.sh weekly
I receive
-ash: ./ghettoVCB.sh: not found

I can't use the new script ghetto2 because I have the free version of ESXi
Can someone help me?

May 26, 2009 10:44 AM lamw  says: in response to: freez267

ghettoVCB has not been tested/certified for ESX(i) 4.0, we've been pretty busy and just have not had the time to test. This holds true for variety of other scripts and some will fail due to changes in vSphere release. In general, I think the script should work and it does not utilize anything that is not already available in ESXi 4.0, make sure the permissions on your script is set to executable. I'll try to do a quick run tonight and post back.


=========================================================================
William Lam
VMware vExpert 2009
VMware ESX/ESXi scripts and resources at: http://engineering.ucsb.edu/~duonglt/vmware/
vGhetto Script Repository
VMware Code Central - Scripts/Sample code for Developers and Administrators
http://twitter.com/lamw

http://engineering.ucsb.edu/~duonglt/vmware/vexpert_silver_icon.jpg

If you find this information useful, please award points for "correct" or "helpful".

May 26, 2009 10:56 AM mhanby  says: in response to: freez267

The error is telling you that ghettoVCB.sh doesn't exist in your current directory.

Was your ghettoVCB.sh script on a datastore or on one of the OS file systems? If it wasn't on a datastore, the version 4 upgrade probably wiped it.

Copy it back to a datastore and execute it from there and see if it will work with v4.

May 26, 2009 11:15 AM lamw  says: in response to: freez267

Okay, just got ESXi 4.0 installed as a VM on our ESX 4 (physical) and I see the problem, looks like there was some change to their shell environment and if you didn't know the busybox console runs on ash ... kinda like bash. It looks like it has a problem when you try to do the following:

./ghettoVCB.sh

-ash: ./ghettoVCB.sh: not found


clearly, I can see the script is in the current directory.

The workaround is the following:

1) Edit the script and change the top line to:

#!/bin/ash


OR

2) execute with the ash interpreter:

ash ./ghettoVCB.sh


Give this a go and it should work, in terms of the actual backup ... I have not verified, so let me know if this works. In the coming weeks, will be verifying some of older scripts and release updates for ESX(i) 4.0 if necessary.


=========================================================================
William Lam
VMware vExpert 2009
VMware ESX/ESXi scripts and resources at: http://engineering.ucsb.edu/~duonglt/vmware/
vGhetto Script Repository
VMware Code Central - Scripts/Sample code for Developers and Administrators
http://twitter.com/lamw

http://engineering.ucsb.edu/~duonglt/vmware/vexpert_silver_icon.jpg

If you find this information useful, please award points for "correct" or "helpful".

May 26, 2009 11:56 PM freez267  says: in response to: lamw

Ok now I can launch the script but I have these errors:

/vmfs/volumes/48c6a14e-37156b78-c647-001a925eca7c # ash ./ghettoVCB.sh weekly
Error: failed to extract VM_ID for 2008Termi!
Error: failed to extract VM_ID for al
Assiste!
Error: failed to extract VM_ID for zaVista2!

Start time: Wed May 27 06:51:45 UTC 2009
End time: Wed May 27 06:51:48 UTC 2009
Duration : 3 Seconds

Completed backing up specified Virtual Machines!

May 27, 2009 12:06 AM lamw  says: in response to: freez267

I ran a quick test earlier and realized some of the code may need to be updated. I have not had a chance to dig further on what needs to be modified.


=========================================================================
William Lam
VMware vExpert 2009
VMware ESX/ESXi scripts and resources at: http://engineering.ucsb.edu/~duonglt/vmware/
vGhetto Script Repository
VMware Code Central - Scripts/Sample code for Developers and Administrators
http://twitter.com/lamw

http://engineering.ucsb.edu/~duonglt/vmware/vexpert_silver_icon.jpg

If you find this information useful, please award points for "correct" or "helpful".

May 29, 2009 2:24 PM lamw  says: in response to: freez267

Have you had a chance to try out ghettoVCB4i.sh to see if the issues are resolved?


=========================================================================
William Lam
VMware vExpert 2009
VMware ESX/ESXi scripts and resources at: http://engineering.ucsb.edu/~duonglt/vmware/
vGhetto Script Repository
VMware Code Central - Scripts/Sample code for Developers and Administrators
http://twitter.com/lamw

http://engineering.ucsb.edu/~duonglt/vmware/vexpert_silver_icon.jpg

If you find this information useful, please award points for "correct" or "helpful".

May 30, 2009 5:20 AM atrussokc  says:

Thanks for the great script. I finally have an option for backing up my ESXi (free) 4 VMs. I downloaded ghettoVCB4i.sh and attempted to use it but encountered a problem where all of my VMs with the letter 'n' in the name where causing an error. This is similar to the issue described by freez267 above. I don't fully understand why but commenting out the line

IFS=$'\n'

appeared to resolve the issue for me. I am not sure what this line does.

Thanks
Tony Russell

May 30, 2009 1:44 PM lamw  says: in response to: atrussokc

That's odd... it should work, I've verified on our ESXi 4 system.

IFS=$'\n'


Is called the Internal Field Separator:
http://en.wikipedia.org/wiki/Internal_Field_Separator
http://learnlinux.tsf.org.za/courses/build/shell-scripting/ch11s11.html

Basically when I loop through the list of VM(s) after some sanity checks and extracting out the VmId which is used in the vimsh to uniquely identify a VM, I'm also extracting the VM(s) displayName. When user's use white space in their VM names, by default the IFS is looking for a 'space' as a delimiter to denote the next line of data so a workaround is to redefine the IFS to look for a newline that way you can capture the line of text with spaces. In either cases it will not hurt if you don't have spaces in your VM name, but it'll only help if you do.

Again, I'm not exactly sure why you're having an issue since '\n' is representing a newline and not the 'n' char.


=========================================================================
William Lam
VMware vExpert 2009
VMware ESX/ESXi scripts and resources at: http://engineering.ucsb.edu/~duonglt/vmware/
vGhetto Script Repository
VMware Code Central - Scripts/Sample code for Developers and Administrators
http://twitter.com/lamw

http://engineering.ucsb.edu/~duonglt/vmware/vexpert_silver_icon.jpg

If you find this information useful, please award points for "correct" or "helpful".

May 31, 2009 1:28 PM paulkc  says:

I am running this on ESXi 4. I have tried to follow the directions very closely but I keep getting a Permission Denied error every time I try to actually run the script. Any suggestions?

May 31, 2009 1:37 PM lamw  says: in response to: paulkc

Did you change the permissions on the script to have executable permissions?

chmod +x ghettoVCB4i.sh


=========================================================================
William Lam
VMware vExpert 2009
VMware ESX/ESXi scripts and resources at: http://engineering.ucsb.edu/~duonglt/vmware/
vGhetto Script Repository
VMware Code Central - Scripts/Sample code for Developers and Administrators
http://twitter.com/lamw

http://engineering.ucsb.edu/~duonglt/vmware/vexpert_silver_icon.jpg

If you find this information useful, please award points for "correct" or "helpful".

Jun 1, 2009 12:23 AM freez267  says: in response to: paulkc

I continue to have this error

Error: failed to extract VM_ID

On the VMs that have 'n' in the name. For example is impossible to backup the VM '2008Terminal' and 'AssistenzaXp' but no problem with 'W7RC'

Jun 1, 2009 9:10 AM lamw  says: in response to: freez267

I just created some dummy VM(s) using the names you've specified and it looks like the environment is not treating $'\n' as a newline char but the literal char 'n'. This cause the loop to break off once it sees the first char 'n' and a VM with the name: "2008Terminal" will be chompped into "2008Termin" and "al" which neither of which will match, hence it fails. If you specify a list of VM(s) w/o 'n' then the backup is successful. This is pretty odd and I'll have to look into it some more.

You said if you remove the IFS=$'\n', everything works?


=========================================================================
William Lam
VMware vExpert 2009
VMware ESX/ESXi scripts and resources at: http://engineering.ucsb.edu/~duonglt/vmware/
vGhetto Script Repository
VMware Code Central - Scripts/Sample code for Developers and Administrators
http://twitter.com/lamw

http://engineering.ucsb.edu/~duonglt/vmware/vexpert_silver_icon.jpg

If you find this information useful, please award points for "correct" or "helpful".

Jun 1, 2009 9:16 AM paulkc  says: in response to: lamw

That is exactly what happened in my case. If I commented out that line, it worked fine.

Jun 1, 2009 9:53 AM lamw  says: in response to: paulkc

Looks like the newline needs to be defined in the following manner:

IFS='
'


notice where the single tick starts and ends which is on the following line. If you make this modification, it will backup all 3 VM(s) properly .... I'll push an update out tonight, it looks like the new Busybox ash environment may have been slightly changed and does not recognize $'\n' as a newline. The real reason for redefining the IFS is for VM(s) names that may contain spaces, which I consider a bad practice...but if you know your naming convention will never have spaces,tabs,etc...then you should be fine.


=========================================================================
William Lam
VMware vExpert 2009
VMware ESX/ESXi scripts and resources at: http://engineering.ucsb.edu/~duonglt/vmware/
vGhetto Script Repository
VMware Code Central - Scripts/Sample code for Developers and Administrators
http://twitter.com/lamw

http://engineering.ucsb.edu/~duonglt/vmware/vexpert_silver_icon.jpg

If you find this information useful, please award points for "correct" or "helpful".

Jun 7, 2009 9:44 AM carlotto  says:

Hi,
Thanks for a fantastic script. Works well, but I have a problem with the rotation. Anyone with a clue why it won't work? See output from a sh -x command (last bit):

Cloning disk '/vmfs/volumes/HB-ESXi2-scsi1/HB-Label/HB-Label.vmdk'...
Clone: 100% done.
+ IFS=

+ ! 0 -eq 1
+ /bin/vim-cmd vmsvc/snapshot.remove 256
+ echo Removing snapshot from HB-Label ...
+ ls /vmfs/volumes/HB-ESXi2-scsi1/HB-Label
+ grep -q delta
+ sleep 3
+ ls /vmfs/volumes/HB-ESXi2-scsi1/HB-Label
+ grep -q delta
+ sleep 3
+ ls /vmfs/volumes/HB-ESXi2-scsi1/HB-Label
+ grep -q delta
+ checkVMBackupRotation /vmfs/volumes/Thecus/Backup/HB-Label /vmfs/volumes/Thecus/Backup/HB-Label/HB-Label-2009-06-07
+ local BACKUP_DIR_PATH=/vmfs/volumes/Thecus/Backup/HB-Label
+ local BACKUP_VM_NAMING_CONVENTION=/vmfs/volumes/Thecus/Backup/HB-Label/HB-Label-2009-06-07
+ ls -tr /vmfs/volumes/Thecus/Backup/HB-Label
+ LIST_BACKUPS=HB-Label-2009-05-31
HB-Label-2009-06-05
HB-Label-2009-06-07
+ -z 4
+ TMP_DIR=/vmfs/volumes/Thecus/Backup/HB-Label/HB-Label-2009-05-31
+ echo /vmfs/volumes/Thecus/Backup/HB-Label/HB-Label-2009-05-31
+ TMP=/vmfs/volumes/Thecus/Backup/HB-Label/HB-Label-2009-05-31
+ /vmfs/volumes/Thecus/Backup/HB-Label/HB-Label-2009-05-31 = /vmfs/volumes/Thecus/Backup/HB-Label/HB-Label-2009-06-07
+ /vmfs/volumes/Thecus/Backup/HB-Label/HB-Label-2009-05-31 -ge 4
sh: /vmfs/volumes/Thecus/Backup/HB-Label/HB-Label-2009-05-31: bad number
+ echo /vmfs/volumes/Thecus/Backup/HB-Label/HB-Label-2009-05-31
+ BASE=/vmfs/volumes/Thecus/Backup/HB-Label/HB-Label-2009-05-31
/vmfs/volumes/Thecus/Script/HB-Label/ghettoVCB.sh: /vmfs/volumes/Thecus/Script/HB-Label/ghettoVCB.sh: 340: Syntax error: /vmfs/volumes/Thecus/Backup/HB-Label/HB-Label-2009-05-31+1
/vmfs/volumes/46df116f-b561be9a/Log #

I have a few different VM's with different rotation counts (I keep several copies of the script). At least one does not delete when more than the rotation count, I suspect the problem above. Any clue anyone? Why syntax error etc?

Jun 8, 2009 2:20 PM Ritchy059  says: in response to: carlotto

I had a similiar Problem here an a ESX Server 3.5:

At first: Script works well from the Root-Shell...

But, when i Plan the Script via Crontab, the Script starts, make's the Snapshot, Backup the *.vmdk and *.vmx File in a newly created Folder BUT at the End of the Backup the Script don't rename the Folder in correct manner.

Script stopped working. I have to rename the Folders manually in correct Name und Numbers and have to remove the Snapshot manually from ESX Server's VI Client.

The Script only Backups the First VM in the Backup List and after Backup stopped working !

mfg
Ritchy059

Jun 15, 2009 3:02 AM adiXSG  says: in response to: Ritchy059

I've running an ESXi 4.0 Server.
The backup does function well. But there is one problem wich is descibed by other persons above. It is the empty log file. Is there a solution for it or not? At the moment I see through all folders if the backup success. But that can't be the final solution.
Further the current version of ghettoVCB4i.sh runs great if there are no whitespaces in the vm name. Otherwise the rename after backup fails and because of that the backup rotation fails too. So I've removed all whitespaces in VM names like lamw describes as best practice :)

Thanks lamw for his great work... really really usefull script!!

Jun 18, 2009 5:40 AM voro  says:

For those interested in logging, maybe this helps you:

Run the script like this

sh ghettoVCB.sh vmlist.txt 2>&1 | busybox logger -t ghettoVCB -p 1

and then, push ESXi logs to a remote syslog-ng server.

Jun 27, 2009 6:17 PM relegated  says: in response to: voro

Does this backup method copy the vmdk to the same vmfs volume and to get it offsite you would have to download the backup vmdk file from the datastore?

Jun 28, 2009 10:50 PM mylesw  says: in response to: relegated

You could just mount a NFS volume and select that as the destination for the backup. That's what we do. Works great.

Jun 28, 2009 10:54 PM mylesw  says: in response to: lamw

Quick question... I am running ghettoVCB.sh on ESXi 3.5 and its working great on most of my VMs. However I have a VM that seems to have a snapshot associated with it. I checked in the datastore, and there was a 'delta' file there. I powered down the VM and then deleted the delta file. Powered it back on, and then attempted to run ghettoVCB.sh. It again wouldn't do the backup because it said that there was a snapshot there. I checked it again, and the delta file had returned...

I know this isn't a specific question of ghettoVCB, but how do I get rid of these snapshots so I can do a clean backup of a live VM?

Jun 28, 2009 11:42 PM voro  says: in response to: mylesw

You shouldn't delete deltas by hand. Instead, remove snapshots from VIC.
Deltas track changes to the main vmdk, but information about snapshots are kept in vmsd files.

Jun 29, 2009 12:38 PM relegated  says: in response to: voro

Unfortunately I only have my one ESXi box at home with 1 HDD in it and my desktop so I have no NFS volume to mount :(

So for me, does that mean that this script just backs up the running vmdk and I would need to manually download it?

If that is the case the primary function of the script would be to reduce downtime as otherwise I would need to power down the VM and copy the vmdk correct?

Jun 29, 2009 12:43 PM mylesw  says: in response to: relegated

So for me, does that mean that this script just backs up the running vmdk and I would need to manually download it?

What it basically does is a clone of the VM. The location of the cloned VM is up to you, and set in the script preferences. If you have NFS storage mounted and available to ESXi, then you can elect to use that as the storage location. Otherwise you'd have to store it in a local datastore somewhere. Once its done, then you can download it to offline storage using SCP or some other form of transport that you can support with ESXi.

If that is the case the primary function of the script would be to reduce downtime as otherwise I would need to power down the VM and copy the vmdk correct?

You do not need to power down the VM to do the clone. The script will create a snapshot before it starts its processing and remove it when its completed it. You do have the option to power down the VM in the script, however, but by default it is turned off.

Jun 29, 2009 12:46 PM mylesw  says: in response to: voro

You shouldn't delete deltas by hand. Instead, remove snapshots from VIC.
Deltas track changes to the main vmdk, but information about snapshots are kept in vmsd files.

Thanks. I found a way to remove the snapshots using VM Explorer's free edition (http://www.trilead.com). Works great.

Jul 1, 2009 4:55 AM bjoernbergmann  says:

Great script! Thank you so much... We are backing up 4 ESXi 3.5 servers successfully with about 6VMs each.

There is only one problem to backup a Debian Linux 60GB virtual harddisk. Snapshot seems to be OK. Backing up the disk (one of two) failed after some time with the following error message.

Taking backup snapshot for DEASX401 ... ################
Destination disk format: VMFS thick
Cloning disk '/vmfs/volumes/datastore1/DEASX401/DEASX401_2.vmdk'...
Clone: 79% done.Failed to clone disk : Insufficient permission to access file (38).
Destination disk format: VMFS thick
Cloning disk '/vmfs/volumes/datastore1/DEASX401/DEASX401.vmdk'...
Clone: 100% done.
Removing snapshot from DEASX401 ...
Completed backup for DEASX401! ####################
Start time: Wed Jul 1 06:45:27 UTC 2009
End time: Wed Jul 1 09:21:51 UTC 2009
Duration : 156.40 Minutes
Completed backing up specified Virtual Machines!

Might it be possible that the disk is to big? Is there any timeout?

Here some more information:
./ghettoVCB.sh is running with root account!
backup disk mouted vie NSF from a Win2003R2 server
share is an external 1GB USB disk
nsf shared folder is windows compressed (can save until 75% of original space)

Regards,

Bjoern