VMware
1 ... 12 13 14 15 16 ... 31 Previous Next 453 Replies Last post: Sep 30, 2009 8:55 AM by cincinnerdi   Go to original post

Re: Free ESXi Backup Solution for Windows

195. Nov 26, 2008 1:06 PM in response to: JoSte
Click to view aremmes's profile Enthusiast 22 posts since
Sep 10, 2008
JoSte wrote:
I am by no means an expert in this, but what if there is a VM_NAME == 'test vm' and 'test'. Wouldn't that 'test' also match the 'test vm', then?
~ # grep -E "\<test\>" /tmp/vms_list
"16";"test vm";"[datastore-1]";"New Virtual Machine/New Virtual Machine.vmx"
~ # grep -E "\<test vm\>" /tmp/vms_list
"16";"test vm";"[datastore-1]";"New Virtual Machine/New Virtual Machine.vmx"
~ #

How about this:
grep -E "\"${VM_NAME}\"" /tmp/vms_list | ...

It should work also with spaces in $VM_NAME.
~ # grep -E "\"test vm\"" /tmp/vms_list
"16";"test vm";"[datastore-1]";"New Virtual Machine/New Virtual Machine.vmx"
~ # grep -E "\"test\"" /tmp/vms_list
~ #

You're quite correct. Seeing the file format gives me an idea, though...
#!/bin/bash

# Backup the current IFS
OLD_IFS="${IFS}"
IFS=";"

# Name of VM we're looking for
MY_VMNAME="vm"

# Walk through stdin, stop at first match
while read VM_ID VM_NAME VM_DS VM_VMX; do
        test "${MY_VMNAME}" = "${VM_NAME}" && break
done << __VMS_LIST__
1;vm01;[datastore-1];New Virtual Machine #1/New Virtual Machine #1.vmx
2;vm02;[datastore-1];New Virtual Machine #2/New Virtual Machine #2.vmx
3;vm03;[datastore-1];New Virtual Machine #3/New Virtual Machine #3.vmx
4;vm04;[datastore-1];New Virtual Machine #4/New Virtual Machine #4.vmx
5;random vm name;[datastore-1];random vm name/random vm name.vmx
6;vm;[datastore-1];vm/vm.vmx
7;vm05;[datastore-1];New Virtual Machine #5/New Virtual Machine #5.vmx
8;vm06;[datastore-1];New Virtual Machine #6/New Virtual Machine #6.vmx
__VMS_LIST__

# Print the values of the last record read
echo ${VM_ID} ${VM_NAME} ${VM_DS} ${VM_VMX}

# Restore IFS
IFS="${OLD_IFS}"


This assumes that the lines in the input is unique, but they don't have to be sorted. Running this gives:
 ~ # ./testvmlist.sh
6 vm [datastore-1] vm/vm.vmx
 ~ #


Message was edited by: aremmes -- some code was getting munched

Re: Free ESXi Backup Solution for Windows

196. Nov 26, 2008 1:12 PM in response to: aremmes
Click to view lamw's profile Champion 2,803 posts since
Nov 27, 2007
Hi JoSte and aremmes,

Thanks for your guys feedback and suggestions! Definitely great ideas and I guess we've learned that there's more than one way to accomplish a task. I've been pretty busy this morning but I'll be taking some of that feedback and update the script. I'll need to run further tests to ensure it does not cause any side affects to the current functionality. I'll post it up once it's ready to go

Re: Free ESXi Backup Solution for Windows

197. Nov 26, 2008 3:04 PM in response to: lamw
Click to view JoSte's profile Novice 6 posts since
Nov 20, 2008
I have tried to change the script with aremmes solution:
	#dump out all virtual machines allowing for spaces now
	${VMWARE_CMD} vmsvc/getallvms | sed 's/[[:blank:]]\{3,\}/   /g' | awk -F'   ' '{print $1";"$2";"$3}' |  sed 's/\] /\];/g' | sed '1,1d' > /tmp/vms_list

	IFS=$'\n'
	for MY_VMNAME in `cat ${VM_INPUT} | sed '/^$/d' | sed -e 's/^[[:blank:]]*//;s/[[:blank:]]*$//'`;
        do
		OLD_IFS="${IFS}"
		IFS=";"
		while read VM_ID VM_NAME VM_DS VMX_CONF; do
			test "${MY_VMNAME}" = "${VM_NAME}" && break
		done </tmp/vms_list
		# Restore IFS
		IFS="${OLD_IFS}"

		VMFS_VOLUME=`echo "${VM_DS}" | sed 's/\[//;s/\]//g'`
		
#		VM_ID=`grep "${VM_NAME}" /tmp/vms_list | awk -F ";" '{print $1}' | sed 's/"//g'`

		#ensure default value if one is not selected or variable is null
		if [ -z ${VM_BACKUP_DIR_NAMING_CONVENTION} ]; then
			VM_BACKUP_DIR_NAMING_CONVENTION="$(date +%F)"
		fi

		#esx
		if ; then
			#VMFS_VOLUME=`grep \"${VM_NAME}\" /tmp/vms_list | awk -F ";" '{print $3}' | sed 's/\[//;s/\]//;s/"//g'`
			#VMX_CONF=`grep \"${VM_NAME}\" /tmp/vms_list | awk -F ";" '{print $4}' | sed 's/"//g'`
			VMX_PATH="/vmfs/volumes/${VMFS_VOLUME}/${VMX_CONF}"
			VMX_DIR=`dirname "${VMX_PATH}"`
			#VMX_DIR=`echo ${VMX_DIR%/*}`
		fi
		#esxi
		if ; then
			#VMFS_VOLUME=`grep -E \"${VM_NAME}\" /tmp/vms_list | awk -F ";" '{print $3}' | sed 's/\[//;s/\]//;s/"//g'`
			#VMX_CONF=`grep -E \"${VM_NAME}\" /tmp/vms_list | awk -F ";" '{print $4}'  | sed 's/\[//;s/\]//;s/"//g'`
			VMX_DIR=`dirname "${VMX_CONF}"` 	
			#VMX_DIR=`echo ${VMX_DIR%/*}`
			VMX_DIR=/vmfs/volumes/${VMFS_VOLUME}/${VMX_DIR}
			VMX_PATH=/vmfs/volumes/${VMFS_VOLUME}/${VMX_CONF}
		fi

		#checks to see if we can pull out the VM_ID
		if [ -z ${VM_ID} ]; then
			echo "Error: failed to extract VM_ID for ${MY_VMNAME}!"

It looks promising so far.

Error: failed to extract VM_ID for test!
##########################################
Type: light
Virtual Machine: test vm
VM_ID: 16
VMX_PATH: /vmfs/volumes/datastore-1/New Virtual Machine/New Virtual Machine.vmx
VMX_DIR: /vmfs/volumes/datastore-1/New Virtual Machine
VMX_CONF: New Virtual Machine/New Virtual Machine.vmx
VMFS_VOLUME: datastore-1
##########################################

Error: failed to extract VM_ID for test1!
Error: failed to extract VM_ID for tesat2!
Error: failed to extract VM_ID for test3 und4 5!

Start time: Wed Nov 26 22:08:19 UTC 2008
End   time: Wed Nov 26 22:08:19 UTC 2008
Duration  : 0 Seconds

Completed backing up specified Virtual Machines!


This is the input file:
test
test vm
test1
tesat2
test3 und4 5

It has some garbage in it by purpose...

Once the now unneeded code would be removed, the script should look nicer. Overall I think aremmes had a very nice idea here. It may be worth going that direction. It is less complicated.

Re: Free ESXi Backup Solution for Windows

198. Nov 26, 2008 4:38 PM in response to: JoSte
Click to view lamw's profile Champion 2,803 posts since
Nov 27, 2007
Taking a look at the code and with the suggestions, we can actually remove some unnecessary code, since we rely on the "VMware vimsh" to pull out information. We don't have to break up the code on parsing the Virtual Machine information whether the script is running on an ESX w/SC or ESXi because the information can be pulled the same way.

Here is the code that'll get the modification:

${VMWARE_CMD} vmsvc/getallvms | sed 's/[[:blank:]]\{3,\}/   /g' | awk -F'   ' '{print "\""$1"\";\""$2"\";\""$3"\""}' |  sed 's/\] /\]\";\"/g' | sed '1,1d' > /tmp/vms_list

        IFS=$'\n'
        for VM_NAME in `cat "${VM_INPUT}" | sed '/^$/d' | sed -e 's/^[[:blank:]]*//;s/[[:blank:]]*$//'`;
        do
                VM_ID=`grep -E "\"${VM_NAME}\"" /tmp/vms_list | awk -F ";" '{print $1}' | sed 's/"//g'`

                #ensure default value if one is not selected or variable is null
                if [ -z ${VM_BACKUP_DIR_NAMING_CONVENTION} ]; then
                        VM_BACKUP_DIR_NAMING_CONVENTION="$(date +%F)"
                fi

                VMFS_VOLUME=`grep -E "\"${VM_NAME}\"" /tmp/vms_list | awk -F ";" '{print $3}' | sed 's/\[//;s/\]//;s/"//g'`
                VMX_CONF=`grep -E "\"${VM_NAME}\"" /tmp/vms_list | awk -F ";" '{print $4}' | sed 's/\[//;s/\]//;s/"//g'`
                VMX_PATH="/vmfs/volumes/${VMFS_VOLUME}/${VMX_CONF}"
                VMX_DIR=`dirname "${VMX_PATH}"`


I will not have time to finalize testing but I'll do so after coming back from the holidays.

I'll be out of town and I hope everyone has a safe and happy holiday!

Re: Free ESXi Backup Solution for Windows

199. Nov 27, 2008 8:01 AM in response to: lamw
Click to view kpc's profile Enthusiast 98 posts since
Jul 12, 2006
Sorry, double post...

Re: Free ESXi Backup Solution for Windows

200. Nov 27, 2008 7:58 AM in response to: kpc
Click to view kpc's profile Enthusiast 98 posts since
Jul 12, 2006
Both version 1 and 2 of the script has always worked in that fashion. You would feed in a list of the Virtual Machines you would like backed up and they're queried and backed up. How else would you do it http://communities.vmware.com/images/emoticons/wink.gif

The reason for the display name, which I think most user's have both the display name and the actual name of their directory/etc all match up, is that VMware allows you to have a different display name than the actual directory and possibly some of the other attributes. The queries that are return by using either "vmware-cmd -l" or "vmware-vim-cmd vmsvc/getallvms" show only display name

Hopefully this answered your question. "

Hi lamw
I was asking this as I used vmware-cmd -l on my old script to generate the VM's file location and then went through and just backed up all VM's that didn't have a snapshot - no need to add displaynames to a file. I was just wondering if this can be done for your script or if you choose to do it this way in order to give the user some choice over what they wanted to backup.

Have a great thanksgiving from across the pond....

Re: Free ESXi Backup Solution for Windows

201. Nov 27, 2008 1:28 PM in response to: kpc
Click to view Bolgard's profile Hot Shot 113 posts since
Aug 11, 2007
Hello,

I've scanned through the whole thread, and there's a lot of information and several good scripts in here. However, I can't find any instructions on how to actually set them up and schedule them. Where do the scripts run? At the "unsupported" ESXi console? At the target backup machine? What is required to run them (PHP, perl, etc)? Would be great if someone could provide a summary for relating back to the topic "Free ESXi Backup Solution for Windows" or if that's to much to ask, maybe some short information on how to setup a script!

I'm looking for a backup script that can be scheduled and can do backups of the VMs to a offsite location in some way (ftp, vpn, ssh, scp?).

Thanks in advance!

Re: Free ESXi Backup Solution for Windows

202. Nov 29, 2008 6:43 AM in response to: kpc
Click to view DSTAVERT's profile Virtuoso 2,388 posts since
Nov 30, 2003
Rather than a list of machines to backup use the existance of a file in the VM directory to determine whether to backup or not. I have had a quick look through the code and I don't think it should be tto dificult. When I get back to the office I can start working at it.

I think the code could generate vms_list rather than manually editing a file. Loop through the VMs with this sort of logic to to create the list. Seperating the selection logic from the backup process would clean up the code as wel.

If exist daily


then backup


if exist weekly and today = saturday


then backup.

If exist snapshot then don't backup

If exist raw disk then don't backup.

etc


It would be easy to have a complex backup strategy by extending the selection logic (last_day_month, wednesdays, etc) and add or remove a single file in the VM directory.

Re: Free ESXi Backup Solution for Windows

203. Nov 29, 2008 11:02 AM in response to: DSTAVERT
Click to view tlduong's profile Novice 20 posts since
Apr 6, 2008
Can you please justify how this (placing a file in each VM directory in need of a backup and removing it to prevent backup) would be more convenient than editing a single file that contained all virtual machines in need of a backup?

Granular control over which VMs should be backed up more often than others sounds more useful in your suggestion. This, however, looks like it can be achieved by making cronjobs that run the script (ghettoVCBni.sh) on different sets of backup_vm* files at the desired periods.

Re: Free ESXi Backup Solution for Windows

204. Nov 29, 2008 12:27 PM in response to: tlduong
Click to view blueivy's profile Novice 22 posts since
Aug 9, 2008
This is a sort of 'how long is a piece of string' question, however.

I am using FreeNAS (with NFS) to backup my ESXi box to using the lamw script. It's working great, however throughput is awful. I've checked the NIC graphs in FreeNAS and it fluctuates between 10Mbps and 15Mbps. The CPU graphs are in the low 1-3%.

The network stats for the management NIC in ESXi (using the VI client) are around 1251Kbps.

The ESXi server is a ML350 G5 using a Smary Array 200i with write caching turned on and the array accelerator turned on (50% read / write).It is running 3 x 1TB SATA disks in a RAID 5 configuration.

The FreeNAS box is an old PC using an Adaptec 2410 RAID card with 4 x 500GB SATA drives in a RAID 5 array. Write caching is turned on for the drives (even though there is no BBWC - turned it on purely for testing just now).

The server and the NAS box are both on a Gigabit network with Gigabit cards and connected at 1000Mbps to a Gigabit switch.

I am backing up a test VM (8GB) that is powered down. The ESXo server has very little else going on on (I'm running 5 other VM's - one SBS and 4 linux but they are not doing much at the moment).


My first thought was to replace FreeNAS with say OpenFiler and try that as I have read FreeNAS has problems with NFS.


Any suggestions?

Re: Free ESXi Backup Solution for Windows

205. Nov 29, 2008 3:46 PM in response to: tlduong
Click to view DSTAVERT's profile Virtuoso 2,388 posts since
Nov 30, 2003
Using a single file in a VM folder makes it simple to have a single script / cron job to create a fairly complex strategy. No editing /adding cron jobs as new VMs come on line. Decide your strategy as you create the VM add the appropriate file. Easy to add the file through the VI client. Instead of a file in the VM folder use the presence of a folder in the VM. Then the folder can be created from within the VI client. Need to change the strategy just rename the folder.

I'm not suggesting that this IS the way to do things just that this might mean a simple way to set it up once and have it do more.

Re: Free ESXi Backup Solution for Windows

206. Nov 29, 2008 3:44 PM in response to: blueivy
Click to view kpc's profile Enthusiast 98 posts since
Jul 12, 2006
blueivy
Make sure on your nfs exports you have 'async' switched on. I don't use Windows but did this on my Linux NFS box and speed was back to what it should....

Re: Free ESXi Backup Solution for Windows

207. Nov 29, 2008 4:03 PM in response to: blueivy
Click to view DSTAVERT's profile Virtuoso 2,388 posts since
Nov 30, 2003
After using both Openfiler and Freenas I have come to the conclusion that If all you want is NFS then you are probably better using your preferred distro and hand configuring. Too much extra stuff in both of them. NFS is really simple to set up especially since ESX doesn't handle any advanced features (NFS4) anyway. If you need a web interface just install webmin. If you need access from Windows just add the Unix for Windows download from Microsoft or add the NFS client in w3k R2 or server 2008.

Re: Free ESXi Backup Solution for Windows

208. Nov 29, 2008 4:10 PM in response to: Bolgard
Click to view kpc's profile Enthusiast 98 posts since
Jul 12, 2006
Hi Bolgard.

The name is the thread is slightly misleading as the script has evolved over the past few months to one that runs directly on the ESXi server itself. A new thread should read Free Linux backup script http://communities.vmware.com/images/emoticons/happy.gif If you still want to use Windows then the info is there (somewhere). Personally I'd recommend the LInux/ESXi route as it's far more more contained and versatile. You need to first enable SSH on the ESXi server (plenty of guides around), then SCP iawm's script to it and setup your NFS share. It all depends on what technologies you are using, e.g. Windows, Linux and how good/bad your linux is.

I use only Linux where I work, so have a 1TB NFS share that I use for my VM backups, I mount the NFS through the VI client, run the script - nice and simple... Hope that helped a little.....

Re: Free ESXi Backup Solution for Windows

209. Nov 29, 2008 4:24 PM in response to: DSTAVERT
Click to view blueivy's profile Novice 22 posts since
Aug 9, 2008

I have now setup Openfiler and I do get a huge performance gain on it - I'm at 60Mb ish transfer which is closer to where it should be.

I never actually thought about just installing linux and doing it that way! I think I will give that a go but before I do I may try what kpc suggested and change NFS to async on FreeNAS. I much prefer FreeNAS over Openfiler (which I feel has very poor support to encourage you to pay for it) as it's much easier to use and setup. It also has lower hardware requirements compared to Openfiler. It took me about 45 minutes to setup an NFS share on Openfiler as it just wasn't clear how it was done, at least to me.

Will post back my findings.

VMware Developer

SDKs, APIs, Videos, Learn and much more in the Developer community.

Learn More

Developer Sample Code

Increase your developer productivity with VMware API sample code.

Learn More

VMworld Sessions & Labs

Online access to the latest VMworld Sessions & Labs and online services.

Learn more

Purchase PSO Credits Online

Purchase credits to redeem training and consulting services online.

Buy Now

Community Hardware Software

View reported configurations or report your own.

Learn More

VMware vSphere

Come witness the next giant leap in virtualization.

Register Today

Communities