VMware Communities: Free ESXi Backup Solution for Windows

Reply Re: Free ESXi Backup Solution for Windows Nov 29, 2008 4:50 PM

in response to: blueivy

DSTAVERT 2,180 posts since
Nov 30, 2003

I think if you have any linux experience you won't have any problems with NFS. Have a look at the /etc/exports file from openfiler.If all you need is NFS no need to have anything not necessary.

Reply Re: Free ESXi Backup Solution for Windows Nov 29, 2008 10:24 PM

in response to: kpc

lamw 2,748 posts since
Nov 27, 2007

Hi kpc,

The reason I require some type of input to the script is to allow flexibility to the end users on how and which Virtual Machines to backup. I can't assume that everyone out there will want to backup all their VMs on their ESX or ESXi Server(s). This allows a user to define a set list and then automate via a crontab or even scheduled tasks from a windows system and using plink to remotely connect to an ESX/ESXi host to kick off the backup, as one user has implemented.

Yes, you're right in your script you just make a call to "vmware-cmd -l" to provide the list of all Virtual Machines and then ran a backup, though this solution ONLY works on your standard ESX with the Service Console, what about ESXi?

ESXi does not contain "vmware-cmd", the only way to pull information out is using "vim-cmd" or remotely querying RCLI. I wanted to provide the end users with as much flexibility as possibly including the ability to run a common backup script on both ESX and ESXi.

Now in terms of your specific modification, yes that can be done very easily. If you're looking to backup all Virtual Machines on your ESX 3.5 w/SC then you would just need run the "vmware-cmd -l" and pull out the Virtual Machine name, assuming your directory structure and displayName is the same, you could just use quick shell script to parse that out and create a file when the script is executed. The file can then be read right in, without having you to specify the input file. There will be some tweaks you'll have to make to the script but it's very possibly for your specific scenario.

Reply Re: Free ESXi Backup Solution for Windows Nov 29, 2008 10:34 PM

in response to: aremmes

lamw 2,748 posts since
Nov 27, 2007

Thanks for the suggestions, I've just updated the script and uploaded the changes to http://communities.vmware.com/docs/DOC-8760

Reply Re: Free ESXi Backup Solution for Windows Nov 30, 2008 2:20 AM

in response to: kpc

Bolgard 113 posts since
Aug 11, 2007

kpc wrote:
Hi Bolgard.

The name is the thread is slightly misleading as the script has evolved over the past few months to one that runs directly on the ESXi server itself. A new thread should read Free Linux backup script If you still want to use Windows then the info is there (somewhere). Personally I'd recommend the LInux/ESXi route as it's far more more contained and versatile. You need to first enable SSH on the ESXi server (plenty of guides around), then SCP iawm's script to it and setup your NFS share. It all depends on what technologies you are using, e.g. Windows, Linux and how good/bad your linux is.

I use only Linux where I work, so have a 1TB NFS share that I use for my VM backups, I mount the NFS through the VI client, run the script - nice and simple... Hope that helped a little.....

I think I can manage using Linux. But you answered my questions spot on, what I didn't know was where to run the script and how. Thanks!

EDIT: If I have a virtual disk of size 80 GB, but using only 6 GB, will the backup be 80 GB or 6 GB?

Reply Re: Free ESXi Backup Solution for Windows Nov 30, 2008 8:20 AM

in response to: lamw

DSTAVERT 2,180 posts since
Nov 30, 2003

lamw

I'm just asking for comments here. This is your script after all.

SInce you are already dumping all VM's to a list (/tmp/vms_list) then comparing that list with the input, list why not use that loop to check for the existance of a file in the VM directory. If it exists add it to the /tmp/vms_list if it doesn't then don't. Then use the /tmp/vms_list to perform the backup. As part of the generate the /tmp/vms_list loop do all the checking for raw ot snapshots rather than burry them in the backup loop. In the generate /tmp/vms_list loop you can check for the existance of different files. The existance of a daily file would add that VM to the /tmp/vms_list every day. The existance of a weekly file would add that VM to the /tmp/vms_list only on friday or whatever.

Thanks for listening.

Reply Re: Free ESXi Backup Solution for Windows Nov 30, 2008 10:08 AM

in response to: lamw

alusrc 20 posts since
Nov 24, 2008

Should we be able to use VMWare Disk Mount utility to view these backups? I get an error doing so I was curious if there's an issue because it's a snapshot. I can bring the backup files to a new host and load up the machine that way, but I was looking for a way to open the disk image to do a file level restore and completely eliminate the need to use Backup Exec for these VMs.

This is an awesome script and is saving me lots of hassle backing up VMs. Thanks again.

Reply Re: Free ESXi Backup Solution for Windows Nov 30, 2008 10:12 AM

in response to: DSTAVERT

Intrepidity 2 posts since
Nov 30, 2008

I´m a bit unsure about the ability to schedule. Can I set this up using crontab in ESXi? I did read something about crontab in ESXi getting flushed on reboot? If the shedule can be set in ESXi I will set the backup to a basic NAS device with 3.0 NFS that will get mounted as a datastore in ESXi. If the schedule have to be set on the server/client receiving backups I suppose I will use Windows Services for UNIX on a 2003 server to be able to set the schedule on the server.

Reply Re: Free ESXi Backup Solution for Windows Nov 30, 2008 10:55 AM

in response to: DSTAVERT

lamw 2,748 posts since
Nov 27, 2007

I think your suggested solution is definitely one way of going about this problem. Though I would like to know how manually managing a file (defined as hourly, weekly, montly, etc) at a granular level per virtual machine directory basis helps administration and management of your backup process simpler? If I understand this correctly, what you are proposing can already be implemented using existing tools in a centralized manner.

If you're managing multiple ESX/ESXi Servers with say 15-20 virtual machines each and you begin to implement this backup solution, you will need to SSH to all the systems and figure out which virtual machines fit under which backup process or login via the VIC and tediously create/touch these files within each virtual machine directory. Secondly, you'll definitely need to modify the script (time investment needed for developing and testing) to add some statements that define the different backup schedule. What if the standard hourly, weekly and monthly is not acceptable for another user? Now you have to add more logic (more time investment) within the backup script and add complexity when the sole purpose of the script is to execute the backup process. By focusing on one general idea, the backup script can easily provide for flexibility on the end users' environment. If you're looking for granular control of your backup process (which I believe is a fair ) then this can be achieved by configuring and managing a cronjob on the ESX/ESXi itself or better yet, centrally configuring and managing kickoffs originating from a separate UNIX/Linux system.

If you're running an all Windows solution, you can use Windows task scheduler and plink to create an SSH connection to your host to kick off the backup.

These solutions enable granular control of the frequency of backups per virtual machine by exploiting the flexibility of the script without decentralizing the backup process.

Hopefully this addressed all your questions.

Reply Re: Free ESXi Backup Solution for Windows Nov 30, 2008 11:15 AM

in response to: lamw

DSTAVERT 2,180 posts since
Nov 30, 2003

Thanks for you response.

Kicking off different jobs from a Linux server probably makes more sense or using multiple cron jobs. Locating the input files on an NFS mount would make them easily accessable without needing to directly access the ESXi host.

Thanks again.

Reply Re: Free ESXi Backup Solution for Windows Nov 30, 2008 11:29 AM

in response to: Intrepidity

lamw 2,748 posts since
Nov 27, 2007

Hi Intrepidity,

You can setup a crontab for ESXi, unfortunately it's not as straight forward as it would be on an ESX or any UNIX/Linux host using "crontab". Having set that, there are a few solutions and it depends on how your environmental is setup.

Here are a few posts on some solutions that have worked or recommendations.

http://communities.vmware.com/message/1107603#1107603

http://communities.vmware.com/message/1108641#1108641

http://communities.vmware.com/message/1108327#1108327

If you're looking for a centralize location to configure your backup entries, you can do using Windows Services for UNIX using the scheduling services as you've suggested, or use windows task scheduler and plink.

For UNIX/Linux solution, what you can do is have a centralize Administratrive LUN that is visible to set of ESX/ESXi Server(s) that you need to run backups against and store different set of configuration files based on backup schedules and the backup script itself:

(e.g)

host1_backup_weekly

vm1
vm2
vm3
vm4

host2_backup_daily

vm5
vm6
vm9

host3_backup_hourly

vm4
vm9

Then what you can do is schedule the backup schedules using a cronjob and relying on SSH keys, you can remotely connect from a UNIX/Linux system and execute the backup schedule based on these configuration files.

(example crontab entry)

#weekly backups
22 4 * * 0 ssh root@host1 "/vmfs/volumes/ADMIN_LUN/ghettoVCB.sh /vmfs/volumes/ADMIN_LUN/host1_backup_weekly"
#daily backups
02 4 * * *  ssh root@host2 "/vmfs/volumes/ADMIN_LUN/ghettoVCB.sh /vmfs/volumes/ADMIN_LUN/host2_backup_daily"
#hourly backups
01 * * * * ssh root@host3 "/vmfs/volumes/ADMIN_LUN/ghettoVCB.sh /vmfs/volumes/ADMIN_LUN/host3_backup_hourly"

This again, relies on two keypoints:

1) ESX/ESXi seeing a common LUN to locate these backup files and the script itself, else this will not work.
2) SSH public/private keys are setup/enabled for this to work without having to prompt for a password when making the SSH connection

Sample Run:

This is a quick example that shows the execution of the backup script being executed off of our VIMA box to our remote ESX Server and having all it's configuration files/scripts located on our NFS LUN. This can be extended to as many hosts as you would like and configured to fit your environment.

[vi-admin@vima-primp-industries ~]$ ssh root@172.30.0.254 "/vmfs/volumes/kevin-test-nfs-storage/ghettoVCB.sh /vmfs/volumes/kevin-test-nfs-storage/simple_backup_file"
root@172.30.0.254's password:
################ Taking backup snapshot for RESNET-UCSB ... ################
Destination disk format: VMFS thick
Cloning disk '/vmfs/volumes/himalaya-local-SAS.VMStorage/RESNET-UCSB/RESNET-UCSB.vmdk'...
Clone: 100% done.
Removing snapshot from RESNET-UCSB ...
#################### Completed backup for RESNET-UCSB! ####################


Start time: Sun Nov 30 11:23:14 PST 2008
End   time: Sun Nov 30 11:23:38 PST 2008
Duration  : 24 Seconds

Completed backing up specified Virtual Machines!

Hopefully this helps answer some of your questions.

Reply Re: Free ESXi Backup Solution for Windows Nov 30, 2008 3:19 PM

in response to: lamw

Intrepidity 2 posts since
Nov 30, 2008

Thanks for the speedy response. I now got a good understanding of how I can set up a solution and the scheduling using a nas device, a windows server or a linux server.

However, I have yet another lame newbie question. Well, sorry, but here it comes.

Setup: 2 HP servers running embedded ESXi and hosting the VMs locally. Both servers have enough capacity on the datastore to host the VMs from the other server.

Question: Would it be possible to add the datastore of the other server in both servers and kind of cross-backup them? VMs from server 1 -> server 2 and VMs from server 2 -> server 1. This would have been done daily scheduled by crontab if possible.

Pros: If a server go down I can power on the VM straight away on the other physical server without having to copy it across the network first (slow due to big VMs). Easy testing of backups since they´re allready on a datastore where they can be booted and tested directly (without network connection for the tested VM).

This would be sufficient backup of the VMs since the most important data form within the VMs are backuped to a tape device anyhow. I am pretty much not allowed ANY investments right now due to the economic state of the company, so network storage or a backup server is kind of hard to get funding for right now. My second option is using an old server with server 2003 OS and USB attached drives. The scheduling through windows seems like the easiest solution for me anyhow. Might use both solutions with a backup weekly to the old server just in case. Transfer to/from the old server would probably be too slow to use it for daily backups.

Unless I get a negative response I will test this setup. I suppose it depends if it´s possible to add the datastore of the opposite server or not. I´ll at least give it a shot.

Thanks for taking the time to write such a money-saving script for the rest of us

Reply Re: Free ESXi Backup Solution for Windows Nov 30, 2008 3:45 PM

in response to: Intrepidity

lamw 2,748 posts since
Nov 27, 2007

You might want to check out Xtravirt Virtual SAN Appliance, it does exactly what you've stated, unused local storage.

http://engineering.xtravirt.com/products/phd-technologies/xtravirt-virtual-san.html

They're basically presenting the local storage out as an iSCSI Target to other ESX Server(s). I have not personally used their free product but I've had a quick glance when they had released this about a month or two ago. Good luck with that and post back if it works out for you.

Glad the script is working great for your organization.

Reply Re: Free ESXi Backup Solution for Windows Dec 1, 2008 2:13 AM

in response to: Bolgard

kpc 98 posts since
Jul 12, 2006

Bolgard wrote:

kpc wrote:
Hi Bolgard.

The name is the thread is slightly misleading as the script has evolved over the past few months to one that runs directly on the ESXi server itself. A new thread should read Free Linux backup script If you still want to use Windows then the info is there (somewhere). Personally I'd recommend the LInux/ESXi route as it's far more more contained and versatile. You need to first enable SSH on the ESXi server (plenty of guides around), then SCP iawm's script to it and setup your NFS share. It all depends on what technologies you are using, e.g. Windows, Linux and how good/bad your linux is.

I use only Linux where I work, so have a 1TB NFS share that I use for my VM backups, I mount the NFS through the VI client, run the script - nice and simple... Hope that helped a little.....

I think I can manage using Linux. But you answered my questions spot on, what I didn't know was where to run the script and how. Thanks!

EDIT: If I have a virtual disk of size 80 GB, but using only 6 GB, will the backup be 80 GB or 6 GB?

If you've only used 6GB of an 80GB VM and if you've set your backup to export the VM as 'thin' disks (e.g. -2 gb sparse) it will only copy that 6GB to your backup location, much quicker backups, but they do need to be converted back to 'fat' disks if you need to restore etc. Iams script has a setting that you can set for this. Hope this clears this up

Reply Re: Free ESXi Backup Solution for Windows Dec 1, 2008 6:48 AM

in response to: DSTAVERT

blueivy 22 posts since
Aug 9, 2008

I eventually completed a backup using OpenFiler - 22 hours for approximately 500GB of VM's.

My preferred distro is CentOS so I've just wiped OpenFiler and used that. My first problem appeared to be that I couldn't mount the XFS partition created by OpenFiler. I eventually formatted the partition using XFS again and it worked fine.

I also used the async option (thanks kpc) and I'm now getting a huge throughout (maximum 13603kbps according to the VI clientbut it's still pretty low. I was hoping for a lot more in a Gigabit network!

A question about async - aync is obviously faster because of the reduced overhead and can cause corruption. Just a thought but would an MD5 hash check at the end (although it's slow) not confirm the file was fine?

Reply Re: Free ESXi Backup Solution for Windows Dec 1, 2008 7:00 AM

in response to: blueivy

kpc 98 posts since
Jul 12, 2006

blueivy wrote:

I eventually completed a backup using OpenFiler - 22 hours for approximately 500GB of VM's.

My preferred distro is CentOS so I've just wiped OpenFiler and used that. My first problem appeared to be that I couldn't mount the XFS partition created by OpenFiler. I eventually formatted the partition using XFS again and it worked fine.

I also used the async option (thanks kpc) and I'm now getting a huge throughout (maximum 13603kbps according to the VI clientbut it's still pretty low. I was hoping for a lot more in a Gigabit network!

A question about async - aync is obviously faster because of the reduced overhead and can cause corruption. Just a thought but would an MD5 hash check at the end (although it's slow) not confirm the file was fine?

Hi blueivy

I'm guessing sync does a few extra things like resend packets, manage failures etc - just a guess. I've been wanted to read up on it for a while just never enough time All I know is that with async set my backups are about 5 times quicker. If you need some timings to compare post some of yours and i'll post some back.

Cheers

Actions

Incoming Links