more questions abouts backing up VMs....

[JRink]

I'll try to provide some background information....

I have two different ESX environments that I manage at different companies. Both ESX implementations are pretty small. Here's the breakdown:

Company 1:

• Single ESX host with currently (3) Windows VMs, soon to grow to (5) Windows VMs.

• Storage of the VMs is currently on the ESX internal drives -- however, I plan to move these to our SAN storage in the next week using iSCSI.

• Total disk space used for the VMs VMDK files is appoximately 400GB at the most, including snapshots, etc.

• Backups of the VMs are currently done using a BackupExec 11d AGENT on the VMs to backup the 'data'.

• We have an HP Tape Autoloader with (8) slots available where we could potentially store the vmdk backups

Company 2:

• Single ESX host with currently (7) Windows VMs, and (1) Linux VM.

• Storage of the VMs is currently on a Storevault S500 SAN over iSCSI.

• Total disk space used for the VMs VMDK files is approximately 400GB at the most, including snapshots, etc.

• Backups of the VMs are currently done using a BackupExec 11d AGENT on the VMs to backup the 'data'.

• We have an HP Tape Autoloader with (24) slots available where we could potentially store the vmdk backups.

I am considering my options to start backing up the entire VM, as opposed to just the data on the VM. Mainly in the thought of what would happen if my SAN(s) died. Granted, I could rebuild the VMs and restore the data to the VMs from the BackupExec AGENT, it would be time consuming so I'd rather be able to also backup the actual VMs themselves on a consistent basis.

I've read a litlte bit about vRanger, ESXpress, and VCB, but am curious as to what others would recommend based on MY environment.

I'm thinking I would use one of these programs to backup the entire VM to a network disk (probably on the SAME SAN where the VMs are stored since that's the only place I have free disk space available), and then backup the disk backups made by vRanger, ESXpress, etc. to TAPE (via BackupExec) for permantent storage.

Any feedback is appreciated... Like I mentioned, in the past I've just been backing up the data on the VMs and have been content with that. And, if I needed a complete VM backup, I would shut down the VM and use FastSCP to copy off the flat files to a temporary location. But, as I add more and more VMs, I'm starting to think maybe I'd like to backup the entire VMs on a more regular basis in addition to backing up the data ON them.

Thanks.

J

[ctfoster]

Everybody has there own take on backups. For my own viewpoint I'd suggest you have a look esXpress for the reasons below.

Its licensed by the host so would be pretty cost effective for you. It's also easy to set up and remove so you could give it a go and if it doesn't suit you you've lost nothing.

You could back up to local VMFS or a network ftp/smb/ssh target and move to tape from there. However the compression and delta technology makes it very efficient regards storage space. An investment in a 3Tb NAS device would easily allow you store a months backup (4 X 400 + deltas.) before you had to move it to tape. When you are dealing with remote sites having backups on disc rather than tape is a bonus.

Lastly you could make of the local storage that you are going to free up by the move to SAN by recovering the backups back to disc to have some ‘hot copy' VM's. This feature is called ‘simple replication' and is a doddle to set up. Apart from having the VM's from the last backup ready to go, you are also testing every backup with a restore.... and you know what they say about backups.

[mike_laspina]

Hello,

Before looking a a technology or product i would suggest you layout what you need in terms of recovery point and recovery time.

If you need to recovery to yesterdays data after a fire you will need to determine what speed the backup process requires and what technology to use for data shipping.

In this case directly to tape may be fine. Using the likes of Backupexec etc may be sufficient.

Or maybe your local backup is OK and weekly tape shipping will be satisfactory.

Also a solution like esXpress could fit very well. You could have cheap NAS boxes switched out weekly to create DR capability etc. and local backups.

These items depend on the customers data loss tolerance.

Will 7 days put them out of business?

How much is a day's of data worth to them if it were lost?

Hope that helps.

[dmorgan]

We use VCB, mainly because it is licensed as a part of our Enterprise package, and it does what we need it to do, so no need to spend the money on a third-party solution. The one thing you don't mention is what those VM's are, i.e. are they database servers, file servers, etc... the reason I ask, is it sounds as if you want a fast, DR type backup that you could simply restore and power on should the VM die or have problems. The simplest way, using VCB, is to do a fullvm backup. That would allow you to restore the VM in it's entirety, and simply power it on. If any of these VM's, however, are database servers, or Exchange servers, or similar type systems, this doesn't work as smoothly. You would only have a crash-consistent copy of the database, which may or may not be fine for you. If you already have enterprise licenses for those boxes, and thus a license for VCB, try it out. We do a combination of VCB full backups, file level VCB backups, and database agent backups on our VM's. I take a monthly full backup of our database servers. I then do more frequent backups of just the database using the Arcserve agent. Same principal would apply for Backup Exec. That way, should the system need to be restored, it could be restored using the fullvm, and then the latest database restored on top of that. Again, it really depends on the types of machines you wish to back up as to the best way to accomplish it.

If you found this or any other post helpful please consider the use of the Helpfull/Correct buttons to award points

[khughes]

Both companies are pretty simular, big or small still bulk backups of the vmdk's is a great thing to go for. We use esXpress as well and we love it. You should continue to keep your file level backups going because it's a system that obviously has worked well for you. I'm pretty happy with our setup right now (would tweak a few things but that's another matter)

When we virtualized our network we keep our BackupExec file level backs ups going for small file restores (ie: users deleting the wrong files). We decided to use esXpress in addition to our current scheme by doing bulk backups and then using BackupExec to backup the esXpress backup files to tape. We just added that bulk backup on tape to our current rotation of off site tape backs. Great thing about that is we can run the backup of the esXpress backup files at any time, usually during the middle of the day. With this setup we have a full vmdk file backup if something happens to the system onsite to restore it right then, or if we have a total building failure like a fire or we loose the SAN we have the offsite backup of the vmdk files which we can restore from. It does all depends on how fast you need the systems backed up and such. Hope that information helped out.

[JRink]

Can you elaborate on the term "crash consistent"? I've read this before, but not sure I understand exactly what it means.

My VMs are a mix of things. Some are simple file/print servers, some are basic web servers, some are Active Directory servers, and some are Exchange servers.

Will esXpress, VCM, vRanger, backup all these kinds of VMs without problems? I'm mainly concerned with if there would be problems with live backups of any VM hosting a DB (Exchange, Active Directory, etc.)

JR

[dmorgan]

Databases are always the kicker when it comes to backups, VM or otherwise. Crash consistent basically means that the backup would be equivilent to the system crashing and restarting. The backup would be restored, and the database would have to recover as if the system had crashed and is now just coming back online. This works in many cases, but is not ideal by any means. What you really are after is a backup that would restore the database to the exact state that it is in at the time of the backup. In order to do this, you either need to use backup agents, Arcserve, Legato, BackupExec, etc... or you need to quiese the databases/shut them down/put them in a backup-able state prior to backing them up. VCB, and I assume esXpress and vRanger as well, don't do anything specifically to address applications such as databases running on VM's. If a file server VM gets backed up, it's easy. VCB, for example, takes a snapshot, mounts that as read-only while the delta is recorded to seperate files, and you back up the read-only portion. That gives you an exact copy of the VM at that moment in time. File-locks are irrelevent because of the snapshot. However, databases deal with more than just locked files. You want to make sure that no transactions are lost. The only way to do this is to put that database into a state that ensures this. We use VCB to do full level backups, enabling us to restore the whole system to that point in time. Then, I do more frequent backups using Arcserve and oracle/sql/exchange agents to back up the databases. The agent handles everything, enabling you to restore that database to that point in time with no data loss. Thus, if I lost our exchange server lets say, I would restore the full VM, and then restore the Exchange backup on top of that. If I just had an exchange problem, I could restore just that portion. If you can shut down the database prior to backup, then you could just do a full backup using VCB, or esXpress or vRanger or whatever, and it would be fine. However, most of the time you cannot just shut down your database for the time it takes for the backup to complete.

If you found this or any other post helpful please consider the use of the Helpfull/Correct buttons to award points

[mike_laspina]

Hello,

In addition to Don's good explanation I can add further detail to help understand VMware's term ... "Crash Consistent" ...

In real hardware the disk adapter can use cache to improve I/O performance unfortunately this can harm a database because it will trust that the data was wrtien to disk. It leaves room for problems if the system were to hang, power off etc.

As well adaptors that have cache functions usually implement battery backup to recover from a power/hang event.

To fix or and prevent this issue vendors came up with a SCSI tag bit called forced-unit-access and when this bit is set the hardware will not cache the write. Thus maintaining integrity at the app level if supported.

Unfortunately a hypervisor does not have application knowledge and cannot validate which writes should occur in what order because it is performing I/O cuncurrently across all VM's and the performance cost would be to great to track them.

So the solution is to queice the writes using the LGTOsync driver in the VM so that the virtual disk adapter memory elements can be physically written to disk and thus the VM applications are in a crash consistent state.

This queice is performed by running a snapshot right before a backup.