Time Machine Backups of Virtual Machines

[gbullman]

A note for those who believe Leopard's Time Machine is backing up their VMware Virtual Machines, they are likely not being backed up in any usable form.

VMware Fusion is tagging each virtual disk file, and other critical VM files with a metadata value that prevents Time Machine from backing them up. If, in Terminal, you cd to the directory of one of your Virtual Machines and run the command ls -l@ you will discover this metadata value has been attached to some files;

com.apple.metadata:com_apple_backup_excludeItem

I haven't been able to determine if this metadata setting has been available since Leopard 10.5, or whether it was added in 10.5.1, or 10.5.2. If you've been using using Time Machine and Fusion for a while you may have some older backups, but certainly not after one of those OS updates were applied.

Given that any backups while the VM is running are pretty much useless anyway it does make sense to exclude VMs from Time Machine Backups. What bothers me is that has not been made clear to users of VMWare Fusion so they take the proper steps to ensure their VMs are backed up. I assumed any time that Time Machine ran and I did not have any virtual machines running I was getting a good backup, luckily I found that I wasn't before I had to depend on a backup.

Hope this information prevents headaches for someone out there.

[asatoran]

...IF ANYONE FROM VMWARE IS READING THIS OR ANYONE ELSE WHO HAS THE ANSWER, PLEASE TELL ME WHAT THIS IS SO I CAN FIX MY VITAL BACKUPS.

Simple answer: Don't use Time Machine to backup the VMs. I simply attach an external HD to my Mac and drag the shut down VMs manually. You could automate this but I find it's trivial to just start the copy and let it run overnight just before I go to sleep. (It doesn't take that long, of course. I just don't want to wait. ) I've been doing this for almost a year and have had to do a clean reload of my Mac twice (for various unrelated reasons.) But with the simple backups I perform, restoring my virtual machines after each reload was a non-issue. I have several mission critical VMs that I need and have not had a problem with restoring any of them with this simple backup strategy.

To be clear, I still use Time Machine to backup the rest of the Mac. Just not the virtual machines.

[MichaelMonschau]

I wish I could do that, but to backup 20-30 gig of virtual machines from 2 Macs at a moments notice is not possible and my time is very precious to waste on tasks that should be automated. I need them to be part of the automated backups across the network. It isn't always me who uses them. I also need them backed up regularly during the day. It is just too much of a burden and just to easy to forget. I think this needs to be sorted.

[asatoran]

I wish I could do that, but to backup 20-30 gig of virtual machines from 2 Macs at a moments notice is not possible

Why a "moment's notice?" It sounds like you need to use snapshots more.

>and my time is very precious to waste on tasks that should be automated.

That is the reason I run the backups overnight. Like I said, the process can be automated with something like Applescript is you really need to. On my terabyte HD, I have multiple copies of each VM. So I have historical backups, but mostly I use the multiple copies as a way to quickly test something "at a moment's notice." After the test, I delete the test VM, and make another copy. The copy could be of one of the other previously made copies. This way I always have a test VM ready "at a moment's notice."

I need them to be part of the automated backups across the network. It isn't always me who uses them. I also need them backed up regularly during the day. It is just too much of a burden and just to easy to forget. I think this needs to be sorted.

My procedure for making additional copies for "a moment's notice" nearly eliminates my need to completely automate my backup during the day. YMMV of course. But at my office, mission critical machines that need to have more than one backup during the day (i.e.: hourly backups), we treat the VMs just like we would physical machines. For example, Backup Exec System Recovery within the running virtual machine doing differential backup images. In essence, what Time Machine does, but without involving OSX. We do this because we're backing up physical machines as well as virtual. However, the mission critical systems that need backups that often, we more rely on other means to backup the data, rather than the whole machine. (e.g. SQL transactional backups to separate files on network.) That reduces the amount of data in each backup, thus saving precious time again. Again, YMMV.

My point is that you do not need to rely on Time Machine. The issues you're facing with your backup scenarios have been around well before Time Machine was released. So there are other methods that you can use, and should use, for your mission critical items. My suggestion works for me in what I consider mission critical and "at a moment's notice."

[gbullman]

You need to read the beginning of this thread, unless you at least suspend the VMs when Time Machine is backing (perferable to shut them down) you are unlikely to get a useable backup of your Virtual Machines with what Time Machine captures. From my experience with FW400 drives I would expect it to take you 20 to 30 minutes to back them up, maybe faster if they reside on your internal disk and you want to back them up to an external drive. Even though we've all gotten accustom to large drives and have lots of storage it still takes a while to copy files of that size.

[MichaelMonschau]

OK, lets put this to rest. For, if you like, my own personal reasons, I need to use Time Machine. Our three Mac Desktop machines, two wireless laptops, and the additional virtual machines are backed up at all times to single network drive, that I can unplug at any time and take away.

Time Machine has solved all our problems and all was well while we used Parallels Desktop. For some silly reason I let my self talk into switching to VMWare. And from that day onwards, our virtual machines did not back up properly.

The backups should work and do work for all machines created with Fusion 1.1.3 or better. After running a virtual machine we all now to just select 'Backup Now' from the menu and we have a good backup. Anyone of us can manage that.

A backup of anyone of my virtual machines only takes 5 minutes across our gigabit network so within any 10 minute period I can have the latest to take with me. But to try and manually backup these things is not workable. Time Machine for us is perfect.

[gbullman]

Did you ever restore a Parallels VM from TM and test that it works? You may have been backing up without errors, but it is a differnt question if something like a running virtual machine backed up in this manner is consistent and useable. As far as I know with proper configuration you can get Fusion VMs to back up with Time Machine. I'm trying to save you major headaches down the line if you ever have to restore one of these backups (would expect Fusion & Parallels to have the same behavior for this).

Basically have a collection of large files that are consistent at any given point in time. No matter what backup method you use it will take time to back up the entire set of files. By the time you get to the last file, previously backed up files will probably have changed because of activity within the VM. If you suspend, or shutdown the VM during the backup then this is not a problem, if it is running while being backed up there is no guarantee the resulting backup will work.

[asatoran]

...

A backup of anyone of my virtual machines only takes 5 minutes across our gigabit network so within any 10 minute period I can have the latest to take with me. But to try and manually backup these things is not workable. Time Machine for us is perfect.

You do realize that the 5 minutes is not a full backup, just a differential. So your restore in a DR case will still have to copy the entire 20-30 gigs (or whatever.) Like I said, you really should be looking at snapshots if you need moment in time restore points. Snapshots are quick to create and restore. I don't use them much because the procedures in my dev environment didn't want the complexity of restoring differential backups in DR situaltions. So the procedures have taken into account the time needed for full backups. (20-40 min for a full Backup Exec System Restore image of most of my servers. But this is while the systems are running so can be done at nearly anytime with no downtime. BESR can do this because it snapshots the OS and backs up the snapshot.)

But if you've made up your mind about Time Machine backups, then yes, I'll stop here. You asked for suggestions, and I gave some, although obviously they are not what you wanted to hear.

[MichaelMonschau]

Yes, they work. I tested all this, I am very paranoid and very thorough about these things. Been doing this ever since Parallels released their first version.

When you finished with a virtual machine you simply hit "Backup Now" and you have all the healthy bits backed up. As I said before, that we can manage. We have too many virtual machines (too many platforms, various linux, Windows XP/Vista/2000) to have separate backup procedures for every one. As I said, Time Machine is perfect if you know how to use it and we do. It's just Fusion that is the problem at the moment

Look Guys, this is distracting from the real issue. Virtual machines created with versions prior to 1.1.3 have just one disk image file in the package that will not back up, no matter what we try. So if anyone knows why that could be, that would be great. If no one knows the answer, I will be forced to switch back to Parallels.