Backing Up a Mac and a VM with Time Machine

[plpost]

I've been using an external back up hard drive for my macbook. After installing VMware Fusion I couldn't help but wonder if time machine is backing up the WIndows XP virtual machine as well or do I need to run a separate back up hard drive from within Windows to back up the Windows VM?

[blackpuma]

I've been running VMware workstation since its initial version. VMware has never produced a version that has VMs that can be snapshotted and backed up while "hot". I understand the reasons given, but as a software engineer think it'd be an interesting problem to solve.

Anyhow, work off of that constraint.

Here's what I do on the Mac:

1) Do not use TimeMachine. It's not designed to handle VMs. It will screw up any large file that's being written to frequently. Complain to Apple, not VMware for the piss poor excuse for a backup program. (Another rant for another time.)

2) I use SuperDuper! for backups when I'm done working. It's not perfect, but has produced rock solid backups that I can boot off of. Ioften just let it run in the wee hours of the morning.

2a) I keep two aged copies, yesterday's backup and the day before yesterday's backup.

3) No need to shut down VMs before backing them up. If you move them to other machines, you'll want to shut them down, but not needed for backups.

On Linux machines, I just use cron+rsync in the wee hours, which also works on the Mac if you have a gigabit connection to a fast backup machine.

[Technogeezer]

My advice: Don't back up the virtual disk files of an active VM. You have to treat it like a database - unless you have facilities to quiesce the VM so that all of the files comprising the vmdk are in the same state for consistency (which VCB does on ESX), backing up the vmdks of a running guest is a crap shoot.

Yes you might get away with it. Until the first time your VM is writing data while the vmdks are being backed up. Think about it: how good is your backup if you do an image level backup of your disk while the disk is changing? Not very because those changes, if done in the right places at the right time, can result in a backup that is useless - either partial or complete data loss. If you're really lucky, the changes occur in the directory structures of the NTFS or FAT file systems. Inconsistent changes captured in those areas could turn your VM into toast.

Apple never designed TIme Machine to back up large database-like files while they are in use. It's designed for the much more prevalent case of smaller files that comprise the OS and user documents. And for that purpose it works well.

In case you can't tell - I'm also not a fan of Time Machine backups of VMs.... Wastes too much time and disk space.

[dp_fusion]

I'm not sure why people fight the idea that backing up a running machine is a bad idea. You can't even do it on a physical machine. All that is needed is to put the shut down VM in a container of some kind. That can be a zip file, a tar file, or even a disk image (mountable dmg). This can be done with a simple AppleScript or a shell script.

To do the latter:

1) shut down the machine - don't suspend it. This is a backup so you don't want surprises later

2) Locate the virtual machine in finder. Ctrl-click on it and choose Get Info

3) Note the size - it will be severa gigabytes large.

4) open the disk utility and select disk image. Give it a meaningful name and destination somewhere in your home directory tree, make it an encrypted read/write disk image, and 10% larger than the size of the VM seen in Get Info in step 3)

5) When the image creation is completed, mount it, and drag your vm into it. This will produce a copy of the original.

6) Unmount the disk image and TimeMachine will back it up on the next cycle.

Can you mount the image and run that copy? Yes. But if you do it should no longer be considered a backup. Buy a nice 32 gig flash drive and copy the vm (not the dmg file) to it - you now have a traveling VM you can run in Windows, Linux, and Mac OS X. Toss in a copy of VMPlayer installer for Windows or Linux, or a dmg copy of your Fusion package and you have the complete kit.

If the worst happens and your original copy is corrupt, shut down Fusion and rename the defective vm object. Don't delete it until you've exhausted every option at running it again. Mount the disk image and copy the vm backup to the folder it was copied from originally. That's it - backup and restore.

[Dysan819]

Why such complicated procedures?

This works just fine:

Shut down VM. Copy VM folder to back-up drive and wait 45 minutes (eat dinner). Done.

Now, lets focus on why we can't use Time Machine, which is what this thread is about. I don't care about step 1, 2, 3, 4, 5. I want the OS to do the work. Don't tell me this isn't possible. Someone can figure this out.

-Brett

[Dysan819]

I don't think Fusion users as a whole care about platform independence of their VM. They bought Fusion to run on a Mac, and that's probably the only host OS they'll ever touch. So I don't believe that argument is valid in this case. If sparsebundle is a viable option, it should be explored, and the user can be given the choice.

[dp_fusion]

If it were easy it would be done. It's definitely not easy. As for why so many options, options are good. More options can be better. In my case it was an opportunity to show several non-TimeMachine based, easily accomplished methods and opportunities that are available. In any event this is not a Fusion problem. It is a user education issue.

What is wanted is not possible with conventionally sized disks. There's also another distinction that has gone unidentified. A backup is not the same as an archive. If it is needed to restore something to a point in time then you need clever software that makes proper archives and plenty of room to store it all. That is not TimeMachine. If all you need is disaster recovery of recent data then backups can work. TimeMachine is useful here, but not on the native files. If you don't care that TimeMachine is involved (your solution) then there are myriad ways to do the job, and some offer unexpected benefits.

In summary: TimeMachine is the wrong tool for incremental and differential backups of virtual machines. That is not a Fusion or even virtual machine problem. It's really that simple.

[blackpuma]

" lets focus on why we can't use Time Machine"

Wrong tool for the job. Besdies the fact that TM will fill the disk with corrupted disk files… TM deletes files without warning, for example, when disk space gets tight… and VMs will fill up the external drive FAST.

Try having one of the large disk files in your VM disappear without warning, leaving you with a non-functional VM… with no option other than to just delete the freaking thing.

Been there…

"Don't tell me this isn't possible. Someone can figure this out."Why not tell the truth? TM was never designed to handle VMs. Use not the right tool for the job instead of trying to shoehorn an elephant through a keyhole?

[wila]

Hi,

I am a programmer and unfortunately I cannot store my "important data" in a shared folder on the Mac side, because it increases the compile times by 1000%. It literally takes at least 10 times as long. It would be great if the shared folders weren't accessed through a "network" (why is this done?) and actually ran at normal speeds.

So don't ask me how, I'm an end user. It's called a feature request.

Why not try this then (it is roughly what i use)

So putting the files on your shared folder to compile them isn't fast enough, but... you could use a simple batch script to unconditionally copy ALL your program files to a share on the host (or a network share if you prefer). Then setup a task in your VM to run the script automatically at a certain time.

If you copy the files to the host, they now are part of the time machine backup at a file level instead of disk level.

Of course as a programmer you are also already using a source code control system, so your files should already be reasonably safe.

--

Wil

_____________________________________________________

VI-Toolkit & scripts wiki at http://www.vi-toolkit.com

[blackpuma]

"I am a programmer and unfortunately I cannot store my "important data"

in a shared folder on the Mac side, because it increases the compile

times by 1000%."

… uhm … you're using some form of revision control, aren't you? Sounds like a clear-cut case of wrong tool for the job.

$ svn update ; make

[wila]

Heheh...e xactly, but use of source code control is not automatic in all programming environments whereas time machine does make an automatic backup.

So he does have a point there. This is why i add another layer of protection and copy my files onto the LAN (or host share) on a scheduled task.

--

Wil

_____________________________________________________

VI-Toolkit & scripts wiki at http://www.vi-toolkit.com

[Dysan819]

Being the only programmer at my company, version control has not been an issue, and simple backups should suffice. I do backup all data to a server on a semi-regular basis, so I am not to worried about losing everything. I'm just saying that there's a resource that can be used (Time Machine), which does a pretty good job keeping EVERYTHING else I do backed up, so why not press the point?

I like the idea of setting up a script to copy files to the Mac, but then it would take twice as much disk space. It's the best idea I've heard so far.

I also never said that VMWare is to blame for this situation, although they did create Fusion and are probably in the best position to create a solution.

For clarification, I don't necessarily need to back up the entire VM, just my data, to a Time Machine drive. This should be do-able without scheduled tasks and scripts.

(For additional clarification, I'm not dumb, just very lazy.)

[wila]

Being the only programmer at my company, version control has not been an issue, and simple backups should suffice.

Actually I disagree, version control offers a a lot more as just backups. I'm a single developer as well and use version control systems for my sources. Sometimes because my customers demand it, other times because i can't live without them

Would never consider to go back to the simple backup zip version method. Yes it works, but it takes time and does not offer much. Here's a link to an article i wrote about using subversion for a programming language called "Visual DataFlex". Disregard the language, but check out what you can do. Subversion - what is it?

I do backup all data to a server on a semi-regular basis, so I am not to worried about losing everything. I'm just saying that there's a resource that can be used (Time Machine), which does a pretty good job keeping EVERYTHING else I do backed up, so why not press the point?

The problem is disk level vs. file level backups. I wish the host guest filesystem share was more performant as that seems the best way in solving this.

I like the idea of setting up a script to copy files to the Mac, but then it would take twice as much disk space. It's the best idea I've heard so far.

It works quite nicely, if you have a compact flash slot, you could even write it out to a CF card. Actually I'm zipping my source files all to a network share or if that's not available to a host share. If you want to use TM then it is probably best to write it out unzipped to a normal host share. You could use rsync on windows as well instead of copy if you want to speed up the process.

For clarification, I don't necessarily need to back up the entire VM, just my data, to a Time Machine drive. This should be do-able without scheduled tasks and scripts.

(For additional clarification, I'm not dumb, just very lazy.)

Understood, just pointing out a way how you can work around the issue. I also use version control, but don't like to trust a single backup method as then that one has to work when things go wrong. So I backup in layers, never had an issue with too many backups, but not enough backups has bitten me in the rear end before and I need my backups to be made automatically as that way at least they get created. I don't check my sources in each day, but do so when i finish a feature.

--

Wil

_____________________________________________________

VI-Toolkit & scripts wiki at http://www.vi-toolkit.com

[SvenGus]

An interesting link that I just found:

http://www.markwheadon.com/blog/2009/06/backing-up-virtual-machine-using-sparse-bundle

Not exactly making a .vmwarevm in .sparsebundle format, but rather putting a VM inside an ad hoc sparse bundle: might be better than nothing, anyway, if it really works; it could maybe even be made automatically in some form, so that it is transparent to the end user.

Anyway, it would be really cool if Time Machine could eventually take care of every form of backup, including virtual machines...

[ColoradoMarmot]

Interesting, but again, not cross platform compatible.

Comments to other postings:

Until the underlying file system supports hot file backup, or VMWare supports hot backups (like Oracle databases, by writing to log files during the backup of the main database, then applying changes later), it's not going to happen. Fusion is a consumer product, and that's a server technology. The cost to implement it in Fusion would be prohibitive (it took Oracle years to get it right). Maybe when we see ZFS, but not until then. Time Machine is not fundamentally architected to handle large files that incrementally change - for example if you change the metadata in a 2GB .mv4 movie file in your iTunes Library, it'll backup the entire 2GB file again, not just the 10 bytes that change. If you want that kind of discrete backup, look to something ilke Tivoli.

[blackpuma]

"Being the only programmer at my company, version control has not been

an issue"

I used to think the same way. Even on my own stuff where I'm tinkering or otherwise goofing off, I religiously use revision control (in my case, Subversion). I'm too lazy to NOT use revision control.

You really, really, really should look at using Subversion or some other revision control system. It takes a modest amount of effort to set up and some time getting used to the discipline of using a RCS. The upshot is that it frees your hands. You don't realize how limited you are when using the archived ZIP file method. When that dawns on you, you'll kick yourself for doing things the "easy" way. The other things a RCS gives you is the ability to see notes on your changes, do comparisons between revisions, branch off to test ideas, take release snapshots, do maintenance on release branches, etc. etc. etc. It's all stuff that as a solo programmer you'll fall in love with once you get into the groove. It's not just for large corporations.

"there's a resource that can be used

(Time Machine), which does a pretty good job keeping EVERYTHING else I

do backed up, so why not press the point?"

Because there are are other tools designed to do what you want… without words like "pretty good". Those words imply points of failure. Define "pretty good"… and your expectations, where you're willing to burn up time during a time crunch because "pretty good" wasn't good enough?

Why do I avoid TM? Wrong tool for the job, specifically:

1) Reliability (or lack thereof)

2) Predictability (zero)

The big reason I stopped using TM was the fact that you have no control over when TM **DELETES** your old files. If you don't mind, in the same directory, having one file that has copies for one file that go way back, and another file that has one copy from ten minutes ago… TM may be for you. I've even found a handful of larger files that had ZERO copies. TM in its infinite wisdom decided it wouldn't back those files up. I found TM to be a horrible crap shoot in that regard.

TM is a great help for home use. People rarely back up their stuff at home. It's a fabulous tool to help people have casual copies of their data. That's what it was designed for. It seems to do a decent job of that. However, do not trust your important data to TM.

" I'm not dumb, just very lazy."

LOL One of the Four Great Virtues of programmers. I hate recovering from lost data.

For me, Subversion + SuperDuper! prevents aggravation and premature hair loss.

[ColoradoMarmot]

I understand having your souce code local in windows for performance reasons (the shared folders do slow things down). One solution to consider is Rational Team Concert - it's a cross-platform solution that provides a lot more than just version control (it works with both .Net and Java development). www.jazz.net

[admin]

I had thought, wrongly I now suppose, that the suspended VM was independent of the Fusion application? Does suspending the VM somehow tie it to the particular version of Fusion running at the time?

Officially, we don't guarantee that Fusion will be backwards compatible with the suspend format, so bringing a suspended virtual machine across versions could cause problems.

Unofficially, we'll try not to break it, but best practice is to shut down virtual machines down before updating Fusion (or any other VMware product).

[blackpuma]

"Unofficially, we'll try not to break it, but best practice is to shut

down virtual machines down before updating Fusion (or any other VMware

product)."

Well if you didn't do such a good job of not breaking things, there wouldn't be the perception that it's all good. Geeeez… if only you guys wrote crappy code… oh, wait, never mind. :smileysilly:

[Carofhull]

Your external hard drive should be backing up both, your Mac and your Windows.

Your back up drive will fill up fast due to Windows on the PC. I just got a 1 TB and was told to back up only once a night so not to fill it up so fast. My 500 GB filled up in less than a year due to the PC on my Mac.

I don't think time machine is backing up the PC. You can get another program to do this for the PC. I think it is called Handy Backup by Argentum go to www.argentuma.com check to be sure this is correct.

Good luck

Carol

[davidb2]

Officially, we don't guarantee that Fusion will be backwards compatible with the suspend format, so bringing a suspended virtual machine across versions could cause problems.

Thank you for the explanation--now I get it.

David