How do I restore a virtualized OS X Lion 10.7 client with Time Machine?

[ShinySteelRobot]

Hi all,

I've got a Time Capsule set up so that my virtual Lion clients back themselves up. This works great; my Lion clients see it as a remote Time Machine and happily back up regularly. So far so good.

However, if I have a problem with a virtual Lion client and need to restore it from the remote Time Machine backup, I always get an error that says something like this:

"You can't restore this backup because it was created by a different model of mac"

What? It's the very same virtual client that I created the backup with!?!?!  How do I fix this problem?

(On a tangential note, in a virtual Lion client when I try to enter Time Machine to recover a lost file or two, the client appears to freeze up.)

Thanks for any help!

[ShinySteelRobot]

PS:
I worked around the problem by fully reinstalling Lion (argh) within the virutal Lion client, and then using Migration Assistant to "Transfer information from a Time Machine backup" (argh). This two-step process is not as reliable as a direct restore from TM backup.

Obviously this is very slow and tedious when I should be able to simply start a TM restore. Any help appreciated, thanks!

[ColoradoMarmot]

Doing a full restore from time machine backups isn't reliable to begin with.

Just backup the entire virtual machine file on the host (which *cannot* be safely done via time machine).

[ShinySteelRobot]

Thanks for the suggestions.

Doing a full restore from time machine backups isn't reliable to begin with.

Of the four times I've used Time Machine for a full restore on my Macs over the past 4 years or so I've never had a problem with it. Millions of Mac users trust it. It has the advantages of being the official Apple solution for Mac backups, it's incremental so it doesn't take much storage space or network transfer time, and normally It Just Works (on real Macs).

Just backup the entire virtual machine file on the host (which *cannot* be safely done via time machine).

A backup of the VM itself is a fully copy every time. Maintaining the last 30 days' worth of backups that way would take too much space (50 GB per VM x 30 days x 3 virtualized servers) not to mention each daily backup would take forever to transfer across the network to the NAS drive we have. OTOH, Time Machine backups are hourly and don't take nearly as much space since they are merely deltas, not full backups.

Ideally the VMware Fusion team can TM backup/restore work reliably like it does for real Macs. Hopefully it's coming, they just haven't gotten around to it yet.

[WoodyZ]

Count yourself very lucky and  probably in the minority as it is a known fact that Time Machine is not 100% reliable backing up/restoring Virtual Machines under all circumstances/conditions.  Also backing up Virtual Machines via Time Machine is disk/time intensive and wastes a tremendous amount of space for something that may be corrupt and worthless come time to restore it.  At a minimum I would exclude Virtual Machines from Time Machine and with the Virtual Machines shutdown, not suspended, and VMware Fusion closed then manually copy the Virtual Machines Packages to an alternate location, preferably on to a different physical hard disk.  Then keep the User Data that is stored within the Virtual Machine backed up off of the Virtual Machine on a regular basis so as to always have a current User Data Backup.  If you have to restore a properly backed up Virtual Machine that is not as current at least you'll have a working Virtual Machine and current User Data to go forward with when you find out your Time Machine backup of the Virtual Machine fails.

[Technogeezer]

ShinySteelRobot wrote:

.

it's incremental so it doesn't take much storage space or network transfer time,

.A backup of the VM itself is a fully copy every time. Maintaining the last 30 days' worth of backups that way would take too much space (50 GB per VM x 30 days x 3 virtualized servers) not to mention each daily backup would take forever to transfer across the network to the NAS drive we have. OTOH, Time Machine backups are hourly and don't take nearly as much space since they are merely deltas, not full backups.

TM backups are incremental at the file level, not the block level. This behavior is one of the reasons why a TM backup of a VM is inefficient. Change one byte of a vmdk file and the entire file is backed up. Vmdk files tend to be large, so the backup volume is large too.

So those hourly backups will still take up a ton of disk space. It will be the eqivalent of a full copy of the VM each time TM runs.

Also, if you do decide that you want use TM to back up a VM the only safe way to do it is if the VM is shut down. You are almost guaranteed of having a corrupt VM if you are backing it up while the VM is wiriting to its virtual disks   TM doesn't do any kind of snapshot or open file backup, nor does it "quiesce" running vms so that a consistent version of the virtual disks are on physical disk. Backing up files while they are changing is just as bad as not backing them up at all.

Unless and until Apple changes TM , I don't hold out much hope that VMware can make TM  work for VM backups.

[Technogeezer]

However, if you are running TM inside a virtualized Lion guest, that is pretty efficient for protecting the guest. Just remember that your recovery is going to be similar to recovering a physical machine -a file by file recovery that will take a good bit of time. Not as simple or quick as replacing a handful of vmdk files on the host.

[ShinySteelRobot]

Right, as described in my first post, I'm using TM within my virtualized Macs to back them up to a remote Time Machine device.

But the problem is that if that Mac VM client needs to be recovered from backup, you can't simply boot that VM and restore it due to the error described in my first post (i.e., "You can't restore this backup because it was created by a different model of mac"). This seems to be a problem with VMware Fusion.

I'm not backing up the *.vmwarevm files with TM from the host Mac. That would be pretty inefficient as well as probably pretty unreliable.

Also, as mentioned above, entering the Time Machine "stars" UI in a virtualized Mac results in the client freezing up, which also appears to be a problem with VMware.

In a nutshell, the current version of Fusion and Time Machine don't get along.

[ColoradoMarmot]

Time machine is notorious for silent corruption of backups - particularly if run over a wireless network.  The first notice that it's corrupt is a failed restore.  I have thousands of mac users in my organization, and between dozens and hundreds of examples of failed restores.  It's fine for individual files (assuming it backs them all up - which it doesn't), but it's a 10-15% failure rate isn't remotely a reliable backup solution.

Backing up inside a client is very different from backing up the VM itself - that works just fine (run the migration wizard after building a new OS install inside the client).  Backing up the host VM files simply isn't possible.  Every time Fusion runs, the time/date stamps on the entire set of disk files change, which triggers a full time machine backup.

[ColoradoMarmot]

Actually, once the backups prune, you're *guarenteed* of having a corrupt backup, even with it shut down.  TM will prune various parts of the virtual machine at different times, leaving you with a backup of different parts of different states.

[ColoradoMarmot]

For that case, just install a new version of OSX, and run the migration wizard.

[ShinySteelRobot]

Thanks for the replies.

Actually, once the backups prune, you're *guarenteed* of having a corrupt backup, even with it shut down.  TM will prune various parts of the virtual machine at different times, leaving you with a backup of different parts of different states.

I'm not sure I understand this. Once TM starts pruning it leaves the initial backup alone and prunes the intermediate deltas (worst case is that you have the original backup and a delta to the current state). How does that result in a guarateed corrupt backup?

I believe that TM corruptions occur, I just don't believe that it's guaranteed once pruning starts. As evidence, I've done a full TM restore from a backup that was pruned for weeks (possibly months) without a problem.

For that case, just install a new version of OSX, and run the migration wizard.

Ya, that's what I had to resort to (see post #2 above). Reinstalling OS X and then running the migration assistant in my experience isn't as reliable as a full restore from TM, since there are some things that the Migration Assistant fails to migrate. It's unfortunate that this is the only option we currently have.

If VMware can fix Fusion so we don't get the "You can't restore this backup because it was created by a different model of Mac" error within the client during TM restore, then things would be great. If they would fix the "stars" TM UI to work too, that would be awesome.

[ColoradoMarmot]

There are two issues - TM backups themselves become corrupted on a regular basis.  I just lost two here in my office, and my internal forums have had a half-dozen more this month - including one that was unrecoverable and was only discovered after a hard drive crash.  This is irrelvent to pruning.  One thing in particular that appears to have a high chance of corrupting a backup is to sleep the machine while a backup is active via wi-fi, but there are clearly others (I lost mine on a hardwire connection). There are others, but in the end, the structure that time machine uses internally (tons of links) is highly prone to corruption, and repairs are almost always fruitless.

As for corrupting backups of VM's, that's where pruning comes in (and has nothing to do with the corruption I talk about above).   There's two kinds of pruning - the dropping of hourly/daily backups and only keeping weekly (for example), and then pruning the oldest backup when it runs out of space.  In the latter case, it will prune the oldest files.  In the former, it prunes out of the middle.

Remember that a virtual machine isn't a single file, it's a collection of files.  So if TM needs space, it will prune individual files out of the bundle, rather than the entire thing, so you end up with an inconsistent backup.  For example, if it needs 2GB of space, it'll whack one of the disk segment backups while leaving the others (until the next time it needs space).  So you only have a partial virtual disk on that oldest backup.  You can partially mitigigate that by using a single disk file, but the supporting files (e.g. vmx) may then get out of date.  Again though, since that single file is touched every time Fusion is open (and continually while it is), you'd be backing up the entire disk file every hour your VM was running.  And how are you going to know which backups were made when the VM was shut down versus live? 

There is software that can do this, but it's far more advanced (and far more expensive) than time machine.  The same issue exists for when editing large video files, or collection of files.  Either you backup one big 2GB file every hour while it's changing, or you're backing up individual segments of an active project.  The result is the same - inconsistent data.  Much better to exclude them, and then snapshot them manually at appropriate intervals.

For the internal restore, I'm guessing that you've run into an apple issue with the hardware identifer - a virtual machine doesn't actually appear to be a Mac to the OS.  They will probably have to file a bug with Apple, though I would guess that's not high on Apple's priority list.

The question is, why do you need hourly or daily backups of the entire VM?  Can't you just go back to a weekly backup that was done manually, and then use time machine internally to restore the relevant data files that changed in the iterim?

[ShinySteelRobot]

Remember that a virtual machine isn't a single file, it's a collection of files.

Ah I see, I think we were talking about two different things. I'm only talking about performing a TM backup within a virtualized Lion client (not performing a TM backup on the *vmdisk files that VMware uses).

For the internal restore, I'm guessing that you've run into an apple issue with the hardware identifer - a virtual machine doesn't actually appear to be a Mac to the OS.  They will probably have to file a bug with Apple, though I would guess that's not high on Apple's priority list.

I suspect that's partially true. According to the System Profiler, Fusion presents an OS X client thusly:

Model Name: Mac

Model Identifier: VMware7,1

At client boot time and when the Lion recovery utility is running, Fusion isn't presenting the virtual client as Mac/VMware7,1, inexplicably. So I think it's acutally a bug in Fusion.

The question, to my mind, is... Why does Fusion present itself as one kind of Mac at startup / recovery mode, and a different kind of Mac when the virtual clients performs a TM backup?

The question is, why do you need hourly or daily backups of the entire VM?  Can't you just go back to a weekly backup that was done manually, and then use time machine internally to restore the relevant data files that changed in the iterim?

Unfortunately the "stars" UI that you use to restore individual files from within TM doesn't work in a virtualized Lion client. It freezes the client, forcing a reboot of the client.

[dariusd]

Hi ShinySteelRobot,

We always identify the virtual client as Model ID VMware7,1 (unless steps are taken to change the VM configuration to do otherwise)... we certainly don't intentionally change the reported Model ID of our own volition.  To check this in your environment, go into the guest's Lion Recovery Environment, choose Utilities > Terminal, and run:

   system_profiler SPHardwareDataType

and it should show that it's still a VMware7,1 "Mac" when booted this way.

I've started a Time Machine backup inside a VM here and I'll see if I can reproduce and troubleshoot both the problems you reported.  It's entirely possible that Time Machine is just getting confused by our Model ID and throwing an error message that isn't accurate, but there is no shortage of other things that can go wrong...

Thanks,

--

Darius

[dariusd · Accepted solution]

Following up on this...

The "stars" Time Machine interface certainly doesn't work right.  A scarce few parts do work, but extremely slowly, it seems -- i.e. wait a minute or three, and don't expect much to appear in the way of starry background or usable controls.  Pressing Esc or closing the foremost window seems to close Time Machine and eventually revive the OS, in my testing at least, but, again, you might have to wait for a minute or more before things come back to life...  This is almost certainly a pathological victim of the lack of a fully-accelerated Mac OS guest video driver.

When using the Lion Recovery environment to restore the Time Machine backup, I get the same "You can't restore this backup because it was created by a different model of Mac." as you reported.  The Installer Log (see Window > Installer Log) claims "Chosen snapshot determined incompatible based on board-id", which tells us where the problem is.  The board-id is loosely related to the Model ID, and it isn't shown anywhere in System Profiler... you can see your board-id by going into Terminal and running:

   ioreg -p IODeviceTree -r -n / -d 1

On a physical Mac, it'll be something like "Mac-12345678", and in a virtual machine, it'll be "440BX Desktop Reference Platform".  I'll file an internal bug here and see what we can do.

Cheers,

--

Darius

[ColoradoMarmot]

It does require 3d acceleration to work.  There's lots of things in mountain lion that will require 3d (which is why apple is dropping support for certain mac models).

Hopefully (fingers crossed) this means we might see some rudimentary level of 3d support in the next release of fusion 🙂

[ShinySteelRobot]

Darius Davis wrote:

When using the Lion Recovery environment to restore the Time Machine backup, I get the same "You can't restore this backup because it was created by a different model of Mac." as you reported.  The Installer Log (see Window > Installer Log) claims "Chosen snapshot determined incompatible based on board-id", which tells us where the problem is.  The board-id is loosely related to the Model ID, and it isn't shown anywhere in System Profiler... you can see your board-id by going into Terminal and running:

   ioreg -p IODeviceTree -r -n / -d 1


On a physical Mac, it'll be something like "Mac-12345678", and in a virtual machine, it'll be "440BX Desktop Reference Platform".  I'll file an internal bug here and see what we can do.

Wow, THANK YOU very much for following up on that. If you guys can get TM restores working within virtualized clients, then life will be grand.

Darius Davis wrote:

The "stars" Time Machine interface certainly doesn't work right.  A scarce few parts do work, but extremely slowly, it seems -- i.e. wait a minute or three, and don't expect much to appear in the way of starry background or usable controls.  Pressing Esc or closing the foremost window seems to close Time Machine and eventually revive the OS, in my testing at least, but, again, you might have to wait for a minute or more before things come back to life...  This is almost certainly a pathological victim of the lack of a fully-accelerated Mac OS guest video driver.

Ok that's good to know.  As dlhotka pointed out above, unaccellerated video may be a growing problem with future versions of OS X since they seem to be adding more and more video-accellerated bits into the OS. Maybe in Fusion 5.0?

[ShinySteelRobot]

When using the Lion Recovery environment to restore the Time Machine backup, I get the same "You can't restore this backup because it was created by a different model of Mac." as you reported.  ...<SNIP>... On a physical Mac, it'll be something like "Mac-12345678", and in a virtual machine, it'll be "440BX Desktop Reference Platform".  I'll file an internal bug here and see what we can do.

Anyone know if this is fixed in the newly released Fusion 5?

[dariusd]

Time Machine backups are still not restorable in Fusion 5 VMs.  We didn't have the opportunity to get this working right in time for release.  Sorry for the disappointment.

--

Darius