I have been doing some research on why VMware Fusion does not backup 'well' on TimeMachine.
I found that a fully 'shutdown' VM's back's up well and restores fine. Not that I have done this exhaustively but for the few times I have it is consistent.
VM's that were in a 'suspended' state do not backup correctly. Key files get missed.
It appears that only part of the backup is created therefore leaving certain vital parts missing. In my case critical vmdk 'split' files. My VM's VMDK's are split into smaller more manageable slices.
The VM Container contains the following files:
************----------------*************
mac:Windows 10 x64.vmwarevm iaitken$ ls -al
total 278515696
drwxrwxrwx@ 70 iaitken staff 2240 6 Feb 12:29 .
drwxr-xr-x 8 iaitken staff 256 4 Jul 2018 ..
drwxr-xr-x 67 iaitken staff 2144 11 Jan 16:59 Applications
-rw------- 1 iaitken staff 4083875840 6 Jul 2017 Virtual Disk-000001-s001.vmdk
-rw------- 1 iaitken staff 4261937152 6 Jul 2017 Virtual Disk-000001-s002.vmdk
-rw------- 1 iaitken staff 4256628736 6 Jul 2017 Virtual Disk-000001-s003.vmdk
-rw------- 1 iaitken staff 4260757504 6 Jul 2017 Virtual Disk-000001-s005.vmdk
-rw------- 1 iaitken staff 4261937152 6 Jul 2017 Virtual Disk-000001-s006.vmdk
-rw------- 1 iaitken staff 4258660352 6 Jul 2017 Virtual Disk-000001-s007.vmdk
-rw------- 1 iaitken staff 4261937152 6 Jul 2017 Virtual Disk-000001-s008.vmdk
-rw------- 1 iaitken staff 4261937152 6 Jul 2017 Virtual Disk-000001-s009.vmdk
-rw------- 1 iaitken staff 4261937152 6 Jul 2017 Virtual Disk-000001-s010.vmdk
-rw------- 1 iaitken staff 4261937152 6 Jul 2017 Virtual Disk-000001-s011.vmdk
-rw------- 1 iaitken staff 4261937152 6 Jul 2017 Virtual Disk-000001-s012.vmdk
-rw------- 1 iaitken staff 2646736896 6 Jul 2017 Virtual Disk-000001-s013.vmdk
-rw------- 1 iaitken staff 524288 30 Mar 2016 Virtual Disk-000001-s014.vmdk
-rw------- 1 iaitken staff 524288 30 Mar 2016 Virtual Disk-000001-s015.vmdk
-rw------- 1 iaitken staff 65536 30 Mar 2016 Virtual Disk-000001-s016.vmdk
-rw-------@ 1 iaitken staff 1134 6 Jul 2017 Virtual Disk-000001.vmdk
-rw------- 1 iaitken staff 4019650560 3 Feb 13:10 Virtual Disk-000002-s001.vmdk
-rw------- 1 iaitken staff 4207214592 3 Feb 13:10 Virtual Disk-000002-s002.vmdk
-rw------- 1 iaitken staff 3918725120 3 Feb 13:09 Virtual Disk-000002-s003.vmdk
-rw------- 1 iaitken staff 3766353920 3 Feb 13:10 Virtual Disk-000002-s004.vmdk
-rw------- 1 iaitken staff 3722051584 3 Feb 13:09 Virtual Disk-000002-s005.vmdk
-rw------- 1 iaitken staff 4149018624 3 Feb 13:10 Virtual Disk-000002-s006.vmdk
-rw------- 1 iaitken staff 4204593152 3 Feb 13:10 Virtual Disk-000002-s007.vmdk
-rw------- 1 iaitken staff 4202496000 3 Feb 12:43 Virtual Disk-000002-s008.vmdk
-rw------- 1 iaitken staff 4227596288 3 Feb 13:10 Virtual Disk-000002-s009.vmdk
-rw------- 1 iaitken staff 4147249152 3 Feb 13:09 Virtual Disk-000002-s010.vmdk
-rw------- 1 iaitken staff 3912695808 3 Feb 13:09 Virtual Disk-000002-s011.vmdk
-rw------- 1 iaitken staff 4155703296 3 Feb 13:09 Virtual Disk-000002-s012.vmdk
-rw------- 1 iaitken staff 4204199936 3 Feb 13:10 Virtual Disk-000002-s013.vmdk
-rw------- 1 iaitken staff 4261937152 3 Feb 13:09 Virtual Disk-000002-s014.vmdk
-rw------- 1 iaitken staff 4261937152 3 Feb 13:09 Virtual Disk-000002-s015.vmdk
-rw------- 1 iaitken staff 502333440 3 Feb 11:55 Virtual Disk-000002-s016.vmdk
-rw-------@ 1 iaitken staff 1119 3 Feb 10:02 Virtual Disk-000002.vmdk
-rw------- 1 iaitken staff 3716415488 8 Jan 2017 Virtual Disk-s001.vmdk
-rw------- 1 iaitken staff 4204396544 8 Jan 2017 Virtual Disk-s002.vmdk
-rw------- 1 iaitken staff 3838377984 8 Jan 2017 Virtual Disk-s003.vmdk
-rw------- 1 iaitken staff 4260364288 8 Jan 2017 Virtual Disk-s004.vmdk
-rw------- 1 iaitken staff 3019243520 8 Jan 2017 Virtual Disk-s005.vmdk
-rw------- 1 iaitken staff 4261937152 30 Mar 2016 Virtual Disk-s007.vmdk
-rw------- 1 iaitken staff 1417543680 16 Mar 2016 Virtual Disk-s009.vmdk
-rw------- 1 iaitken staff 524288 1 Feb 2016 Virtual Disk-s010.vmdk
-rw------- 1 iaitken staff 524288 1 Feb 2016 Virtual Disk-s011.vmdk
-rw------- 1 iaitken staff 524288 1 Feb 2016 Virtual Disk-s012.vmdk
-rw------- 1 iaitken staff 524288 1 Feb 2016 Virtual Disk-s013.vmdk
-rw------- 1 iaitken staff 524288 1 Feb 2016 Virtual Disk-s014.vmdk
-rw------- 1 iaitken staff 524288 1 Feb 2016 Virtual Disk-s015.vmdk
-rw------- 1 iaitken staff 131072 2 Feb 2016 Virtual Disk-s016.vmdk
-rw-------@ 1 iaitken staff 1161 30 Mar 2016 Virtual Disk.vmdk
-rw------- 1 iaitken staff 2147483648 3 Feb 13:11 Windows 10 x64-10e441de.vmem
-rw------- 1 iaitken staff 2544048 3 Feb 13:10 Windows 10 x64-10e441de.vmss
-rw------- 1 iaitken staff 2147483648 30 Mar 2016 Windows 10 x64-Snapshot1.vmem
-rw------- 1 iaitken staff 1761634 8 Jan 2017 Windows 10 x64-Snapshot1.vmsn
-rw------- 1 iaitken staff 2147483648 6 Jul 2017 Windows 10 x64-Snapshot2.vmem
-rw------- 1 iaitken staff 2869952 6 Jul 2017 Windows 10 x64-Snapshot2.vmsn
-rw------- 1 iaitken staff 8684 3 Feb 10:08 Windows 10 x64.nvram
-rw-r--r-- 1 iaitken staff 935 6 Feb 12:12 Windows 10 x64.plist
-rw-r--r-- 1 iaitken staff 792 6 Jul 2017 Windows 10 x64.vmsd
-rwxr-xr-x 1 iaitken staff 6513 3 Feb 13:10 Windows 10 x64.vmx
-rw-r--r-- 1 iaitken staff 372 15 Oct 06:59 Windows 10 x64.vmxf
drwxrwxrwx 7 iaitken staff 224 3 Feb 2016 appListCache
drwxr-xr-x 3 iaitken staff 96 6 Feb 09:41 caches
-rw-r--r-- 1 iaitken staff 0 6 Feb 12:10 quicklook-cache.png
-rw-r--r-- 1 iaitken staff 834999 3 Feb 13:10 startMenu.plist
-rw-r--r-- 1 iaitken staff 345117 14 Jan 16:54 vmware-0.log
-rw-r--r-- 1 iaitken staff 471050 13 Jan 13:39 vmware-1.log
-rw-r--r-- 1 iaitken staff 682285 12 Jan 08:53 vmware-2.log
-rw-r--r-- 1 iaitken staff 708294 3 Feb 13:11 vmware.log
************----------------*************
The VMDK file "Virtual Disk-000001.vmdk" contains the links to the splits:
************----------------*************
# Disk DescriptorFile
version=1
encoding="UTF-8"
CID=531b0359
parentCID=c7a883ae
isNativeSnapshot="no"
createType="twoGbMaxExtentSparse"
parentFileNameHint="Virtual Disk.vmdk"
# Extent description
RW 8323072 SPARSE "Virtual Disk-000001-s001.vmdk"
RW 8323072 SPARSE "Virtual Disk-000001-s002.vmdk"
RW 8323072 SPARSE "Virtual Disk-000001-s003.vmdk"
RW 8323072 SPARSE "Virtual Disk-000001-s004.vmdk"
RW 8323072 SPARSE "Virtual Disk-000001-s005.vmdk"
RW 8323072 SPARSE "Virtual Disk-000001-s006.vmdk"
RW 8323072 SPARSE "Virtual Disk-000001-s007.vmdk"
RW 8323072 SPARSE "Virtual Disk-000001-s008.vmdk"
RW 8323072 SPARSE "Virtual Disk-000001-s009.vmdk"
RW 8323072 SPARSE "Virtual Disk-000001-s010.vmdk"
RW 8323072 SPARSE "Virtual Disk-000001-s011.vmdk"
RW 8323072 SPARSE "Virtual Disk-000001-s012.vmdk"
RW 8323072 SPARSE "Virtual Disk-000001-s013.vmdk"
RW 8323072 SPARSE "Virtual Disk-000001-s014.vmdk"
RW 8323072 SPARSE "Virtual Disk-000001-s015.vmdk"
RW 983040 SPARSE "Virtual Disk-000001-s016.vmdk"
# The Disk Data Base
#DDB
ddb.longContentID = "554da9a833d25d44076aac91531b0359"
ddb.toolsInstallType = "1"
ddb.toolsVersion = "10278"
************----------------*************
Unfortunately, In my case, the container holding all of the VM data is missing the file "Virtual Disk-000001-s004.vmdk". See the directory listing.
The VMDK file "Virtual Disk-000002.vmdk" contains the following links:
************----------------*************
# Disk DescriptorFile
version=1
encoding="UTF-8"
CID=67253c12
parentCID=531b0359
createType="twoGbMaxExtentSparse"
parentFileNameHint="Virtual Disk-000001.vmdk"
# Extent description
RW 8323072 SPARSE "Virtual Disk-000002-s001.vmdk"
RW 8323072 SPARSE "Virtual Disk-000002-s002.vmdk"
RW 8323072 SPARSE "Virtual Disk-000002-s003.vmdk"
RW 8323072 SPARSE "Virtual Disk-000002-s004.vmdk"
RW 8323072 SPARSE "Virtual Disk-000002-s005.vmdk"
RW 8323072 SPARSE "Virtual Disk-000002-s006.vmdk"
RW 8323072 SPARSE "Virtual Disk-000002-s007.vmdk"
RW 8323072 SPARSE "Virtual Disk-000002-s008.vmdk"
RW 8323072 SPARSE "Virtual Disk-000002-s009.vmdk"
RW 8323072 SPARSE "Virtual Disk-000002-s010.vmdk"
RW 8323072 SPARSE "Virtual Disk-000002-s011.vmdk"
RW 8323072 SPARSE "Virtual Disk-000002-s012.vmdk"
RW 8323072 SPARSE "Virtual Disk-000002-s013.vmdk"
RW 8323072 SPARSE "Virtual Disk-000002-s014.vmdk"
RW 8323072 SPARSE "Virtual Disk-000002-s015.vmdk"
RW 983040 SPARSE "Virtual Disk-000002-s016.vmdk"
# The Disk Data Base
#DDB
ddb.longContentID = "06c3743b21d9de207bef021767253c12"
ddb.toolsInstallType = "1"
ddb.toolsVersion = "10309"
************----------------*************
The container has all the splits for this VMDK and Fusion recognises this as a ready to mount disk.
The VMDK file "Virtual Disk.vmdk" contains the following links:
************----------------*************
# Disk DescriptorFile
version=1
encoding="UTF-8"
CID=c7a883ae
parentCID=ffffffff
isNativeSnapshot="no"
createType="twoGbMaxExtentSparse"
# Extent description
RW 8323072 SPARSE "Virtual Disk-s001.vmdk"
RW 8323072 SPARSE "Virtual Disk-s002.vmdk"
RW 8323072 SPARSE "Virtual Disk-s003.vmdk"
RW 8323072 SPARSE "Virtual Disk-s004.vmdk"
RW 8323072 SPARSE "Virtual Disk-s005.vmdk"
RW 8323072 SPARSE "Virtual Disk-s006.vmdk"
RW 8323072 SPARSE "Virtual Disk-s007.vmdk"
RW 8323072 SPARSE "Virtual Disk-s008.vmdk"
RW 8323072 SPARSE "Virtual Disk-s009.vmdk"
RW 8323072 SPARSE "Virtual Disk-s010.vmdk"
RW 8323072 SPARSE "Virtual Disk-s011.vmdk"
RW 8323072 SPARSE "Virtual Disk-s012.vmdk"
RW 8323072 SPARSE "Virtual Disk-s013.vmdk"
RW 8323072 SPARSE "Virtual Disk-s014.vmdk"
RW 8323072 SPARSE "Virtual Disk-s015.vmdk"
RW 983040 SPARSE "Virtual Disk-s016.vmdk"
# The Disk Data Base
#DDB
ddb.adapterType = "lsilogic"
ddb.geometry.cylinders = "7832"
ddb.geometry.heads = "255"
ddb.geometry.sectors = "63"
ddb.longContentID = "f25b73ead7f87545466076efc7a883ae"
ddb.toolsVersion = "9508"
ddb.uuid = "60 00 C2 90 18 d9 11 b3-19 a2 31 ae e8 8a 30 ce"
ddb.virtualHWVersion = "11"
************----------------*************
As the directory listing shows I am missing the following splits:
Virtual Disk-s006.vmdk
Virtual Disk-s008.vmdk
Both VMDK's for "Virtual Disk.vmdk" and "Virtual Disk-000001.vmdk" are 'greyed out' in the Fusion app indicating the Fusion app check that all the splits are all present and correct.
So why does Time Machine not backup split files "Virtual Disk-000001-s004.vmdk", "Virtual Disk-s006.vmdk" and "Virtual Disk-s008.vmdk"?
I would suggest that as Time Machine uses 'snapshots' to take an image of a file, that the files in question are in a state where a snapshop cannot be taken. I suspect the files are 'locked'. So why is the file in that 'locked' state? Simply put that is a question for VMware to answer as I would suspect that if the VM or Fusion is shut down fully all file locks should be removed. Would this fix the problem?
I would like to also add that Time Machine should state that the backup is 'not' complete in situations where it cannot obtain a file lock and take a snapshot. Poor form Apple. Don't tell us the backup complete when it clearly is not.
Hi,
Sorry to hear about your failed backup, I'm afraid it happens quite often.
Time Machine will rotate out older files when it needs more free space. If your disk slice hasn't changed for a while then it might become "ready to discard" and you end up with a broken backup.
VMware recommends against using Time Machine for backing up virtual machines, there are more reasons as just this, see also:
https://kb.vmware.com/kb/1013628
or if you want to read about all the "why not to use TM for virtual machine backups" read my article at:
https://www.vimalin.com/why-is-time-machine-not-a-good-backup-for-virtual-machines/
(edit) disclaimer: I have authored a backup program designed for backing up VMware Fusion / VMware Workstation virtual machines which can be found at https://vimalin.com
hope this helps,
--
Wil
And let me echo those comments. While there are circumstances in which you will get lucky with a time machine backup, all too often, you can't restore in a critical situation.
Every hour when a VM is running you'll get the entire disk backed up. That will fill the target disk quickly, causing old backups to age out, creating the broken backup situation even more rapidly. Either use a package like his, or back up the VM's manually. Time machine simply is not a reliable option.
Thanks Wil,
I have read this as well and I agree it (TM Backups with VMware) is partially broken.
Notwithstanding this, I still think energy should be expended so a fix can be found.
In the event where a split has not been modified for longer than the maximum age of the TM backup and the backup store dumps the file as part of it max backup size routines would indicate the issue is with TM and as such Apple should do something. This would be a major bug in TM and as such many many problems would have surfaced outside fo the VMware TM combination. I suspect this is not the case as I have a lot of files that have not been modified for years and my latest backup has a copy. Again I surmise this is a file lock issue where the snapshot (used by TM) cannot be taken for some files within the VM container of Suspended VM's. It is time to look at the root cause and fix it. Knowing something is broken and not doing anything to fix it is very defeating.
I should add that TM just works and I have had great success restoring fully shutdown VM's. While this testing is not exhaustive I think it is enough for VMware to investigate further. I have a case open with them and hope this can be worked on soon.
I will have a look at your software shortly. Thank you for the suggestion. I need another solution for now even if VMware can fix this in time.
Kind regards
Hi,
On macOS you can copy files that are open for write, "no problem".
Easy enough to test, open a VM, run it and hit duplicate in Finder while highlighting the bundle.
It will copy the VM, all files are getting duplicated. But as the files are open for write there are no guarantees you can use that copy.
In that case you are copying files locally, so it is fast.
With TM backups the files are copied over a network link, so the disk will change over time and chances the backup is actually usable is much lower.
If a VM is suspended, no files are open by VMware Fusion and you can copy it, as long as you use it again on the same hardware and as long as there haven't been CPU/GPU errata updates.
As mentioned in the article, from what I understand the TM algorithm is age and file size.
So when it needs more space it will first delete larger files that are a bit older.
I might be wrong though, but this is a problem that has existed since VMware Fusion 1, VMware has looked into it before and AFAIK it is not fixable on their end.
FWIW, I use TM myself and like it a lot.
But I do not trust it with my virtual machine backups.
--
Wil
On APFS volumes, time machine first creates a snapshot of the drive, then backs it up asynchronously to the external volume. Once snapshotted, it works on a file by file basis, not on a 'bundle' basis, because it doesn't have an understanding of the overarching application file structure.
There is no way to reliably back up a running virtual machine, outside of running a backup process *inside* the guest. That's basic computing 101 - backing up open files may or may not work because of transient writes. Notice I said 'reliably' - with snapshots it may work, but you can't guarantee that it will. And on non APFS disks, all bets are off as there's no snapshotting involved.
But there's a second risk: because of the age-out function, different portions of the VMDK file can be aged out at different times. Time Machine has no application-level awareness - it just operates on the file level. You may have luck restoring shut down virtual machines, but if one of the bands of the virtual disk was aged out and deleted, you'll end up with a broken machine. And since if you have time machine turned on for a virtual machine, it's going to churn through disk space - an 80GB VM running for 8 hours will generate 640GB of backups, which makes the age-out probability much higher.
It's not a VMWare problem, nor an Apple problem - it's the interaction between two incompatible use cases. You'll need to find another way to backup your virtual machines. Wila's application is a great option for that. I myself use Carbon Copy Cloner to take periodic complete backups of my machine - far far better for full restores than time machine (which is not a complete machine backup BTW), including the virtual machines. But when I've done a lot of VM work, like after patch Tuesday, my backup approach is to shut them down and use finder to copy to an external backup.