5 Replies Latest reply on Feb 10, 2019 9:41 AM by dlhotka

    Bug found in Fusion. TimeMachine Backup failure on VMDK splits

    iaitken Lurker

      I have been doing some research on why VMware Fusion does not backup 'well' on TimeMachine.

       

      I found that a fully 'shutdown' VM's back's up well and restores fine. Not that I have done this exhaustively but for the few times I have it is consistent.

       

      VM's that were in a 'suspended' state do not backup correctly. Key files get missed.

       

      It appears that only part of the backup is created therefore leaving certain vital parts missing. In my case critical vmdk 'split' files. My VM's VMDK's are split into smaller more manageable slices.

       

      The VM Container contains the following files:

       

      ************----------------*************

      mac:Windows 10 x64.vmwarevm iaitken$ ls -al

      total 278515696

      drwxrwxrwx@ 70 iaitken  staff        2240  6 Feb 12:29 .

      drwxr-xr-x   8 iaitken  staff         256  4 Jul  2018 ..

      drwxr-xr-x  67 iaitken  staff        2144 11 Jan 16:59 Applications

      -rw-------   1 iaitken  staff  4083875840  6 Jul  2017 Virtual Disk-000001-s001.vmdk

      -rw-------   1 iaitken  staff  4261937152  6 Jul  2017 Virtual Disk-000001-s002.vmdk

      -rw-------   1 iaitken  staff  4256628736  6 Jul  2017 Virtual Disk-000001-s003.vmdk

      -rw-------   1 iaitken  staff  4260757504  6 Jul  2017 Virtual Disk-000001-s005.vmdk

      -rw-------   1 iaitken  staff  4261937152  6 Jul  2017 Virtual Disk-000001-s006.vmdk

      -rw-------   1 iaitken  staff  4258660352  6 Jul  2017 Virtual Disk-000001-s007.vmdk

      -rw-------   1 iaitken  staff  4261937152  6 Jul  2017 Virtual Disk-000001-s008.vmdk

      -rw-------   1 iaitken  staff  4261937152  6 Jul  2017 Virtual Disk-000001-s009.vmdk

      -rw-------   1 iaitken  staff  4261937152  6 Jul  2017 Virtual Disk-000001-s010.vmdk

      -rw-------   1 iaitken  staff  4261937152  6 Jul  2017 Virtual Disk-000001-s011.vmdk

      -rw-------   1 iaitken  staff  4261937152  6 Jul  2017 Virtual Disk-000001-s012.vmdk

      -rw-------   1 iaitken  staff  2646736896  6 Jul  2017 Virtual Disk-000001-s013.vmdk

      -rw-------   1 iaitken  staff      524288 30 Mar  2016 Virtual Disk-000001-s014.vmdk

      -rw-------   1 iaitken  staff      524288 30 Mar  2016 Virtual Disk-000001-s015.vmdk

      -rw-------   1 iaitken  staff       65536 30 Mar  2016 Virtual Disk-000001-s016.vmdk

      -rw-------@  1 iaitken  staff        1134  6 Jul  2017 Virtual Disk-000001.vmdk

      -rw-------   1 iaitken  staff  4019650560  3 Feb 13:10 Virtual Disk-000002-s001.vmdk

      -rw-------   1 iaitken  staff  4207214592  3 Feb 13:10 Virtual Disk-000002-s002.vmdk

      -rw-------   1 iaitken  staff  3918725120  3 Feb 13:09 Virtual Disk-000002-s003.vmdk

      -rw-------   1 iaitken  staff  3766353920  3 Feb 13:10 Virtual Disk-000002-s004.vmdk

      -rw-------   1 iaitken  staff  3722051584  3 Feb 13:09 Virtual Disk-000002-s005.vmdk

      -rw-------   1 iaitken  staff  4149018624  3 Feb 13:10 Virtual Disk-000002-s006.vmdk

      -rw-------   1 iaitken  staff  4204593152  3 Feb 13:10 Virtual Disk-000002-s007.vmdk

      -rw-------   1 iaitken  staff  4202496000  3 Feb 12:43 Virtual Disk-000002-s008.vmdk

      -rw-------   1 iaitken  staff  4227596288  3 Feb 13:10 Virtual Disk-000002-s009.vmdk

      -rw-------   1 iaitken  staff  4147249152  3 Feb 13:09 Virtual Disk-000002-s010.vmdk

      -rw-------   1 iaitken  staff  3912695808  3 Feb 13:09 Virtual Disk-000002-s011.vmdk

      -rw-------   1 iaitken  staff  4155703296  3 Feb 13:09 Virtual Disk-000002-s012.vmdk

      -rw-------   1 iaitken  staff  4204199936  3 Feb 13:10 Virtual Disk-000002-s013.vmdk

      -rw-------   1 iaitken  staff  4261937152  3 Feb 13:09 Virtual Disk-000002-s014.vmdk

      -rw-------   1 iaitken  staff  4261937152  3 Feb 13:09 Virtual Disk-000002-s015.vmdk

      -rw-------   1 iaitken  staff   502333440  3 Feb 11:55 Virtual Disk-000002-s016.vmdk

      -rw-------@  1 iaitken  staff        1119  3 Feb 10:02 Virtual Disk-000002.vmdk

      -rw-------   1 iaitken  staff  3716415488  8 Jan  2017 Virtual Disk-s001.vmdk

      -rw-------   1 iaitken  staff  4204396544  8 Jan  2017 Virtual Disk-s002.vmdk

      -rw-------   1 iaitken  staff  3838377984  8 Jan  2017 Virtual Disk-s003.vmdk

      -rw-------   1 iaitken  staff  4260364288  8 Jan  2017 Virtual Disk-s004.vmdk

      -rw-------   1 iaitken  staff  3019243520  8 Jan  2017 Virtual Disk-s005.vmdk

      -rw-------   1 iaitken  staff  4261937152 30 Mar  2016 Virtual Disk-s007.vmdk

      -rw-------   1 iaitken  staff  1417543680 16 Mar  2016 Virtual Disk-s009.vmdk

      -rw-------   1 iaitken  staff      524288  1 Feb  2016 Virtual Disk-s010.vmdk

      -rw-------   1 iaitken  staff      524288  1 Feb  2016 Virtual Disk-s011.vmdk

      -rw-------   1 iaitken  staff      524288  1 Feb  2016 Virtual Disk-s012.vmdk

      -rw-------   1 iaitken  staff      524288  1 Feb  2016 Virtual Disk-s013.vmdk

      -rw-------   1 iaitken  staff      524288  1 Feb  2016 Virtual Disk-s014.vmdk

      -rw-------   1 iaitken  staff      524288  1 Feb  2016 Virtual Disk-s015.vmdk

      -rw-------   1 iaitken  staff      131072  2 Feb  2016 Virtual Disk-s016.vmdk

      -rw-------@  1 iaitken  staff        1161 30 Mar  2016 Virtual Disk.vmdk

      -rw-------   1 iaitken  staff  2147483648  3 Feb 13:11 Windows 10 x64-10e441de.vmem

      -rw-------   1 iaitken  staff     2544048  3 Feb 13:10 Windows 10 x64-10e441de.vmss

      -rw-------   1 iaitken  staff  2147483648 30 Mar  2016 Windows 10 x64-Snapshot1.vmem

      -rw-------   1 iaitken  staff     1761634  8 Jan  2017 Windows 10 x64-Snapshot1.vmsn

      -rw-------   1 iaitken  staff  2147483648  6 Jul  2017 Windows 10 x64-Snapshot2.vmem

      -rw-------   1 iaitken  staff     2869952  6 Jul  2017 Windows 10 x64-Snapshot2.vmsn

      -rw-------   1 iaitken  staff        8684  3 Feb 10:08 Windows 10 x64.nvram

      -rw-r--r--   1 iaitken  staff         935  6 Feb 12:12 Windows 10 x64.plist

      -rw-r--r--   1 iaitken  staff         792  6 Jul  2017 Windows 10 x64.vmsd

      -rwxr-xr-x   1 iaitken  staff        6513  3 Feb 13:10 Windows 10 x64.vmx

      -rw-r--r--   1 iaitken  staff         372 15 Oct 06:59 Windows 10 x64.vmxf

      drwxrwxrwx   7 iaitken  staff         224  3 Feb  2016 appListCache

      drwxr-xr-x   3 iaitken  staff          96  6 Feb 09:41 caches

      -rw-r--r--   1 iaitken  staff           0  6 Feb 12:10 quicklook-cache.png

      -rw-r--r--   1 iaitken  staff      834999  3 Feb 13:10 startMenu.plist

      -rw-r--r--   1 iaitken  staff      345117 14 Jan 16:54 vmware-0.log

      -rw-r--r--   1 iaitken  staff      471050 13 Jan 13:39 vmware-1.log

      -rw-r--r--   1 iaitken  staff      682285 12 Jan 08:53 vmware-2.log

      -rw-r--r--   1 iaitken  staff      708294  3 Feb 13:11 vmware.log

      ************----------------*************

       

      The VMDK file "Virtual Disk-000001.vmdk" contains the links to the splits:

       

      ************----------------*************

      # Disk DescriptorFile

      version=1

      encoding="UTF-8"

      CID=531b0359

      parentCID=c7a883ae

      isNativeSnapshot="no"

      createType="twoGbMaxExtentSparse"

      parentFileNameHint="Virtual Disk.vmdk"

      # Extent description

      RW 8323072 SPARSE "Virtual Disk-000001-s001.vmdk"

      RW 8323072 SPARSE "Virtual Disk-000001-s002.vmdk"

      RW 8323072 SPARSE "Virtual Disk-000001-s003.vmdk"

      RW 8323072 SPARSE "Virtual Disk-000001-s004.vmdk"

      RW 8323072 SPARSE "Virtual Disk-000001-s005.vmdk"

      RW 8323072 SPARSE "Virtual Disk-000001-s006.vmdk"

      RW 8323072 SPARSE "Virtual Disk-000001-s007.vmdk"

      RW 8323072 SPARSE "Virtual Disk-000001-s008.vmdk"

      RW 8323072 SPARSE "Virtual Disk-000001-s009.vmdk"

      RW 8323072 SPARSE "Virtual Disk-000001-s010.vmdk"

      RW 8323072 SPARSE "Virtual Disk-000001-s011.vmdk"

      RW 8323072 SPARSE "Virtual Disk-000001-s012.vmdk"

      RW 8323072 SPARSE "Virtual Disk-000001-s013.vmdk"

      RW 8323072 SPARSE "Virtual Disk-000001-s014.vmdk"

      RW 8323072 SPARSE "Virtual Disk-000001-s015.vmdk"

      RW 983040 SPARSE "Virtual Disk-000001-s016.vmdk"

       

       

      # The Disk Data Base

      #DDB

       

       

      ddb.longContentID = "554da9a833d25d44076aac91531b0359"

      ddb.toolsInstallType = "1"

      ddb.toolsVersion = "10278"

      ************----------------*************

       

      Unfortunately, In my case, the container holding all of the VM data is missing the file "Virtual Disk-000001-s004.vmdk". See the directory listing.

       

      The VMDK file "Virtual Disk-000002.vmdk" contains the following links:

       

      ************----------------*************

      # Disk DescriptorFile

      version=1

      encoding="UTF-8"

      CID=67253c12

      parentCID=531b0359

      createType="twoGbMaxExtentSparse"

      parentFileNameHint="Virtual Disk-000001.vmdk"

      # Extent description

      RW 8323072 SPARSE "Virtual Disk-000002-s001.vmdk"

      RW 8323072 SPARSE "Virtual Disk-000002-s002.vmdk"

      RW 8323072 SPARSE "Virtual Disk-000002-s003.vmdk"

      RW 8323072 SPARSE "Virtual Disk-000002-s004.vmdk"

      RW 8323072 SPARSE "Virtual Disk-000002-s005.vmdk"

      RW 8323072 SPARSE "Virtual Disk-000002-s006.vmdk"

      RW 8323072 SPARSE "Virtual Disk-000002-s007.vmdk"

      RW 8323072 SPARSE "Virtual Disk-000002-s008.vmdk"

      RW 8323072 SPARSE "Virtual Disk-000002-s009.vmdk"

      RW 8323072 SPARSE "Virtual Disk-000002-s010.vmdk"

      RW 8323072 SPARSE "Virtual Disk-000002-s011.vmdk"

      RW 8323072 SPARSE "Virtual Disk-000002-s012.vmdk"

      RW 8323072 SPARSE "Virtual Disk-000002-s013.vmdk"

      RW 8323072 SPARSE "Virtual Disk-000002-s014.vmdk"

      RW 8323072 SPARSE "Virtual Disk-000002-s015.vmdk"

      RW 983040 SPARSE "Virtual Disk-000002-s016.vmdk"

       

       

      # The Disk Data Base

      #DDB

       

       

      ddb.longContentID = "06c3743b21d9de207bef021767253c12"

      ddb.toolsInstallType = "1"

      ddb.toolsVersion = "10309"

      ************----------------*************

       

      The container has all the splits for this VMDK and Fusion recognises this as a ready to mount disk.

       

      The VMDK file "Virtual Disk.vmdk" contains the following links:

       

      ************----------------*************

      # Disk DescriptorFile

      version=1

      encoding="UTF-8"

      CID=c7a883ae

      parentCID=ffffffff

      isNativeSnapshot="no"

      createType="twoGbMaxExtentSparse"

       

       

      # Extent description

      RW 8323072 SPARSE "Virtual Disk-s001.vmdk"

      RW 8323072 SPARSE "Virtual Disk-s002.vmdk"

      RW 8323072 SPARSE "Virtual Disk-s003.vmdk"

      RW 8323072 SPARSE "Virtual Disk-s004.vmdk"

      RW 8323072 SPARSE "Virtual Disk-s005.vmdk"

      RW 8323072 SPARSE "Virtual Disk-s006.vmdk"

      RW 8323072 SPARSE "Virtual Disk-s007.vmdk"

      RW 8323072 SPARSE "Virtual Disk-s008.vmdk"

      RW 8323072 SPARSE "Virtual Disk-s009.vmdk"

      RW 8323072 SPARSE "Virtual Disk-s010.vmdk"

      RW 8323072 SPARSE "Virtual Disk-s011.vmdk"

      RW 8323072 SPARSE "Virtual Disk-s012.vmdk"

      RW 8323072 SPARSE "Virtual Disk-s013.vmdk"

      RW 8323072 SPARSE "Virtual Disk-s014.vmdk"

      RW 8323072 SPARSE "Virtual Disk-s015.vmdk"

      RW 983040 SPARSE "Virtual Disk-s016.vmdk"

       

       

      # The Disk Data Base

      #DDB

       

       

      ddb.adapterType = "lsilogic"

      ddb.geometry.cylinders = "7832"

      ddb.geometry.heads = "255"

      ddb.geometry.sectors = "63"

      ddb.longContentID = "f25b73ead7f87545466076efc7a883ae"

      ddb.toolsVersion = "9508"

      ddb.uuid = "60 00 C2 90 18 d9 11 b3-19 a2 31 ae e8 8a 30 ce"

      ddb.virtualHWVersion = "11"

      ************----------------*************

       

      As the directory listing shows I am missing the following splits:

       

      Virtual Disk-s006.vmdk

      Virtual Disk-s008.vmdk

       

      Both VMDK's for "Virtual Disk.vmdk" and "Virtual Disk-000001.vmdk" are 'greyed out' in the Fusion app indicating the Fusion app check that all the splits are all present and correct.

       

      So why does Time Machine not backup split files "Virtual Disk-000001-s004.vmdk", "Virtual Disk-s006.vmdk" and "Virtual Disk-s008.vmdk"?

       

      I would suggest that as Time Machine uses 'snapshots' to take an image of a file, that the files in question are in a state where a snapshop cannot be taken. I suspect the files are 'locked'. So why is the file in that 'locked' state? Simply put that is a question for VMware to answer as I would suspect that if the VM or Fusion is shut down fully all file locks should be removed. Would this fix the problem?

       

      I would like to also add that Time Machine should state that the backup is 'not' complete in situations where it cannot obtain a file lock and take a snapshot. Poor form Apple. Don't tell us the backup complete when it clearly is not.

        • 1. Re: Bug found in Fusion. TimeMachine Backup failure on VMDK splits
          wila Guru
          Community WarriorsUser Moderators

          Hi,

           

          Sorry to hear about your failed backup, I'm afraid it happens quite often.

          Time Machine will rotate out older files when it needs more free space. If your disk slice hasn't changed for a while then it might become "ready to discard" and you end up with a broken backup.

          VMware recommends against using Time Machine for backing up virtual machines, there are more reasons as just this, see also:

           

          https://kb.vmware.com/kb/1013628

           

          or if you want to read about all the "why not to use TM for virtual machine backups" read my article at:

           

          https://www.vimalin.com/why-is-time-machine-not-a-good-backup-for-virtual-machines/

           

          (edit) disclaimer: I have authored a backup program designed for backing up VMware Fusion / VMware Workstation virtual machines which can be found at https://vimalin.com

           

          hope this helps,

          --

          Wil

          | Author of Vimalin. The virtual machine Backup app for VMware Desktop Products
          | Vimalin : Automated backups for VMware Fusion and VMware Workstation Professional
          | More info at https://www.vimalin.com
          | Twitter @wilva
          | VMware Wiki at http://www.vi-toolkit.com
          • 2. Re: Bug found in Fusion. TimeMachine Backup failure on VMDK splits
            dlhotka Virtuoso

            And let me echo those comments.  While there are circumstances in which you will get lucky with a time machine backup, all too often, you can't restore in a critical situation.

             

            Every hour when a VM is running you'll get the entire disk backed up.  That will fill the target disk quickly, causing old backups to age out, creating the broken backup situation even more rapidly.  Either use a package like his, or back up the VM's manually.  Time machine simply is not a reliable option.

            • 3. Re: Bug found in Fusion. TimeMachine Backup failure on VMDK splits
              iaitken Lurker

              Thanks Wil,

               

              I have read this as well and I agree it (TM Backups with VMware) is partially broken.

               

              Notwithstanding this, I still think energy should be expended so a fix can be found.

               

              In the event where a split has not been modified for longer than the maximum age of the TM backup and the backup store dumps the file as part of it max backup size routines would indicate the issue is with TM and as such Apple should do something. This would be a major bug in TM and as such many many problems would have surfaced outside fo the VMware TM combination. I suspect this is not the case as I have a lot of files that have not been modified for years and my latest backup has a copy. Again I surmise this is a file lock issue where the snapshot (used by TM) cannot be taken for some files within the VM container of Suspended VM's. It is time to look at the root cause and fix it. Knowing something is broken and not doing anything to fix it is very defeating.

               

              I should add that TM just works and I have had great success restoring fully shutdown VM's. While this testing is not exhaustive I think it is enough for VMware to investigate further. I have a case open with them and hope this can be worked on soon.

               

              I will have a look at your software shortly. Thank you for the suggestion. I need another solution for now even if VMware can fix this in time.

               

              Kind regards

              • 4. Re: Bug found in Fusion. TimeMachine Backup failure on VMDK splits
                wila Guru
                Community WarriorsUser Moderators

                Hi,

                 

                On macOS you can copy files that are open for write, "no problem".

                Easy enough to test, open a VM, run it and hit duplicate in Finder while highlighting the bundle.

                It will copy the VM, all files are getting duplicated. But as the files are open for write there are no guarantees you can use that copy.

                In that case you are copying files locally, so it is fast.

                With TM backups the files are copied over a network link, so the disk will change over time and chances the backup is actually usable is much lower.

                If a VM is suspended, no files are open by VMware Fusion and you can copy it, as long as you use it again on the same hardware and as long as there haven't been CPU/GPU errata updates.

                 

                As mentioned in the article, from what I understand the TM algorithm is age and file size.

                So when it needs more space it will first delete larger files that are a bit older.

                I might be wrong though, but this is a problem that has existed since VMware Fusion 1, VMware has looked into it before and AFAIK it is not fixable on their end.

                 

                FWIW, I use TM myself and like it a lot.

                But I do not trust it with my virtual machine backups.

                 

                --

                Wil

                | Author of Vimalin. The virtual machine Backup app for VMware Desktop Products
                | Vimalin : Automated backups for VMware Fusion and VMware Workstation Professional
                | More info at https://www.vimalin.com
                | Twitter @wilva
                | VMware Wiki at http://www.vi-toolkit.com
                • 5. Re: Bug found in Fusion. TimeMachine Backup failure on VMDK splits
                  dlhotka Virtuoso

                  On APFS volumes, time machine first creates a snapshot of the drive, then backs it up asynchronously to the external volume.  Once snapshotted, it works on a file by file basis, not on a 'bundle' basis, because it doesn't have an understanding of the overarching application file structure.  

                   

                  There is no way to reliably back up a running virtual machine, outside of running a backup process *inside* the guest.  That's basic computing 101 - backing up open files may or may not work because of transient writes.  Notice I said 'reliably' -  with snapshots it may work, but you can't guarantee that it will.  And on non APFS disks, all bets are off as there's no snapshotting involved.

                   

                  But there's a second risk: because of the age-out function, different portions of the VMDK file can be aged out at different times.    Time Machine has no application-level awareness - it just operates on the file level.  You may have luck restoring shut down virtual machines, but if one of the bands of the virtual disk was aged out and deleted, you'll end up with a broken machine.  And since if you have time machine turned on for a virtual machine, it's going to churn through disk space - an 80GB VM running for 8 hours will generate 640GB of backups, which makes the age-out probability much higher.

                   

                  It's not a VMWare problem, nor an Apple problem - it's the interaction between two incompatible use cases.  You'll need to find another way to backup your virtual machines.  Wila's application is a great option for that.  I myself use Carbon Copy Cloner to take periodic complete backups of my machine - far far better for full restores than time machine (which is not a complete machine backup BTW), including the virtual machines.  But when I've done a lot of VM work, like after patch Tuesday, my backup approach is to shut them down and use finder to copy to an external backup.