1 2 Previous Next 27 Replies Latest reply on Dec 10, 2017 8:13 AM by EKroboter

    Cannot delete snaphsots, but can create new ones.

    EKroboter Novice

      First of all, I want to thank everyone here for all the help and resources, without all your help I wouldn't have been able to make the switch to ESXi.

       

      Now, some background before I dwell into my current issue. I'm the sysadmin for a medium sized company, our infrastructure uses two ESXi hosts (no vCenter, just two standalone hosts) with about half a dozen VMs on each (domain controllers, print server, UniFi controller, ESET Remote Adminitrator, SQL server, etc.). Both hosts are exactly alike (HPE ProLiant ML110 gen9's, 48 GB of RAM on each, a RAID1 for the main datastore plus an SSD for the swap datastore).

       

      I backup all the VMs about once a week using Veeam Backup & Replication Free Edition, VeeamZipping them into packages on external storage. This has worked without any problems so far.

      I don't usually take snapshots of the VMs except when making important changes, such as prior to installing a new SQL instance. When I do take snapshots though, I make sure to delete them and consolidate the disks after I confirm that everything is working. I don't want to store any cruft, I want clean and lean VMs.

       

      Yesterday I upgrade both hosts to ESXi 6.5U1 (from 6.5). Everything went OK, no error messages whatsoever. There was one oversight though: I forgot to delete some snapshots from one VM prior to the update.

      All the VMs work, except for one  in which I cannot remove any snapshots. Windows boots just fine and everything works, but I receive errors while trying to delete all the snapshots and when trying to consolidate the disks. I read about something regarding the CDROM drive that could impact the process if they're connected to an ISO image. It was, but the problem persisted even after removing the mount.

       

      I tried to create a new snapshot to see If it'd worked. It did; but I cannot delete it. I VeeamZipped the VM to see if I could, and I was able to back it up. Veeam did ended the job with a warning saying that it wasn't able to delete the temporary snapshot it created for the job, which I then confirmed in the Snapshot Manager in the ESXi UI.

       

      The message I received when trying to delete all snapshots is the following:

      Failed - A general system error occurred: vim.fault.GenericVmConfigFault

       

      And the one I get when trying to consolidate:

      Failed - Unable to access file since it is locked

      An error occurred while consolidating disks: One of the disks in this virtual machine is already in use by a virtual machine or by a snapshot.

       

      Things I tried so far:

      1. Tried removing the snapshots with the VM on and off. No luck.
      2. Rebooting the ESXi host with the troubled VM. No luck.
      3. Restarting my workstation, which runs VeeamZip, to see if it had anything to do with it. No luck.

       

      I read several articles online regarding this problem and quite frankly I'm a bit overwhelmed. I don't know where to start, everyone seems to be having this problem in different scenarios and setups, none of which apply to mine.

       

      Here are some screenshots if it's of any help:

       

      The VM in question:

      VM details.png

       

      The error popup:

      Error popup.png

       

      And the description:

      Error description.png

       

      The current snapshots:

      Snapshots.png

       

      Contents of the datastore:

      Datastore ls.png

       

      I will appreciate all the help I can get. Thank you

        • 1. Re: Cannot delete snaphsots, but can create new ones.
          daphnissov Virtuoso
          vExpert

          What is interesting about your screenshot is you appear to have 2 disks on this VM and 4 snapshots on one disk but only *3* on the other. First thing to check is to make sure neither of those drives (VMDKs) are mounted to any other VMs on that host. Second, if you haven't already, update your ESXi embedded host client to the latest version posted to the fling site. You can update your client directly from a web browser using instructions in the fling. Third, attempt a "delete all snapshots" once again and record the time you initiated that operation. Then, pull the vmware.log file from the VMs home directory and upload it to your thread.

          • 2. Re: Cannot delete snaphsots, but can create new ones.
            EKroboter Novice

            Yes, I noticed that as well. Hard Disk 1 has four vmdks and Hard Disk 2 only three. I can only assume this is because at one point, I tried to increase the size of the disk from 80 to 120 GB. The VM was turned off at the time. There was no change applied, I figured it was some UI issue.  Hard Disk 1 is still 80 GB, it's the C: drive in the VM containing the OS (thin provisiones) and Hard Disk 2 is a thick provisioned 120 GB second drive to host SQL databases.

             

            I did took a snapshot prior to try to expand the disk. I thought about reverting back to it but I assumed it’d fail as well, so I didn’t

             

            No other VM in the host has any vmdks from this VM mounted.

             

            The current Client version I have is 1.21.0. The Fling you mentioned is now at v1.24.

            Client version.png

             

            So, I downloaded the vib file, renamed to esxiui.vib (for faster typing), put in the /tmp directory and then did:

            esxcli software vib install -v /tmp/esxui.vib

             

            That returned the following result:

            Screen Shot 2017-12-09 at 17.01.44.png

            And now I'm on v1.24:

            Screen Shot 2017-12-09 at 17.02.53.png

             

            I then tried to delete all the snapshots one more time, and heres the VMware.log file that I downloaded immediately after from the VM home folder (/vmfs/volumes/58da71a5-afc838b0-2fb7-1c98ec52f2f8/EKR-SVR02-SQL)

            • 3. Re: Cannot delete snaphsots, but can create new ones.
              daphnissov Virtuoso
              vExpert

              In the home directory, perform ls -lah and post the output here.

              • 4. Re: Cannot delete snaphsots, but can create new ones.
                daphnissov Virtuoso
                vExpert

                Please also run the following command from the VM's home directory and attach the output file which will appear in the same directory named "vmdkChain".

                 

                find . -maxdepth 1 -iname '*.vmdk' -not -name '*-sesparse.vmdk' -not -name '*-flat.vmdk' -not -name '*-ctk.vmdk' -exec cat {} +>vmdkChain

                • 5. Re: Cannot delete snaphsots, but can create new ones.
                  EKroboter Novice

                  Here's the output of ls -lah:

                   

                  total 182033984

                  drwxr-xr-x    1 root     root       80.0K Dec  9 02:40 .

                  drwxr-xr-t    1 root     root       76.0K Nov 19 20:57 ..

                  -rw-------    1 root     root       31.7K Nov 18 13:35 EKR-SVR02-SQL-Snapshot23.vmsn

                  -rw-------    1 root     root       31.7K Dec  9 01:57 EKR-SVR02-SQL-Snapshot25.vmsn

                  -rw-------    1 root     root       31.7K Dec  9 02:05 EKR-SVR02-SQL-Snapshot26.vmsn

                  -rw-r--r--    1 root     root          13 Nov  3 19:20 EKR-SVR02-SQL-aux.xml

                  -rw-------    1 root     root        8.5K Dec  9 16:07 EKR-SVR02-SQL.nvram

                  -rw-------    1 root     root        1.6K Dec  9 02:05 EKR-SVR02-SQL.vmsd

                  -rwx------    1 root     root        4.0K Dec  9 14:46 EKR-SVR02-SQL.vmx

                  -rw-------    1 root     root           0 Dec  9 02:39 EKR-SVR02-SQL.vmx.lck

                  -rw-------    1 root     root        3.1K Dec  9 01:56 EKR-SVR02-SQL.vmxf

                  -rwx------    1 root     root        4.0K Dec  9 14:46 EKR-SVR02-SQL.vmx~

                  -rw-------    1 root     root        5.0M Dec  9 02:03 EKR-SVR02-SQL0-000001-ctk.vmdk

                  -rw-------    1 root     root        8.8G Dec  9 02:03 EKR-SVR02-SQL0-000001-sesparse.vmdk

                  -rw-------    1 root     root         481 Dec  9 01:58 EKR-SVR02-SQL0-000001.vmdk

                  -rw-------    1 root     root        5.0M Nov 18 13:34 EKR-SVR02-SQL0-000002-ctk.vmdk

                  -rw-------    1 root     root       80.0G Nov 18 13:34 EKR-SVR02-SQL0-000002-flat.vmdk

                  -rw-------    1 root     root         637 Nov 18 00:19 EKR-SVR02-SQL0-000002.vmdk

                  -rw-------    1 root     root        5.0M Dec  9 01:57 EKR-SVR02-SQL0-000003-ctk.vmdk

                  -rw-------    1 root     root      326.0M Dec  9 01:57 EKR-SVR02-SQL0-000003-sesparse.vmdk

                  -rw-------    1 root     root         427 Dec  9 01:57 EKR-SVR02-SQL0-000003.vmdk

                  -rw-------    1 root     root        5.0M Dec  9 02:40 EKR-SVR02-SQL0-000004-ctk.vmdk

                  -rw-------    1 root     root      537.2M Dec  9 22:06 EKR-SVR02-SQL0-000004-sesparse.vmdk

                  -rw-------    1 root     root         427 Dec  9 02:39 EKR-SVR02-SQL0-000004.vmdk

                  -rw-------    1 root     root        7.5M Dec  9 02:03 EKR-SVR02-SQL_1-000001-ctk.vmdk

                  -rw-------    1 root     root        2.2G Dec  9 02:03 EKR-SVR02-SQL_1-000001-sesparse.vmdk

                  -rw-------    1 root     root         477 Dec  9 01:58 EKR-SVR02-SQL_1-000001.vmdk

                  -rw-------    1 root     root        7.5M Dec  9 01:57 EKR-SVR02-SQL_1-000002-ctk.vmdk

                  -rw-------    1 root     root      487.0M Dec  9 01:57 EKR-SVR02-SQL_1-000002-sesparse.vmdk

                  -rw-------    1 root     root         430 Dec  9 01:57 EKR-SVR02-SQL_1-000002.vmdk

                  -rw-------    1 root     root        7.5M Dec  9 02:40 EKR-SVR02-SQL_1-000003-ctk.vmdk

                  -rw-------    1 root     root      536.0M Dec  9 22:06 EKR-SVR02-SQL_1-000003-sesparse.vmdk

                  -rw-------    1 root     root         430 Dec  9 02:40 EKR-SVR02-SQL_1-000003.vmdk

                  -rw-------    1 root     root        7.5M Nov 18 13:34 EKR-SVR02-SQL_1-ctk.vmdk

                  -rw-------    1 root     root      120.0G Nov 18 13:34 EKR-SVR02-SQL_1-flat.vmdk

                  -rw-------    1 root     root         599 Nov 18 00:19 EKR-SVR02-SQL_1.vmdk

                  -rw-------    1 root     root      462.8K Nov 17 23:41 vmware-27.log

                  -rw-------    1 root     root      318.8K Nov 18 13:34 vmware-28.log

                  -rw-------    1 root     root      540.8K Nov 25 12:20 vmware-29.log

                  -rw-------    1 root     root      431.0K Dec  8 14:53 vmware-30.log

                  -rw-------    1 root     root      397.1K Dec  9 01:56 vmware-31.log

                  -rw-------    1 root     root      324.5K Dec  9 02:03 vmware-32.log

                  -rw-------    1 root     root      526.9K Dec  9 20:28 vmware.log

                   

                  And attached is the output for find

                   

                  Hope I did everything right.

                  • 6. Re: Cannot delete snaphsots, but can create new ones.
                    daphnissov Virtuoso
                    vExpert

                    Ok, after looking at your log and your metadata chain, something has happened to the snapshot sequence IDs. It appears for each disk you have there is one snapshot that is not being actively referenced. For disk 0 this would be EKR-SVR02-SQL0-000003.vmdk (and the -flat and -ctk files that correspond) and for disk 1 this would be EKR-SVR02-SQL_1-000002.vmdk and its accompanying files. Both of these have date stamps of Dec 9 01:57. By looking at the log for the VM, disklib is not invoking these files but is for all the others.

                     

                    2017-12-09T20:12:51.309Z| vmx| I125: DISKLIB-VMFS  : "/vmfs/volumes/58da71a5-afc838b0-2fb7-1c98ec52f2f8/EKR-SVR02-SQL/EKR-SVR02-SQL0-000004-sesparse.vmdk" : open successful (21) size = 563277824, hd = 0. Type 19

                    2017-12-09T20:12:51.309Z| vmx| I125: DISKLIB-VMFS  : "/vmfs/volumes/58da71a5-afc838b0-2fb7-1c98ec52f2f8/EKR-SVR02-SQL/EKR-SVR02-SQL0-000004-sesparse.vmdk" : closed.

                    2017-12-09T20:12:51.310Z| vmx| I125: DISKLIB-VMFS  : "/vmfs/volumes/58da71a5-afc838b0-2fb7-1c98ec52f2f8/EKR-SVR02-SQL/EKR-SVR02-SQL0-000001-sesparse.vmdk" : open successful (21) size = 9495916544, hd = 0. Type 19

                    2017-12-09T20:12:51.310Z| vmx| I125: DISKLIB-VMFS  : "/vmfs/volumes/58da71a5-afc838b0-2fb7-1c98ec52f2f8/EKR-SVR02-SQL/EKR-SVR02-SQL0-000001-sesparse.vmdk" : closed.

                    2017-12-09T20:12:51.310Z| vmx| I125: DISKLIB-VMFS  : "/vmfs/volumes/58da71a5-afc838b0-2fb7-1c98ec52f2f8/EKR-SVR02-SQL/EKR-SVR02-SQL0-000002-flat.vmdk" : open successful (21) size = 85899345920, hd = 0. Type 3

                    2017-12-09T20:12:51.310Z| vmx| I125: DISKLIB-VMFS  : "/vmfs/volumes/58da71a5-afc838b0-2fb7-1c98ec52f2f8/EKR-SVR02-SQL/EKR-SVR02-SQL0-000002-flat.vmdk" : closed.

                    2017-12-09T20:12:51.311Z| vmx| I125: DISKLIB-VMFS  : "/vmfs/volumes/58da71a5-afc838b0-2fb7-1c98ec52f2f8/EKR-SVR02-SQL/EKR-SVR02-SQL_1-000003-sesparse.vmdk" : open successful (21) size = 562040832, hd = 0. Type 19

                    2017-12-09T20:12:51.311Z| vmx| I125: DISKLIB-VMFS  : "/vmfs/volumes/58da71a5-afc838b0-2fb7-1c98ec52f2f8/EKR-SVR02-SQL/EKR-SVR02-SQL_1-000003-sesparse.vmdk" : closed.

                    2017-12-09T20:12:51.311Z| vmx| I125: DISKLIB-VMFS  : "/vmfs/volumes/58da71a5-afc838b0-2fb7-1c98ec52f2f8/EKR-SVR02-SQL/EKR-SVR02-SQL_1-000001-sesparse.vmdk" : open successful (21) size = 2372390912, hd = 0. Type 19

                    2017-12-09T20:12:51.312Z| vmx| I125: DISKLIB-VMFS  : "/vmfs/volumes/58da71a5-afc838b0-2fb7-1c98ec52f2f8/EKR-SVR02-SQL/EKR-SVR02-SQL_1-000001-sesparse.vmdk" : closed.

                    2017-12-09T20:12:51.312Z| vmx| I125: DISKLIB-VMFS  : "/vmfs/volumes/58da71a5-afc838b0-2fb7-1c98ec52f2f8/EKR-SVR02-SQL/EKR-SVR02-SQL_1-flat.vmdk" : open successful (21) size = 128849018880, hd = 0. Type 3

                    2017-12-09T20:12:51.312Z| vmx| I125: DISKLIB-VMFS  : "/vmfs/volumes/58da71a5-afc838b0-2fb7-1c98ec52f2f8/EKR-SVR02-SQL/EKR-SVR02-SQL_1-flat.vmdk" : closed.

                     

                    Normally, in a healthy snapshot chain, all disks should be invoked in the reverse sequence ending with the base -flat extent file, but we don't see that with yours.

                     

                    When I look at your disk metadata which I had you generate with the last file, I can see these orphaned disks don't have valid references to anything else in the chain. What's also interesting is that they appear to have a forward reference to the next delta one minute in the future.

                     

                    I also see only three snapshot descriptors.

                     

                    -rw-------    1 root     root       31.7K Nov 18 13:35 EKR-SVR02-SQL-Snapshot23.vmsn
                    -rw-------    1 root     root       31.7K Dec  9 01:57 EKR-SVR02-SQL-Snapshot25.vmsn
                    -rw-------    1 root     root       31.7K Dec  9 02:05 EKR-SVR02-SQL-Snapshot26.vmsn
                    

                     

                     

                    And the Dec 9 01:57 time stamp appears for the errant descriptor as well. The following error appears in the log file related to this each time you try to commit.

                     

                    2017-12-09T20:12:51.312Z| vmx| I125: SNAPSHOT: SnapshotDiskTreeAddFromSnapshot: Trying to add snapshot EKR-SVR02-SQL-Snapshot26.vmsn to disk /vmfs/volumes/58da71a5-afc838b0-2fb7-1c98ec52f2f8/EKR-SVR02-SQL/EKR-SVR02-SQL0-000001.vmdk which already has snapshot EKR-SVR02-SQL-Snapshot25.vmsn.

                     

                     

                    So it seems, somehow, a snapshot got created but never was referenced by the chain and isn't referenced even now.

                     

                    Before proceeding, I know you said you had a VeeamZIP, but anytime you start messing with disks and their extents, you need to be positive you have a good backup.

                    Do not pass go and do not collect $200 if you think in any way, shape, or form that you do not have a good, solid backup.

                     

                    That said, if you do, let's see if it can correct itself. Delete EKR-SVR02-SQL-Snapshot25.vmsn first with rm -f EKR-SVR02-SQL-Snapshot25.vmsn.

                    The VMSN files are just metadata for the memory points, which, since you didn't capture the memory state in any of the snapshots, essentially have no data. Delete this file and attempt to delete all snapshots once again. If that fails, repeat the ls -lah and attach a new vmware.log file.

                    • 7. Re: Cannot delete snaphsots, but can create new ones.
                      daphnissov Virtuoso
                      vExpert

                      Also, I should have asked earlier, but please attach vmware-30, 31, and 32.log. I'd like to see what lead to this behavior.

                      • 8. Re: Cannot delete snaphsots, but can create new ones.
                        EKroboter Novice

                        Wow, that was extremely thorough of you. Thank you. I will try your suggestions and post back but here are the logs you requested.

                        As for the backups, I do have a recent VeeamZip of the VM that ended with a warning, not an error. That warning was simply stating that Veeam wasn't able to delete the snapshot afterwards.

                        As a precaution, I also did a full bare metal backup from within Windows. Just to be safe.

                        • 9. Re: Cannot delete snaphsots, but can create new ones.
                          EKroboter Novice

                          I deleted the Snapshot25.vmsn file, but the problem remains. Here's the new output and the vmware.log

                           

                          total 182051328

                          drwxr-xr-x    1 root     root       80.0K Dec  9 23:13 .

                          drwxr-xr-t    1 root     root       76.0K Nov 19 20:57 ..

                          -rw-------    1 root     root       31.7K Nov 18 13:35 EKR-SVR02-SQL-Snapshot23.vmsn

                          -rw-------    1 root     root       31.7K Dec  9 02:05 EKR-SVR02-SQL-Snapshot26.vmsn

                          -rw-r--r--    1 root     root          13 Nov  3 19:20 EKR-SVR02-SQL-aux.xml

                          -rw-------    1 root     root        8.5K Dec  9 23:14 EKR-SVR02-SQL.nvram

                          -rw-------    1 root     root        1.6K Dec  9 02:05 EKR-SVR02-SQL.vmsd

                          -rwx------    1 root     root        4.0K Dec  9 23:13 EKR-SVR02-SQL.vmx

                          -rw-------    1 root     root           0 Dec  9 23:13 EKR-SVR02-SQL.vmx.lck

                          -rw-------    1 root     root        3.1K Dec  9 01:56 EKR-SVR02-SQL.vmxf

                          -rwx------    1 root     root        3.9K Dec  9 23:13 EKR-SVR02-SQL.vmx~

                          -rw-------    1 root     root        5.0M Dec  9 02:03 EKR-SVR02-SQL0-000001-ctk.vmdk

                          -rw-------    1 root     root        8.8G Dec  9 02:03 EKR-SVR02-SQL0-000001-sesparse.vmdk

                          -rw-------    1 root     root         481 Dec  9 01:58 EKR-SVR02-SQL0-000001.vmdk

                          -rw-------    1 root     root        5.0M Nov 18 13:34 EKR-SVR02-SQL0-000002-ctk.vmdk

                          -rw-------    1 root     root       80.0G Nov 18 13:34 EKR-SVR02-SQL0-000002-flat.vmdk

                          -rw-------    1 root     root         637 Nov 18 00:19 EKR-SVR02-SQL0-000002.vmdk

                          -rw-------    1 root     root        5.0M Dec  9 01:57 EKR-SVR02-SQL0-000003-ctk.vmdk

                          -rw-------    1 root     root      326.0M Dec  9 01:57 EKR-SVR02-SQL0-000003-sesparse.vmdk

                          -rw-------    1 root     root         427 Dec  9 01:57 EKR-SVR02-SQL0-000003.vmdk

                          -rw-------    1 root     root        5.0M Dec  9 23:14 EKR-SVR02-SQL0-000004-ctk.vmdk

                          -rw-------    1 root     root      537.2M Dec  9 23:15 EKR-SVR02-SQL0-000004-sesparse.vmdk

                          -rw-------    1 root     root         427 Dec  9 23:13 EKR-SVR02-SQL0-000004.vmdk

                          -rw-------    1 root     root        7.5M Dec  9 02:03 EKR-SVR02-SQL_1-000001-ctk.vmdk

                          -rw-------    1 root     root        2.2G Dec  9 02:03 EKR-SVR02-SQL_1-000001-sesparse.vmdk

                          -rw-------    1 root     root         477 Dec  9 01:58 EKR-SVR02-SQL_1-000001.vmdk

                          -rw-------    1 root     root        7.5M Dec  9 01:57 EKR-SVR02-SQL_1-000002-ctk.vmdk

                          -rw-------    1 root     root      487.0M Dec  9 01:57 EKR-SVR02-SQL_1-000002-sesparse.vmdk

                          -rw-------    1 root     root         430 Dec  9 01:57 EKR-SVR02-SQL_1-000002.vmdk

                          -rw-------    1 root     root        7.5M Dec  9 23:14 EKR-SVR02-SQL_1-000003-ctk.vmdk

                          -rw-------    1 root     root      552.0M Dec  9 23:15 EKR-SVR02-SQL_1-000003-sesparse.vmdk

                          -rw-------    1 root     root         430 Dec  9 23:13 EKR-SVR02-SQL_1-000003.vmdk

                          -rw-------    1 root     root        7.5M Nov 18 13:34 EKR-SVR02-SQL_1-ctk.vmdk

                          -rw-------    1 root     root      120.0G Nov 18 13:34 EKR-SVR02-SQL_1-flat.vmdk

                          -rw-------    1 root     root         599 Nov 18 00:19 EKR-SVR02-SQL_1.vmdk

                          -rw-------    1 root     root      318.8K Nov 18 13:34 vmware-28.log

                          -rw-------    1 root     root      540.8K Nov 25 12:20 vmware-29.log

                          -rw-------    1 root     root      431.0K Dec  8 14:53 vmware-30.log

                          -rw-------    1 root     root      397.1K Dec  9 01:56 vmware-31.log

                          -rw-------    1 root     root      324.5K Dec  9 02:03 vmware-32.log

                          -rw-------    1 root     root      559.3K Dec  9 23:03 vmware-33.log

                          -rw-------    1 root     root      268.9K Dec  9 23:14 vmware.log

                          • 10. Re: Cannot delete snaphsots, but can create new ones.
                            daphnissov Virtuoso
                            vExpert

                            Could you please re-attach these logs in a file that doesn't have commas? It's failing to download and I'm just guessing it doesn't like commas.

                            • 11. Re: Cannot delete snaphsots, but can create new ones.
                              daphnissov Virtuoso
                              vExpert

                              Try and create a new directory in that VM's home directory (mkdir backup) and move EKR-SVR02-SQL-Snapshot26.vmsn into it with mv EKR-SVR02-SQL-Snapshot26.vmsn backup/EKR-SVR02-SQL-Snapshot26.vmsn. Try the delete all again and see if it likes that. VMSN files shouldn't effect the removal or consolidation process, but they make it so you can't revert to that state. I probably should have said to move Snapshot25.vmsn into that backup directory before but I figured you're not going to revert.

                              • 12. Re: Cannot delete snaphsots, but can create new ones.
                                EKroboter Novice

                                Sure, here it is without commas.

                                • 13. Re: Cannot delete snaphsots, but can create new ones.
                                  EKroboter Novice

                                  I moved snapshot26, but it still isn't able to delete al the snapshots. Consolidation fails as well. This is getting serious.

                                  The VM still works perfectly fine though.

                                  • 14. Re: Cannot delete snaphsots, but can create new ones.
                                    daphnissov Virtuoso
                                    vExpert

                                    Do a cat EKR-SVR02-SQL.vmsd and paste the output.

                                    1 2 Previous Next