Snapshots not in snapshot manager, consolidation fails

[SWoolley]

We have VMWARE 5.0.0, single VM,  and the sole 800GB datastore is running out of space. Apparently there was a snapshot running for years and it filled up the datastore (only 6GB left). We deleted the snapshot in Snapshot manager but there are still snapshot file(s) growing in the datastore VM folder deleting this final 6GB of free space rather rapidly. The ESXI vSphere client shows “Data consolidation needed” but then we try to consolidate (VM/Snapshots/Consolidate) we get the error after a little bit of time stating “unable to access file <unknown file>"

We are not VMWARE experts and adopted this client (server) from a previous consultant. We are looking to recapture the space and remove any snapshots while keeping the data current. If you are interested, please provide an estimate of cost so we might see how to proceed further.

Attached is a picture of the datastore browser. Thanks in advance

[SWoolley]

Follow up for more info:

We followed the directions in KB1005049, KB1026380, and without successful results.

vim-cmd vmsvc/get.snapshotinfo VMID
Output just returns to command line (no results)

vim-cmd vmsvc/snapshot.get VMID
Output:
"Get Snapshot: " basically empty

find /vmfs/volumes/ -iname "*.vmx" -exec grep -HEie "-[0-9]{6}.vmdk" {} \;
Output:
/vmfs/volumes/514f4e4e-ea0d6e38-a258-9c8e99517379/EPSVR1/EPSVR1.vmx:scsi0:0.fileName = "EPSVR1-000005.vmdk"
/vmfs/volumes/514f4e4e-ea0d6e38-a258-9c8e99517379/EPSVR1/EPSVR1.vmx:scsi0:1.fileName = "EPSVR1_1-000005.vmdk"

[a_p_]

Did you try to consolidate the snapshot with the VM powered on, or powered off?

Since the VM's has thick provisioned virtual disks, I'd suggest that you properly shutdown the VM, create a new snapshot, and then run "Delete All" from the Snapshot Manager. This function should merge all the data from the delta files into the base disk. Due to the snapshot sizes this may take some time, so please remain patient, and do not cancel the task.

If you do have a backup application in place, make sure (as always) that you have a recent backup prior to deleting the snapshots.

André

[SWoolley]

Powered on. Very cool idea. I will try that tonight after hours and post results tomorrow

[SWoolley]

So we followed those steps. Shut down VM, created a "Test" snapshot. New files were created (...00006.vmdk and ..._1-00006.vmdk.) 

We then went to snapshot manager and saw this "test" snapshot and selected "Delete All". It took about an hour and the snapshot is gone from Snapshot Manager but all original files and two new files (mentioned above) remain.

We are trying another "consolidation" task now with the VM still shut off.

[SWoolley]

See new datastore content (attached)

[SWoolley]

It did fail consolidation earlier, although no popup window. I found the failure in the VM event log. See attached. its running again, stuck at 99%. Sure it will fail like previous

Any other ideas?

[SWoolley]

Failed again, new error "I/O error" . See attached

[a_p_]

"I/O errors" doesn't sound good, and the only one here on VMTN who might be able to help you with this is @continuum. You may try to contact him via Skype (see his profile page).

Do you have another datastore with sufficient free disk space, which could be used to manually clone the VM?

André

[continuum]

Do you have free space on another datastore ?
The idea is to use Linux ddrescue to clone the latest delta/sesparse vmdk to another datastore.
If you need help with that - call me via skype.

Ulli

[SWoolley]

We do not have another datastore but will attempt to add a new volume today to hardware. Otherwise we will be taking the path of restoring this server to another temporary platform using our Storagecraft solution. Thanks, might reach out later today 

[SWoolley]

I understand that there is a major issue consolidating what we have now but there is not way to stop the snapshots from growing?

[amansapra19]

You can force consolidation of Snapshot Disks by either migrating the VM to another datastore or by cloning the VM into a new VM. Try migrating or cloning in powered off state and then check. 

[continuum]

With only 6 gb left on the datastore you may fill up the volume any minute from now.
And filling up the datastore will very likely corrupt the latest snapshot !
Best option in such a case is a short downtime in which you add a another snapshot on a datastore with enough free space.
That will buy you the time to look into options for a consolidation.

Ulli

[SWoolley · Accepted solution]

This server has "paused" due to running out of space a few times in the last two weeks so my guess is that the snapshot(s) are corrupt and thus the reason we cannot consolidate. We are going to take the route of setting up a new server and restoring from Storagecraft backup instead. Thanks all for your support

[continuum]

If a VM can not be consolidated this is in 3/4 of the cases due to a invalid vmsd-file.

In your case the problem is the I/O error.