huge snapshot won't merge and storage misreporting

[jchcomputers]

I am having an issue with a esxi 4.0 host, first of all someone created a snapshot in 10/2010 and it has been running off it since, it grew so big that the datastore filled up and the machine would not boot.  I cleared some space and got it to boot and then tried to delete the snapshot and it went down again because deleting a snapshot uses some free space as well.  I cleared more space and booted it and the snapshot does not show in the snapshot manager but it appears to still be in the datastore.  Another issue is that the datastore shows 100gb free but that is not the case, it has nothing free.  Here are my questions, why is the datastore misreporting free space and how can i get rid of and merge the large 130gb old snapshot?

thanks

Josh

[a_p_]

Welcome to the Community,

to be able to help I need some more details.

• the build number of ESXi hosts
• is the base disk thin or thick provisioned
• does the datastore browser show only one snapshot

Please post a screen shot of the datastore browser window, showing all of the VM's files with all the details (names, sizes, time stamps, ...) and attach the VM's latest vmware.log file.

André

[jchcomputers]

The build is ESX 4.0.0 261974

It is a thick provisioned disk, it looks like it's only one snapshot but there are several files that look like they are part of the snapshot???

The snapshot manager showed the snapshot before the system went down during the snapshot delete, now it is not in the manager.  I would like to merge the snapshot back in and get this thing stable again.

[jchcomputers]

screenshot attachment

[a_p_]

Unless there are other VM's running on this host which may grow due to either thin provisioning or active snapshots, I'd recommend you shutdown the VM, open the Snapshot Manager, create a new snapshot and the select "Delete All". This will consolidate all snapshots no matter whether they show up in the snapshot manager or not. The reason I asked for the version you are using is that this is only possible with ESXi 4.0 Update 2 or later, if there are multiple snapshots for a single virtual disk.

Due to the snapshot sizes you need to calculate with a few hours for the snapshots to be committed.

André

[jchcomputers]

There are other VM's running on the host, how will this task affect them.  Is it possible to create the new snapshot and delete the snapshots while the VM is running?

[a_p_]

There are other VM's running on the host, how will this task affect  them.

The delete process will certainly add some load on the storage. Depending on your storage system and it's speed you may see some performace decrease on the other VMs.

Is it possible to create the new snapshot and delete the  snapshots while the VM is running?

Doing this would create new snapshot files for the virtual disks which may grow and use disk space. Since you don't have much disk space, I'd recommend to commit the snapshot with the VM powered off.

André

[jchcomputers]

One last question, I have a virtual disk/partition on the VM in question that I can delete to free up 160gb.  If I do this without deleting the snapshot, will this cause an issue?

[a_p_]

Regarding the sizes of the virtual disks and snapshots I'm not quite sure whether you are talking about a virtual disk or a single partition on a virtual disk within the guest OS!? If the latter, this does not make any difference.

André

[jchcomputers]

I'm talking about a virtual disk that I added to the VM.  If I delete this disk, will I cause an issue with the snapshot that the VM is currently running off?  Someone told me not to add, delete or resize virtual disks when you have an active snapshot.

[a_p_]

Basically you can do anything with the VM that's possible in the VM's settings (in the vSphere Client GUI). You will see that adding and removing virtual disks is possible, but the disk sizes are grayed out (resizing is only allowed with no active snapshots). Removing a virtual disk with snapshots will permanently delete the virtual disk with all its snapshots when you select "Delete from disk".

André

[jchcomputers]

I created another snapshot and removed a 160gb virtual disk from the machine but the VMDK file sis not delete from the datastore.  I read an article that said you are unable to delete a virtual disk from a VM when it is currently running off a snapshot.  I would like to free up enough space to delete the snapshots while  the machine is up and running.  What do you think my best plan of action is, to purchase a VMware support contract and open a case with them?

[a_p_]

What do you think my best plan of action is, to purchase a VMware support contract and open a case with them?

This is certainly an option.

Now I don't want to mess up your data, but I would probably go ahead, power off the VM, create a sub-directory in the VM's folder on the datastore and move the virtual disk (as well as it's snaphots) which you removed from the VM's settings to this sub-directory instead of deleting it right away. Then take another snapshot and power on the VM. If this works, you can delete the previously moved files as well as the new folder and then run the "Delete All" from the Snapshot Manager.

André

[jchcomputers]

And if it doesnt start, I can always move them back?  If It does start I can delete them, right?

Also, how do I tell what snapshot file is associated with that virtual disk that I removed?

[a_p_]

And if it doesnt start, I can always move them back?  If It does start I can delete them, right?

Yes to both!

Also, how do I tell what snapshot file is associated with that virtual disk that I removed?

That's an excellent question. Did you take a look at the virtual disk file name in the settings before you removed the virtual disk? If not, please post (attach) the latest vmware.log as well as the VM's .vmx file to see which of the .vmdk files are not referenced anymore.

Since you mentioned 160GB, I assume you removed the EVI14_3*.vmdk files!?

André

[jchcomputers]

Thank you for all of your help on this.  Here are the files you requested.  I am trying to learn more about VMware, what are you looking for in these files? I would like to know so I am able to work on it as well.

[a_p_]

Interesting. I didn't expect that you hot removed the third virtual disk, without selecting to delete the disk from the datastore.

from the .vmx file:

scsi0:2.fileName = "EVI14_3-000002.vmdk"
...

scsi0:2.present = "FALSE"

from the vmware.log:

Feb 23 19:58:22.147: vcpu-0| DISKLIB-VMFS  : "/vmfs/volumes/4c22539a-c3af83f7-ae27-001a64b53c18/EVI14/EVI14_3-000002-delta.vmdk" : closed.
Feb 23 19:58:22.147: vcpu-0| DISKLIB-VMFS  : "/vmfs/volumes/4c22539a-c3af83f7-ae27-001a64b53c18/EVI14/EVI14_3-000001-delta.vmdk" : closed.
Feb 23 19:58:22.147: vcpu-0| DISKLIB-VMFS  : "/vmfs/volumes/4c22539a-c3af83f7-ae27-001a64b53c18/EVI14/EVI14_3-flat.vmdk" : closed.
Feb 23 19:58:22.185: vcpu-0| VMXVmdb_LoadRawConfig: Loading raw config
Feb 23 19:58:22.191: vcpu-0| Vix: [421559 vmxCommands.c:1580]: VMAutomation: Hot plug completed. err=0, vmdbErr=0, asyncCommand=CC5BAB0, OpCode=139
Feb 23 19:58:22.316: vmx| Vix: [421554 vmxCommands.c:2011]: VMAutomation_HotPlugCommitBatch: commiting the HotPlug Batch.
Feb 23 19:58:22.316: vmx| Vix: [421554 vmxCommands.c:1580]: VMAutomation: Hot plug completed. err=0, vmdbErr=0, asyncCommand=CC57BB0, OpCode=159

From the log file you can determine which files to move to the sub-directory (if you are going to follow the steps in my previous post).

EVI14_3.vmdk
EVI14_3-000001.vmdk
EVI14_3-000002.vmdk

Regarding the file names: The files names shown in the log file (-flat.vmdk and -delta.vmdk) are associated with the 3 files I mentioned, but will not show up in the datastore.

André

[jchcomputers]

I did chose to delete them from the datastore, how can I delete them now? For some reason they must have not deleted

[a_p_]

It seems to be related to the snapshots (as mentioned by you in an earlier post). I reproduced this scenario with a test VM and found a weird behavior. After deleting the virtual disk in the VM's settings, the .vmdk files were still present (ok so far with me), but then - after the "Delete All" - only the latest snapshot of this disconnected virtual disk was deleted and not all of the files!? That's definitely not how I expected this to work.

Anyway, with moving the .vmdk files and then deleting them manually - after verifying the VM powers on without issues - the files will definitely be gone.

André

[jchcomputers]

And hopefully after I delete these files for the 160gb disk, I power the machine back on and I am able to  delete all the snapshots, even those that do not show up.

If not, I guess I will have to do a V2V conversion on this machine, and create a completely new machine, or is there a way to manually delete the snapshots?