Local datastore slowly diminishing (ESXi 4.1)

[VictorST]

Hello folks! Don't know what is going on with my datastore. In my inventory I created several virtual machines with thin provisioned hard disks. I thought that this was the reason why my storage started to slowly diminish. Then I made an inflate procedure on this thin disks because I was pretty sure that this action may help. Anyway it diminishes every day little be little. It's starts getting rather dangerous.

Can I stop it? How can I find the real reason of this issue?

Many thanks in advance!

[continuum]

When you are sure that you have no more thin vmdks check for VMs with snapshots.

[VictorST]

There was a tons of snapshots here and there. Now I deleted all of them but still the size of the storage is diminishing for some unclear purpose. Have no idea of what is going on. Maybe I made an inflate uncorrectly? But everything works fine. How can I find any clue to get the grasp of what is going on?

[a_p_]

Are you sure that you delete all snapshots, i.e. also those which do not show up in the Snapshot Manager? I'd suggest you run RVTools to check whether any of the VMs still has active snapshots. To delete snapshots which do not show up in the Snapshot Manager, simply create a new snapshot, and then run "Delete All" from the Snapshot Manager.

André

[VictorST]

Thanks for the information but it seems like there are no snapsots. Look at this:

Яндекс.Фотки

Where should I check for hidden snapshots in RVtools?

[a_p_]

If RVTools doesn't list snapshots, there are none.

What you may do next is to run ls -lisa in each of the VM's folders to check whether the virtual disks are still thin provisioned. This command will not only show the provisioned size, but also the used space in kB. If both - the used space, and the provisioned size - match, then the .vmdk shouldn't grow anymore.

André

[VictorST]

Here are the screeshots of one of the "suspected" virtual machines. Honestly, I don't see what you wrote about and it is pretty difficult to distinguish anything in such a messy manner of output.

Яндекс.Фотки

And here is the screenshot of the other "suspected" VM inventory:

Яндекс.Фотки

Any advises?

And thank you for your attemps to resolve my strange issue. I don't have a great experience with VMware so this help is greatly appreciated.

[a_p_]

From what I could see in the screen shots, it seems that the virtual disks have been inflated already, i.e. the second column (size in kB) is the same as the size in Bytes (left to the date/time). Not sure what else might eat up the disk space. How much additional disk space is being consumed each day, and what's the free space on the datastore?

André

[VictorST]

Here are the statistics of a storage usage:

11.05.2016 - 388,94 GB

12.05.2016 - 388,78 GB

13.05.2016 - 388,12 GB

16.05.2016 - 387,47 GB

17.05.2016 - 387,11 GB

18.05.2016 - 383,01 GB

19.05.2016 - 382,83 GB

I am using european data.month.year format. Free space now is 382,81 GB.

Did you looked at both of my previous screenshots?

I don't know but maybe it is tied somehow with EMC Avamar backup solution that we are using. I am not sure how it works. Any suggestions?

[a_p_]

Yes, I looked at both screen shots (btw. instead of posting screen shots, pasting the command's text output wold be easier to read).

Anyway, what I'd suggest at this point is to run the ls -lisa command for all of your VMs each day, and compare the output (the second column shows the used disk space in kB). This way you should be able to find out which files are growing/consuming the disk space.

André

[VictorST]

I must admit that when I made an inflate procedure for the .vmdk files of a VM with thin disks I followed this instruction: VMware KB: Changing the thick or thin provisioning of a virtual disk But I didn't accomplished the second part of this instruction (point 10 in the main instruction): VMware KB: Reloading a vmx file without removing the virtual machine from inventory I don't understand what is the real reason for reloading .vmx and after I made inflate I noticed that type of provision changed to thick so I ignored the risky procedure of reloading vmx.

Can it be the real reason for the hard disks not changing their provision type? It's like I am thinking that provision type is thick but in fact it's thin and did not changed. 

[a_p_]

Inflating a thin virtual disk actually allocates the provisioned disk space on the datastore, so from a disk space usage point of view there's no difference between a thin or thick disk.

I didn't use the inflate option lately, but from what I remember, I never re-loaded the .vmx file after doing this.

So as mentioned before, you should monitor the usage by comparing the ls -lisa output for a while to find out which file(s) are causing the issue.

André

[VictorST]

It seems like I found the reason why the datastore is slowly diminishing and as far as I understood it is orphaned snapshots which are hidden for some reason and also some delta-differentials. I've just looked at the wrong virtual machines. Here are the screenshots:

Яндекс.Фотки

Яндекс.Фотки

Яндекс.Фотки

I just don't know what to do because some of the virtual machines are working with this snapshots and I am having some serious doubts that I can delete it.

Any advices?

P.S. also if I am trying to move some trash 000001.vmdk files then the VM simply doesn't function asking for that file.

Reason: The system cannot find the file specified.

Cannot open the disk 'Application_Server-000001.vmdk' or one of the snapshot disks it depends on.

VMware ESX cannot find the virtual disk "Application_Server-000001.vmdk". Verify the path is valid and try again.

[a_p_]

It's unusual that RVTools didn't detect these snapshots!?

Anyway, you shoudn't have any issues with deleting the snapshots. Since the flat.vmdk files have already been inflated - i.e. won't grow any further - deleting the snapshots will only require some temporary additional disk space for a helper snapshot if you delete the snapshots while the VM is powered on. If the snapshots don't show up in the Snapshot Manager anymore, simply create a new snapshot (to enable the "Delete All" button) and the hit "Delete All" any remain patient until the process completes.

André

[VictorST]

Can it be dangerous for any virtual machines that are working utilizing snapshot .vmdk (like ...-000001.vmdk or something like it)?

This "Delete all" button should it delete and consolidate all of the snapshots on ESXi host or it will only delete any snapshots on exact virtual machine? Never tried it.

To make a new snapshot I should turn off the VM. Am I right?

[jwsconsult · Accepted solution]

Running off of a snapshot will slowly begin to cause performance issues,t he longer the VM runs on the snapshots.  In addition, there is the danger of the datastore filling up, as you're watching happen.

Delete all deletes all of the snapshots on that VM.  Other VMs are not affected.

You can, but it is not required, no.

[VictorST]

Thank you so much, my friends! That definitely helped and after I consolidated snapshots for this problematic VM the storage stopped diminishing. That's it. Live and learn. What else can I say.