VM question pending but disk usage does not match between esxi and vm

[pvmwt]

hi,

I have two disks of 2TB capacity under datastore1. I have two vms, one is windows with 250GB and another ubuntu with 3.15TB.

windows is not running. As I was copying a lot of data my ubuntu vm ended up with "pending question" state and stopped working. After searching online I "reserved" memory and that helped it to power on.

After powering on though, I see ubuntu shows that it is using just 56GB

From vsphere events it shows below

There is no more space for virtual disk 'ncaio-000002.vmdk'. You might be able to continue this session by freeing disk space on the relevant volume, and clicking Retry. Click Cancel to terminate this session.

SSH to esxi shows 100% used. See below

So what is going on and what could be done to fix this?

I think I left about 16GB FREE in the datastore. Is that the problem? If so, what should be free size left and how do I calculate how much to leave?

p.s. the VM disk is think provisioned (lazily zeroed).

[pvmwt]

I was able to remove all snapshots "Delete all", please see screenshot on filesystem

There were 2 snapshots. It seems like snapshots were the problem, given there is no space in datastore (only 15GB free) to store them? What can be done to handle the situation?

[pvmwt]

Issue re-occurred after all snapshots have been deleted. It suggested to consolidate disk and I did that but that did not help at all. Then I deleted windows vm to free up 250GB, after which I was able to access ubuntu vm.

Can any one explain what is the reason for this and what do I do to fix the situation?

[pashnal]

Hi , 

In my own experience I feel that when ever you have  single VM with huge HDD you need minimum of 15-20% GB free space as it will cause issues when you take snapshot , Consolidation and Snapshot deletion will also need some extra space , Check the below article if this helps

If you have VMs with smaller VMs you can have 10-15 percent of free space . 

https://kb.vmware.com/s/article/1003412

Thanks , 

Pramod Ashnal 

Please mark my comment as the Correct Answer/Kudos if this solution resolved your problem Thank you.

[Quickdraw]

Your datastore needs some headroom for some functions as pashnal said. You need to be careful with snapshots. They should only be used for short term backups. There is no limit on snapshots and they will continue to grow and fill your storage. We don't let them hang around for more then 2 weeks if possible. You can also thin provision your drives. This allows you to over commit your storage. If you specify a 1TB HDD but only have 50gb of data on it, only 50gb will be used. The rest would be availble to other vm's. You need to be care with this as well that you don't over commit to much and fill the datastore that way. We thin provision all oour drives as it's rare that an OS will use all the space you give it. 

[pvmwt]

Thanks to both of you! After considering the extra storage requirement which is quite a bit, I decided to dismantle virtualization and installed linux on bare metal.