Not enough space to power on all VM's on ESXi 4.1?

[cmiebach]

We have one ESXi 4.1 host server with all guest servers configured as thick provisioned (upgrading to 6.x later this year).  Datastore1 is showing 60GB of free space and Datastore2 is showing 24GB of free space.  However, I cannot power on (sometimes it works) all the servers without getting a message that there is not enough free space.  Before this happened we had Veeam backups failing sporadically, which I'm exploring possibilities of this being an issue with Veeam support.  But the real problem started when Datastore1 had 80GB of free space and we created a new 50GB partition for one of the servers.  We started getting messages that there was not enough free space and so we deleted the new partition.  Now as I mentioned Datastore1 is showing only 60GB of free space and we cannot at times start all the servers.

What can I do to get ESXi stable again?

[MKguy]

What does the output of "df -h" show on the CLI, does it match the values reported in the GUI?

Are you taking the VM memory swap file into account? When you power-on a VM, ESXi creates a VM swap file that is the size of the guest RAM (minus memory reservation, if you've set some) that will be used in case a physical memory contention situation needs pages swapped to disk.

So for example, if you have 60GB of free space on a datastore and you try to power-on 2 VMs with 32GB RAM (and no memory reservation), only one of the VMs will be able to power-on.

A process-swapfile will also be created at power-on time, but it's usually just 100-200MB in size.

[cmiebach]

Below is the output of that command.  Where exactly in the GUI can I compare?

I've considered the swap files, but just to review all the servers on Datastore1 amount to 26GB RAM and we have 60GB free space.  All the servers on Datastore2 amount to 5GB RAM and we have 24GB free space.

[DZ1]

Could there be snapshots taking up too much space? 

[cmiebach]

There was only one Veeam snapshot that the Veeam support tech had me delete.