VMware Cloud Community
NerdsToGo
Contributor
Contributor

ESXi 6.7.0 Says host is full

I have a client with a server running 6.7 and in the host view it says the storage is 1.08TB and its 100% full. This cannot be true because the host has 6TB total and the windows server VM has plenty of free space on the drives allocated to it. Same with all of the other VMs.  Any suggestions on what is happening here? For the record, I did not create this server/vm setup it was inherited from another company that no longer wanted to deal with this customer. I appreciate any feedback on this issue as I believe it is what is causing the host to crash. 

Reply
0 Kudos
13 Replies
fabio1975
Commander
Commander

Ciao 

Hi can you send us some screenshots of the datastore status, or if you can send us the output of the df -m command launched from the ESXi host shell (You connect via SSH etc ...).

How many VMs are there on the datastore?

Are the VM disks in thin format?

Is the datastore local?

Where do you see that there are 6TB available?

Fabio

Visit vmvirtual.blog
If you're satisfied give me a kudos

Reply
0 Kudos
NerdsToGo
Contributor
Contributor

Hello, thank you for your response. I have attached some screenshots of the datastore.

To answer your questions: 

There are two VMs on the datastore, one Windows Server 2012R2 and one Win10 machine. The R2 machine is allocated a C drive with 2TB and using 547GB of that. It is also allocated a 4.5TB E drive and using 215Gb of that. The other machine is allocated 100GB and only using about 30GB. All VM disks are thin provisioned. The datastore is local. In the Host view, under storage it says 1.08TB total and a small amount free (it changes randomly). When the free space runs out all vms shut down without warning. 

Reply
0 Kudos
fabio1975
Commander
Commander

Ciao 

From the information you gave me you surely have the vm with thin disks. But where do you see that the ESXi host have 6TB of disk space?

Fabio

Visit vmvirtual.blog
If you're satisfied give me a kudos

Reply
0 Kudos
NerdsToGo
Contributor
Contributor

I do not see the ESXi host has free space. It is within the windows server vm that there is free space. I guess I should rephrase, is there a way to shrink those vm disks to free space for other needs? (from 4.55 to say 2.5) And why is the datastore only 1TB. 

Reply
0 Kudos
fabio1975
Commander
Commander

Ciao

check in the VM settings if the disks are in thin or thick format.

fabio1975_0-1622185930160.png

 

If they are in thin format it means that the space occupied on the datastore of the host ESXi by the VMs is only that actually allocated for which (547GB + 215GB + 30GB) and then there will be other space occupied by service files.

So you absolutely have to increase the space available to the ESXi host. Adding disks to the physical server etc ... it is correct that when the datastore is filled to 100% the VMs stop going ... on the datastore where the VMs are housed it is recommended to have at least between 10% and 20% of the free space. The space problem gets worse when you go to VM snapshots too

 

Fabio

Visit vmvirtual.blog
If you're satisfied give me a kudos

Reply
0 Kudos
e_espinel
Virtuoso
Virtuoso

Hello.
What brand and model is the physical server?
What disk controller does it have?
You can access the service and management device of the server (BMC, IMM, IDRAC, LIO type).
Is the VMware vSphere installed with a custom image from the manufacturer?

Use the following command to verify if the image is standard (VMware) or customized by a vendor (Lenovo, IBM, HPE, Dell)

# esxcli software profile get

 

Enrique Espinel
Senior Technical Support on IBM, Lenovo, Veeam Backup and VMware vSphere.
VSP-SV, VTSP-SV, VTSP-HCI, VTSP
Please mark my comment as Correct Answer or assign Kudos if my answer was helpful to you, Thank you.
Пожалуйста, отметьте мой комментарий как Правильный ответ или поставьте Кудо, если мой ответ был вам полезен, Спасибо.
Reply
0 Kudos
NerdsToGo
Contributor
Contributor

Hello thank you for your reply and sorry for the delay,

 

I can confirm that the disks are Thin Provisioned. Is there any way to deallocate a disk from a VM and use that space to expand the datastore?

Reply
0 Kudos
a_p_
Leadership
Leadership

To find out how things look like, and what can be done, please connect to the host via an SSH client (e.g. putty), run ls -lisa in each VM's folder, and provide the command's text output (no screenshot please) in your next reply.

Caution: Do not try to delete any of the VMs files manually!

André

Reply
0 Kudos
NerdsToGo
Contributor
Contributor

ls -lisa
total 1105032640
3204 128 drwxr-xr-x 1 root root 77824 Jun 5 19:22 .
4 1024 drwxr-xr-t 1 root root 77824 Jan 19 17:15 ..
92274884 0 -rw-r--r-- 1 root root 53 Jul 15 2019 HATCHCHIROSRV1-NEW-72d035c8.hlog
4194500 574671872 -rw------- 1 root root 2199023255552 Jun 5 19:22 HATCHCHIROSRV1-NEW-flat.vmdk
46137540 64 -rw------- 1 root root 8684 Jun 5 19:13 HATCHCHIROSRV1-NEW.nvram
8388804 0 -rw------- 1 root root 643 Jun 5 19:12 HATCHCHIROSRV1-NEW.vmdk
20971716 0 -rw-r--r-- 1 root root 0 Sep 24 2018 HATCHCHIROSRV1-NEW.vmsd
196 64 -rwxr-xr-x 1 root root 4618 Jun 5 19:12 HATCHCHIROSRV1-NEW.vmx
29360324 0 -rw------- 1 root root 0 Jun 5 19:12 HATCHCHIROSRV1-NEW.vmx.lck
50331844 64 -rw------- 1 root root 150 May 25 11:59 HATCHCHIROSRV1-NEW.vmxf
33554628 64 -rwxr-xr-x 1 root root 4533 Jun 5 19:12 HATCHCHIROSRV1-NEW.vmx~
12583108 530240512 -rw------- 1 root root 4998133972992 Jun 5 19:22 HATCHCHIROSRV1-NEW_1-flat.vmdk
16777412 0 -rw------- 1 root root 645 Jun 5 19:12 HATCHCH IROSRV1-NEW_1.vmdk
71303364 1024 -rw-r--r-- 1 root root 546986 Jun 5 12:52 vmware- 79.log
37748932 1024 -rw-r--r-- 1 root root 270585 Jun 5 13:23 vmware- 80.log
54526148 1024 -rw-r--r-- 1 root root 388264 Jun 5 14:55 vmware- 81.log
58720452 1024 -rw-r--r-- 1 root root 370116 Jun 5 16:14 vmware- 82.log
41943236 64 -rw-r--r-- 1 root root 45252 Jun 5 16:15 vmware- 83.log
67109060 1024 -rw-r--r-- 1 root root 332416 Jun 5 16:16 vmware- 84.log
62914756 1024 -rw-r--r-- 1 root root 478441 Jun 5 19:22 vmware. log
1027604676 112640 -rw------- 1 root root 115343360 Dec 7 15:13 vmx-H ATCHCHIROSRV1-NEW-1926247880-2.vswp

Reply
0 Kudos
NerdsToGo
Contributor
Contributor

This is the output of the main server VM the other VM is not important right now.

Reply
0 Kudos
a_p_
Leadership
Leadership

The available options with almost no free disk space left on the datastore are limited.

You mentioned that the usage on the 4.5TB E: drive is ~215GB. So what I could think of is to backup the data on that drive, then delete the 4.5TB virtual disk, create a new (smaller) one, and restore the data.

Used [kB] Provisioned [Bytes] File name
574.671.872 2.199.023.255.552 HATCHCHIROSRV1-NEW-flat.vmdk
530.240.512 4.998.133.972.992 HATCHCHIROSRV1-NEW_1-flat.vmdk

 

Another option might be to use the VMware Converter to reduce the virtual disks' sizes. This needs to be done in two steps. First step is to convert it to e.g. a VMware Workstation format, and store it on a local disk. In this step you'd also reduce the virtual disks' sizes. Then - in the second step - convert the VM back to the ESXi host.

André

Reply
0 Kudos
NerdsToGo
Contributor
Contributor

André thanks for your effort,

I would love to do option one, but unfortunately the individual who originally set this up (outside my organization) installed their main application on that 4.5TB drive. So I think it is unlikely that I will be able to perform option one. 

I am reading some about using sdelete and vmkfs tools to free up space, do you know anything about that? If I go into the server VM and shrink that drive from windows, will that make it so I can shrink the disk from ESXi and get free space? 

 

- NTG

Reply
0 Kudos
a_p_
Leadership
Leadership

Using sdelete, and vmkfstools in order to zero out, and then shrink the thin provisioned virtual disk will not work, because this will increase the virtual disk to its provisioned size before the zeroed disk space can be reclaimed.

How does the system get backed up right now? If backup&restore on a file level basis is not an option, the I'd go with the conversion to a local disk.
Thinking of this option again, you can as well use VMware Workstation Pro to "Download" the VM from the ESXi host, then use its integrated feature to reclaim guest disk space (i.e. shrink the virtual disks), and finally - after ensuring that the VM works as expected - delete the VM from the ESXi host, and "Upload" the local copy again.

What I'd also suggest - while the VM is on the local disk - is that you either reduce the guest OS partition sizes to sizes that will not exceed the disk space on the datastore. Alternatively use VMware Converter to reduce the virtual disks' sizes (that's what I would do).

André

 

Reply
0 Kudos