Hey all,
I have Ubuntu VM and over the time I would create snapshots. Now I noticed that thin provisioned disk of 80GB and in ESXi it shows that 215GB is being used. Ubuntu terminal shows only 48GB is being used.
I deleted all snapshots and thought that would clear it up but it didn't. I've noticed that VM folder on the host has multiple Ubuntu (1)-000001-sesparse.vmdk, Ubuntu (1)-000002-sesparse.vmdk etc. and are all large files.
Any suggestion how to safely free up this space? Screenshots are attached for the visual.
Thanks!
To understand the current situation, and to find possible solutions, please post a complete file listing (the text output of ls -lisa). In addition to this post the free disk space in this and other datastores. Last but not lease, attach the VM's vmware.log file to your next reply. This will show the snapshot chain.
André
If you have space in this datastore or any other shared storage, you may clone this VM, which should remove and consolidate the snapshots.
Note: Once cloned, power off the old VM and then power on the new VM to avoid any IP conflict.
Please mark this as a solution if this helps.
--------------------------------------------------------------
Although I am a VMware employee I contribute to VMTN voluntarily (i.e., not in any official capacity)
@a_p_ wrote:To understand the current situation, and to find possible solutions, please post a complete file listing (the text output of ls -lisa). In addition to this post the free disk space in this and other datastores. Last but not lease, attach the VM's vmware.log file to your next reply. This will show the snapshot chain.
André
Thanks for the reply!
Today I noticed that Ubuntu won't start due to low storage on Samsung 1TB SSD datastore.
Here's all requested info:
[root@thinkserver:~] ls -lisa
total 817
1 4 drwxr-xr-x 1 root root 512 Apr 24 00:35 .
1 4 drwxr-xr-x 1 root root 512 Apr 24 00:35 ..
5309 4 -rw------- 1 root root 9 Apr 24 00:35 .ash_history
4484 4 -r--r--r-- 1 root root 20 Aug 5 2019 .mtoolsrc
5302 4 lrwxrwxrwx 1 root root 49 Apr 23 07:03 altbootbank -> /vmfs/volumes/36e890e3-a7b7f574-a021-9d6a2d7bf611
10 4 drwxr-xr-x 1 root root 512 Apr 23 07:03 bin
5301 4 lrwxrwxrwx 1 root root 49 Apr 23 07:03 bootbank -> /vmfs/volumes/fefc22e2-8a8ba970-f2c3-bb5e0935f260
4486 336 -r--r--r-- 1 root root 342277 Aug 5 2019 bootpart.gz
4485 224 -r--r--r-- 1 root root 226641 Aug 5 2019 bootpart4kn.gz
1 0 drwxr-xr-x 16 root root 512 Apr 24 00:35 dev
2 4 drwxr-xr-x 1 root root 512 Apr 23 07:03 etc
13 4 drwxr-xr-x 1 root root 512 Apr 23 07:03 lib
12 4 drwxr-xr-x 1 root root 512 Apr 23 07:03 lib64
5300 28 -r-x------ 1 root root 25635 Apr 23 07:01 local.tgz
5305 4 lrwxrwxrwx 1 root root 6 Apr 23 07:03 locker -> /store
4483 4 drwxr-xr-x 1 root root 512 Apr 23 07:03 mbr
3 4 drwxr-xr-x 1 root root 512 Apr 23 07:03 opt
72016948364192 128 drwxr-xr-x 1 root root 131072 Apr 24 00:35 proc
5306 4 lrwxrwxrwx 1 root root 29 Apr 23 07:03 productLocker -> /locker/packages/vmtoolsRepo/
1755 4 lrwxrwxrwx 1 root root 4 Aug 5 2019 sbin -> /bin
5304 4 lrwxrwxrwx 1 root root 57 Apr 23 07:03 scratch -> /vmfs/volumes/5c73314d-b5809724-22f9-000423cb2c48/.locker
5303 4 lrwxrwxrwx 1 root root 49 Apr 23 07:03 store -> /vmfs/volumes/5c756fc2-6d57025c-ca7a-000423cb2c48
5 4 drwxr-xr-x 1 root root 512 Apr 23 07:03 tardisks
6 4 drwxr-xr-x 1 root root 512 Apr 23 07:02 tardisks.noauto
38145 4 drwxrwxrwt 1 root root 512 Apr 24 00:29 tmp
14 4 drwxr-xr-x 1 root root 512 Apr 23 07:03 usr
4 4 drwxr-xr-x 1 root root 512 Apr 23 07:03 var
18 4 drwxr-xr-x 1 root root 512 Apr 23 07:03 vmfs
16 4 drwxr-xr-x 1 root root 512 Apr 23 07:03 vmimages
1762 4 lrwxrwxrwx 1 root root 18 Aug 5 2019 vmupgrade -> /locker/vmupgrade/
[root@thinkserver:~]
[root@thinkserver:~] esxcli storage filesystem list
Mount Point Volume Name UUID Mounted Type Size Free
------------------------------------------------- ------------------- ----------------------------------- ------- ------ ------------ ------------
/vmfs/volumes/5c73314d-b5809724-22f9-000423cb2c48 Samsung 1TB SSD 5c73314d-b5809724-22f9-000423cb2c48 true VMFS-6 999922073600 10844372992
/vmfs/volumes/5ed2d190-cc7cfe51-e757-000423cb2c48 Samsung 1TB SSD 2nd 5ed2d190-cc7cfe51-e757-000423cb2c48 true VMFS-6 999922073600 141553565696
/vmfs/volumes/fefc22e2-8a8ba970-f2c3-bb5e0935f260 fefc22e2-8a8ba970-f2c3-bb5e0935f260 true vfat 261853184 106074112
/vmfs/volumes/5c756fc2-6d57025c-ca7a-000423cb2c48 5c756fc2-6d57025c-ca7a-000423cb2c48 true vfat 299712512 117465088
/vmfs/volumes/36e890e3-a7b7f574-a021-9d6a2d7bf611 36e890e3-a7b7f574-a021-9d6a2d7bf611 true vfat 261853184 107630592
[root@thinkserver:~]
@ksujay wrote:If you have space in this datastore or any other shared storage, you may clone this VM, which should remove and consolidate the snapshots.
Note: Once cloned, power off the old VM and then power on the new VM to avoid any IP conflict.
Please mark this as a solution if this helps.
--------------------------------------------------------------
Although I am a VMware employee I contribute to VMTN voluntarily (i.e., not in any official capacity)
Unfortunately both 1TB datastores are full. The reason why I got another 1TB is because first one was filled. Now I wonder if this is ESXi bug?
Sorry if I wasn't quite clear with what I wanted.
Although it looks like the issue can be solved with cloning the VM's virtual disk to the second datastore, there may be an easier way.
What I need is the file listing for the files in the VM's folder, as well as one for all of the other VM's folders/files on the filled up (first) datastore.
The reason why you datastore fills up, is that you seem to keep multiple snapshots for a long time. Please remember that each single snapshot can grow up to its parent virtual disk's provisioned size. Also keep in mind that snapshots are not a replacement for backups!
André
This is not any ESXi bug. The snapshot file continues to grow in size when it is retained for a longer period. This can cause the snapshot storage location to run out of space and impact the system performance. Do not use a single snapshot for more than 72 hours is what as recommended (https://kb.vmware.com/s/article/1025279)
Are you getting the consolidate option in snapshot manager? If yes, any error messages in the consolidate process?
Also please share the output for (grep vmdk *.vmx) from the VM folder.
---------------------------------------------------------------------------------------------------------
Although I am a VMware employee I contribute to VMTN voluntarily (i.e., not in any official capacity)
@a_p_ wrote:Sorry if I wasn't quite clear with what I wanted.
Although it looks like the issue can be solved with cloning the VM's virtual disk to the second datastore, there may be an easier way.
What I need is the file listing for the files in the VM's folder, as well as one for all of the other VM's folders/files on the filled up (first) datastore.The reason why you datastore fills up, is that you seem to keep multiple snapshots for a long time. Please remember that each single snapshot can grow up to its parent virtual disk's provisioned size. Also keep in mind that snapshots are not a replacement for backups!
André
I'm at the point where there's not enough free space to do anything. I was using snapshots before installing various projects ie. nextcloud, SyncThing etc. in case something goes wrong I can revert back, but I guess at some point I left them longer than 72hrs and I'm this situation.
Can you please elaborate on this "What I need is the file listing for the files in the VM's folder, as well as one for all of the other VM's folders/files on the filled up (first) datastore." You want file listing on each VM folder on first datastore?
>>> You want file listing on each VM folder on first datastore?
Yes, that's what I'm thinking of. This will help to find out how much disk space will be required to consolidate the snapshots, and to find out whether there's an option to free up enough disk space by e.g. temporarily shutting down other VMs.
If that doesn't help, we can clone the Ubuntu VMs virtual disk to the second datastore, and in the same step consolidate all snapshots.
André
@ksujay wrote:This is not any ESXi bug. The snapshot file continues to grow in size when it is retained for a longer period. This can cause the snapshot storage location to run out of space and impact the system performance. Do not use a single snapshot for more than 72 hours is what as recommended (https://kb.vmware.com/s/article/1025279)
Are you getting the consolidate option in snapshot manager? If yes, any error messages in the consolidate process?
Also please share the output for (grep vmdk *.vmx) from the VM folder.
---------------------------------------------------------------------------------------------------------
Although I am a VMware employee I contribute to VMTN voluntarily (i.e., not in any official capacity)
I am getting notice to start consolidate process, but when it starts it says not enough space to complete.
Here are output of the grep command:
[root@thinkserver:/vmfs/volumes/5c73314d-b5809724-22f9-000423cb2c48/Ubuntu (1)] grep vmdk *.vmx
scsi0:0.fileName = "Ubuntu (1)-000010.vmdk"
[root@thinkserver:/vmfs/volumes/5c73314d-b5809724-22f9-000423cb2c48/Ubuntu (1)]
@a_p_ wrote:>>> You want file listing on each VM folder on first datastore?
Yes, that's what I'm thinking of. This will help to find out how much disk space will be required to consolidate the snapshots, and to find out whether there's an option to free up enough disk space by e.g. temporarily shutting down other VMs.
If that doesn't help, we can clone the Ubuntu VMs virtual disk to the second datastore, and in the same step consolidate all snapshots.
André
I guess that is the easiest option to move Ubuntu to 2nd datastore. I can do that by simply using export option on the 1st datastore and importing into the 2nd?
Is attached export option correct way to do it?
Yes, Export/Import is yet another option with the benefit that you can test the imported VM before you delete the old one.
André
Can you share the output for the below from the vmfolder?
grep -i DiskLib_IsCombinePossible vmware.log
@ksujay wrote:Can you share the output for the below from the vmfolder?
grep -i DiskLib_IsCombinePossible vmware.log
When I run that command I grt blank output?
[root@thinkserver:/vmfs/volumes/5c73314d-b5809724-22f9-000423cb2c48] cd "Ubuntu (1)"
[root@thinkserver:/vmfs/volumes/5c73314d-b5809724-22f9-000423cb2c48/Ubuntu (1)] grep -i DiskLib_IsCombinePossible vmware.log
[root@thinkserver:/vmfs/volumes/5c73314d-b5809724-22f9-000423cb2c48/Ubuntu (1)]
Finally got it solved by exporting the image from ESXi and importing it into 2nd datastore. Surprisingly it did not complain during the export process regarding low space.
Thank you all!