topic Re: ESXi 6.7 snapshots have filled my datastore SSD? in ESXi Discussions

ESXi 6.7 snapshots have filled my datastore SSD?

JohnyBeGood — Sat, 23 Apr 2022 07:51:05 GMT

Hey all,

I have Ubuntu VM and over the time I would create snapshots. Now I noticed that thin provisioned disk of 80GB and in ESXi it shows that 215GB is being used. Ubuntu terminal shows only 48GB is being used.

I deleted all snapshots and thought that would clear it up but it didn't. I've noticed that VM folder on the host has multiple Ubuntu (1)-000001-sesparse.vmdk, Ubuntu (1)-000002-sesparse.vmdk etc. and are all large files.

Any suggestion how to safely free up this space? Screenshots are attached for the visual.

Thanks!

Re: ESXi 6.7 snapshots have filled my datastore SSD?

a_p_ — Sat, 23 Apr 2022 09:29:09 GMT

To understand the current situation, and to find possible solutions, please post a complete file listing (the text output of ls -lisa). In addition to this post the free disk space in this and other datastores. Last but not lease, attach the VM's vmware.log file to your next reply. This will show the snapshot chain.

André

Re: ESXi 6.7 snapshots have filled my datastore SSD?

ksujay — Sat, 23 Apr 2022 22:31:55 GMT

If you have space in this datastore or any other shared storage, you may clone this VM, which should remove and consolidate the snapshots.

Note: Once cloned, power off the old VM and then power on the new VM to avoid any IP conflict.

Please mark this as a solution if this helps.

--------------------------------------------------------------

Although I am a VMware employee I contribute to VMTN voluntarily (i.e., not in any official capacity)

Re: ESXi 6.7 snapshots have filled my datastore SSD?

JohnyBeGood — Sun, 24 Apr 2022 01:07:54 GMT

@a_p_ wrote:
To understand the current situation, and to find possible solutions, please post a complete file listing (the text output of ls -lisa). In addition to this post the free disk space in this and other datastores. Last but not lease, attach the VM's vmware.log file to your next reply. This will show the snapshot chain.
André

Thanks for the reply!

Today I noticed that Ubuntu won't start due to low storage on Samsung 1TB SSD datastore.

Here's all requested info:

[root@thinkserver:~] ls -lisa total 817 1 4 drwxr-xr-x 1 root root 512 Apr 24 00:35 . 1 4 drwxr-xr-x 1 root root 512 Apr 24 00:35 .. 5309 4 -rw------- 1 root root 9 Apr 24 00:35 .ash_history 4484 4 -r--r--r-- 1 root root 20 Aug 5 2019 .mtoolsrc 5302 4 lrwxrwxrwx 1 root root 49 Apr 23 07:03 altbootbank -> /vmfs/volumes/36e890e3-a7b7f574-a021-9d6a2d7bf611 10 4 drwxr-xr-x 1 root root 512 Apr 23 07:03 bin 5301 4 lrwxrwxrwx 1 root root 49 Apr 23 07:03 bootbank -> /vmfs/volumes/fefc22e2-8a8ba970-f2c3-bb5e0935f260 4486 336 -r--r--r-- 1 root root 342277 Aug 5 2019 bootpart.gz 4485 224 -r--r--r-- 1 root root 226641 Aug 5 2019 bootpart4kn.gz 1 0 drwxr-xr-x 16 root root 512 Apr 24 00:35 dev 2 4 drwxr-xr-x 1 root root 512 Apr 23 07:03 etc 13 4 drwxr-xr-x 1 root root 512 Apr 23 07:03 lib 12 4 drwxr-xr-x 1 root root 512 Apr 23 07:03 lib64 5300 28 -r-x------ 1 root root 25635 Apr 23 07:01 local.tgz 5305 4 lrwxrwxrwx 1 root root 6 Apr 23 07:03 locker -> /store 4483 4 drwxr-xr-x 1 root root 512 Apr 23 07:03 mbr 3 4 drwxr-xr-x 1 root root 512 Apr 23 07:03 opt 72016948364192 128 drwxr-xr-x 1 root root 131072 Apr 24 00:35 proc 5306 4 lrwxrwxrwx 1 root root 29 Apr 23 07:03 productLocker -> /locker/packages/vmtoolsRepo/ 1755 4 lrwxrwxrwx 1 root root 4 Aug 5 2019 sbin -> /bin 5304 4 lrwxrwxrwx 1 root root 57 Apr 23 07:03 scratch -> /vmfs/volumes/5c73314d-b5809724-22f9-000423cb2c48/.locker 5303 4 lrwxrwxrwx 1 root root 49 Apr 23 07:03 store -> /vmfs/volumes/5c756fc2-6d57025c-ca7a-000423cb2c48 5 4 drwxr-xr-x 1 root root 512 Apr 23 07:03 tardisks 6 4 drwxr-xr-x 1 root root 512 Apr 23 07:02 tardisks.noauto 38145 4 drwxrwxrwt 1 root root 512 Apr 24 00:29 tmp 14 4 drwxr-xr-x 1 root root 512 Apr 23 07:03 usr 4 4 drwxr-xr-x 1 root root 512 Apr 23 07:03 var 18 4 drwxr-xr-x 1 root root 512 Apr 23 07:03 vmfs 16 4 drwxr-xr-x 1 root root 512 Apr 23 07:03 vmimages 1762 4 lrwxrwxrwx 1 root root 18 Aug 5 2019 vmupgrade -> /locker/vmupgrade/ [root@thinkserver:~]

[root@thinkserver:~] esxcli storage filesystem list Mount Point Volume Name UUID Mounted Type Size Free ------------------------------------------------- ------------------- ----------------------------------- ------- ------ ------------ ------------ /vmfs/volumes/5c73314d-b5809724-22f9-000423cb2c48 Samsung 1TB SSD 5c73314d-b5809724-22f9-000423cb2c48 true VMFS-6 999922073600 10844372992 /vmfs/volumes/5ed2d190-cc7cfe51-e757-000423cb2c48 Samsung 1TB SSD 2nd 5ed2d190-cc7cfe51-e757-000423cb2c48 true VMFS-6 999922073600 141553565696 /vmfs/volumes/fefc22e2-8a8ba970-f2c3-bb5e0935f260 fefc22e2-8a8ba970-f2c3-bb5e0935f260 true vfat 261853184 106074112 /vmfs/volumes/5c756fc2-6d57025c-ca7a-000423cb2c48 5c756fc2-6d57025c-ca7a-000423cb2c48 true vfat 299712512 117465088 /vmfs/volumes/36e890e3-a7b7f574-a021-9d6a2d7bf611 36e890e3-a7b7f574-a021-9d6a2d7bf611 true vfat 261853184 107630592 [root@thinkserver:~]

Re: ESXi 6.7 snapshots have filled my datastore SSD?

JohnyBeGood — Sun, 24 Apr 2022 01:06:16 GMT

@ksujay wrote:
If you have space in this datastore or any other shared storage, you may clone this VM, which should remove and consolidate the snapshots.
Note: Once cloned, power off the old VM and then power on the new VM to avoid any IP conflict.
Please mark this as a solution if this helps.
--------------------------------------------------------------
Although I am a VMware employee I contribute to VMTN voluntarily (i.e., not in any official capacity)

Unfortunately both 1TB datastores are full. The reason why I got another 1TB is because first one was filled. Now I wonder if this is ESXi bug?

Re: ESXi 6.7 snapshots have filled my datastore SSD?

a_p_ — Sun, 24 Apr 2022 08:38:11 GMT

Sorry if I wasn't quite clear with what I wanted.

Although it looks like the issue can be solved with cloning the VM's virtual disk to the second datastore, there may be an easier way.
What I need is the file listing for the files in the VM's folder, as well as one for all of the other VM's folders/files on the filled up (first) datastore.

The reason why you datastore fills up, is that you seem to keep multiple snapshots for a long time. Please remember that each single snapshot can grow up to its parent virtual disk's provisioned size. Also keep in mind that snapshots are not a replacement for backups!

André

Re: ESXi 6.7 snapshots have filled my datastore SSD?

ksujay — Sun, 24 Apr 2022 15:58:35 GMT

This is not any ESXi bug. The snapshot file continues to grow in size when it is retained for a longer period. This can cause the snapshot storage location to run out of space and impact the system performance. Do not use a single snapshot for more than 72 hours is what as recommended (https://kb.vmware.com/s/article/1025279)

Are you getting the consolidate option in snapshot manager? If yes, any error messages in the consolidate process?

Also please share the output for (grep vmdk *.vmx) from the VM folder.

---------------------------------------------------------------------------------------------------------

Although I am a VMware employee I contribute to VMTN voluntarily (i.e., not in any official capacity)

Re: ESXi 6.7 snapshots have filled my datastore SSD?

JohnyBeGood — Sun, 24 Apr 2022 16:47:25 GMT

@a_p_ wrote:
Sorry if I wasn't quite clear with what I wanted.
Although it looks like the issue can be solved with cloning the VM's virtual disk to the second datastore, there may be an easier way.
What I need is the file listing for the files in the VM's folder, as well as one for all of the other VM's folders/files on the filled up (first) datastore.
The reason why you datastore fills up, is that you seem to keep multiple snapshots for a long time. Please remember that each single snapshot can grow up to its parent virtual disk's provisioned size. Also keep in mind that snapshots are not a replacement for backups!
André

I'm at the point where there's not enough free space to do anything. I was using snapshots before installing various projects ie. nextcloud, SyncThing etc. in case something goes wrong I can revert back, but I guess at some point I left them longer than 72hrs and I'm this situation.

Can you please elaborate on this "What I need is the file listing for the files in the VM's folder, as well as one for all of the other VM's folders/files on the filled up (first) datastore." You want file listing on each VM folder on first datastore?

Re: ESXi 6.7 snapshots have filled my datastore SSD?

a_p_ — Sun, 24 Apr 2022 16:55:08 GMT

>>> You want file listing on each VM folder on first datastore?

Yes, that's what I'm thinking of. This will help to find out how much disk space will be required to consolidate the snapshots, and to find out whether there's an option to free up enough disk space by e.g. temporarily shutting down other VMs.

If that doesn't help, we can clone the Ubuntu VMs virtual disk to the second datastore, and in the same step consolidate all snapshots.

André

Re: ESXi 6.7 snapshots have filled my datastore SSD?

JohnyBeGood — Sun, 24 Apr 2022 16:57:14 GMT

@ksujay wrote:
This is not any ESXi bug. The snapshot file continues to grow in size when it is retained for a longer period. This can cause the snapshot storage location to run out of space and impact the system performance. Do not use a single snapshot for more than 72 hours is what as recommended (https://kb.vmware.com/s/article/1025279)
Are you getting the consolidate option in snapshot manager? If yes, any error messages in the consolidate process?
Also please share the output for (grep vmdk *.vmx) from the VM folder.

---------------------------------------------------------------------------------------------------------
Although I am a VMware employee I contribute to VMTN voluntarily (i.e., not in any official capacity)

I am getting notice to start consolidate process, but when it starts it says not enough space to complete.

Here are output of the grep command:

[root@thinkserver:/vmfs/volumes/5c73314d-b5809724-22f9-000423cb2c48/Ubuntu (1)] grep vmdk *.vmx scsi0:0.fileName = "Ubuntu (1)-000010.vmdk" [root@thinkserver:/vmfs/volumes/5c73314d-b5809724-22f9-000423cb2c48/Ubuntu (1)]

Re: ESXi 6.7 snapshots have filled my datastore SSD?

JohnyBeGood — Sun, 24 Apr 2022 17:02:34 GMT

@a_p_ wrote:
>>> You want file listing on each VM folder on first datastore?
Yes, that's what I'm thinking of. This will help to find out how much disk space will be required to consolidate the snapshots, and to find out whether there's an option to free up enough disk space by e.g. temporarily shutting down other VMs.
If that doesn't help, we can clone the Ubuntu VMs virtual disk to the second datastore, and in the same step consolidate all snapshots.
André

I guess that is the easiest option to move Ubuntu to 2nd datastore. I can do that by simply using export option on the 1st datastore and importing into the 2nd?

Is attached export option correct way to do it?

Re: ESXi 6.7 snapshots have filled my datastore SSD?

a_p_ — Sun, 24 Apr 2022 17:09:25 GMT

Yes, Export/Import is yet another option with the benefit that you can test the imported VM before you delete the old one.

André

Re: ESXi 6.7 snapshots have filled my datastore SSD?

ksujay — Sun, 24 Apr 2022 19:41:02 GMT

Can you share the output for the below from the vmfolder?

grep -i DiskLib_IsCombinePossible vmware.log

Re: ESXi 6.7 snapshots have filled my datastore SSD?

JohnyBeGood — Sun, 24 Apr 2022 22:26:40 GMT

@ksujay wrote:
Can you share the output for the below from the vmfolder?
grep -i DiskLib_IsCombinePossible vmware.log

When I run that command I grt blank output?

[root@thinkserver:/vmfs/volumes/5c73314d-b5809724-22f9-000423cb2c48] cd "Ubuntu (1)" [root@thinkserver:/vmfs/volumes/5c73314d-b5809724-22f9-000423cb2c48/Ubuntu (1)] grep -i DiskLib_IsCombinePossible vmware.log [root@thinkserver:/vmfs/volumes/5c73314d-b5809724-22f9-000423cb2c48/Ubuntu (1)]

Re: ESXi 6.7 snapshots have filled my datastore SSD?

JohnyBeGod — Tue, 26 Apr 2022 22:05:24 GMT

Finally got it solved by exporting the image from ESXi and importing it into 2nd datastore. Surprisingly it did not complain during the export process regarding low space.

Thank you all!