Re: ESXi 6.7 snapshots have filled my datastore SS...

JohnyBeGood · ‎04-23-2022

Hey all,

I have Ubuntu VM and over the time I would create snapshots. Now I noticed that thin provisioned disk of 80GB and in ESXi it shows that 215GB is being used. Ubuntu terminal shows only 48GB is being used.

I deleted all snapshots and thought that would clear it up but it didn't. I've noticed that VM folder on the host has multiple Ubuntu (1)-000001-sesparse.vmdk, Ubuntu (1)-000002-sesparse.vmdk etc. and are all large files.

Any suggestion how to safely free up this space? Screenshots are attached for the visual.

Thanks!

a_p_ · ‎04-23-2022

To understand the current situation, and to find possible solutions, please post a complete file listing (the text output of ls -lisa). In addition to this post the free disk space in this and other datastores. Last but not lease, attach the VM's vmware.log file to your next reply. This will show the snapshot chain.

André

ksujay · ‎04-23-2022

If you have space in this datastore or any other shared storage, you may clone this VM, which should remove and consolidate the snapshots.

Note: Once cloned, power off the old VM and then power on the new VM to avoid any IP conflict.

Please mark this as a solution if this helps.

--------------------------------------------------------------

Although I am a VMware employee I contribute to VMTN voluntarily (i.e., not in any official capacity)

JohnyBeGood · ‎04-23-2022

@a_p_ wrote:
To understand the current situation, and to find possible solutions, please post a complete file listing (the text output of ls -lisa). In addition to this post the free disk space in this and other datastores. Last but not lease, attach the VM's vmware.log file to your next reply. This will show the snapshot chain.
André

Thanks for the reply!

Today I noticed that Ubuntu won't start due to low storage on Samsung 1TB SSD datastore.

Here's all requested info:

[root@thinkserver:~] ls -lisa
total 817
      1      4 drwxr-xr-x    1 root     root           512 Apr 24 00:35 .
      1      4 drwxr-xr-x    1 root     root           512 Apr 24 00:35 ..
   5309      4 -rw-------    1 root     root             9 Apr 24 00:35 .ash_history
   4484      4 -r--r--r--    1 root     root            20 Aug  5  2019 .mtoolsrc
   5302      4 lrwxrwxrwx    1 root     root            49 Apr 23 07:03 altbootbank -> /vmfs/volumes/36e890e3-a7b7f574-a021-9d6a2d7bf611
     10      4 drwxr-xr-x    1 root     root           512 Apr 23 07:03 bin
   5301      4 lrwxrwxrwx    1 root     root            49 Apr 23 07:03 bootbank -> /vmfs/volumes/fefc22e2-8a8ba970-f2c3-bb5e0935f260
   4486    336 -r--r--r--    1 root     root        342277 Aug  5  2019 bootpart.gz
   4485    224 -r--r--r--    1 root     root        226641 Aug  5  2019 bootpart4kn.gz
      1      0 drwxr-xr-x   16 root     root           512 Apr 24 00:35 dev
      2      4 drwxr-xr-x    1 root     root           512 Apr 23 07:03 etc
     13      4 drwxr-xr-x    1 root     root           512 Apr 23 07:03 lib
     12      4 drwxr-xr-x    1 root     root           512 Apr 23 07:03 lib64
   5300     28 -r-x------    1 root     root         25635 Apr 23 07:01 local.tgz
   5305      4 lrwxrwxrwx    1 root     root             6 Apr 23 07:03 locker -> /store
   4483      4 drwxr-xr-x    1 root     root           512 Apr 23 07:03 mbr
      3      4 drwxr-xr-x    1 root     root           512 Apr 23 07:03 opt
72016948364192    128 drwxr-xr-x    1 root     root        131072 Apr 24 00:35 proc
   5306      4 lrwxrwxrwx    1 root     root            29 Apr 23 07:03 productLocker -> /locker/packages/vmtoolsRepo/
   1755      4 lrwxrwxrwx    1 root     root             4 Aug  5  2019 sbin -> /bin
   5304      4 lrwxrwxrwx    1 root     root            57 Apr 23 07:03 scratch -> /vmfs/volumes/5c73314d-b5809724-22f9-000423cb2c48/.locker
   5303      4 lrwxrwxrwx    1 root     root            49 Apr 23 07:03 store -> /vmfs/volumes/5c756fc2-6d57025c-ca7a-000423cb2c48
      5      4 drwxr-xr-x    1 root     root           512 Apr 23 07:03 tardisks
      6      4 drwxr-xr-x    1 root     root           512 Apr 23 07:02 tardisks.noauto
  38145      4 drwxrwxrwt    1 root     root           512 Apr 24 00:29 tmp
     14      4 drwxr-xr-x    1 root     root           512 Apr 23 07:03 usr
      4      4 drwxr-xr-x    1 root     root           512 Apr 23 07:03 var
     18      4 drwxr-xr-x    1 root     root           512 Apr 23 07:03 vmfs
     16      4 drwxr-xr-x    1 root     root           512 Apr 23 07:03 vmimages
   1762      4 lrwxrwxrwx    1 root     root            18 Aug  5  2019 vmupgrade -> /locker/vmupgrade/
[root@thinkserver:~]

[root@thinkserver:~] esxcli storage filesystem list
Mount Point                                        Volume Name          UUID                                 Mounted  Type            Size          Free
-------------------------------------------------  -------------------  -----------------------------------  -------  ------  ------------  ------------
/vmfs/volumes/5c73314d-b5809724-22f9-000423cb2c48  Samsung 1TB SSD      5c73314d-b5809724-22f9-000423cb2c48     true  VMFS-6  999922073600   10844372992
/vmfs/volumes/5ed2d190-cc7cfe51-e757-000423cb2c48  Samsung 1TB SSD 2nd  5ed2d190-cc7cfe51-e757-000423cb2c48     true  VMFS-6  999922073600  141553565696
/vmfs/volumes/fefc22e2-8a8ba970-f2c3-bb5e0935f260                       fefc22e2-8a8ba970-f2c3-bb5e0935f260     true  vfat       261853184     106074112
/vmfs/volumes/5c756fc2-6d57025c-ca7a-000423cb2c48                       5c756fc2-6d57025c-ca7a-000423cb2c48     true  vfat       299712512     117465088
/vmfs/volumes/36e890e3-a7b7f574-a021-9d6a2d7bf611                       36e890e3-a7b7f574-a021-9d6a2d7bf611     true  vfat       261853184     107630592
[root@thinkserver:~]

JohnyBeGood · ‎04-23-2022

@ksujay wrote:
If you have space in this datastore or any other shared storage, you may clone this VM, which should remove and consolidate the snapshots.
Note: Once cloned, power off the old VM and then power on the new VM to avoid any IP conflict.
Please mark this as a solution if this helps.
--------------------------------------------------------------
Although I am a VMware employee I contribute to VMTN voluntarily (i.e., not in any official capacity)

Unfortunately both 1TB datastores are full. The reason why I got another 1TB is because first one was filled. Now I wonder if this is ESXi bug?

a_p_ · ‎04-24-2022

Sorry if I wasn't quite clear with what I wanted.

Although it looks like the issue can be solved with cloning the VM's virtual disk to the second datastore, there may be an easier way.
What I need is the file listing for the files in the VM's folder, as well as one for all of the other VM's folders/files on the filled up (first) datastore.

The reason why you datastore fills up, is that you seem to keep multiple snapshots for a long time. Please remember that each single snapshot can grow up to its parent virtual disk's provisioned size. Also keep in mind that snapshots are not a replacement for backups!

André

ksujay · ‎04-24-2022

This is not any ESXi bug. The snapshot file continues to grow in size when it is retained for a longer period. This can cause the snapshot storage location to run out of space and impact the system performance. Do not use a single snapshot for more than 72 hours is what as recommended (https://kb.vmware.com/s/article/1025279)

Are you getting the consolidate option in snapshot manager? If yes, any error messages in the consolidate process?

Also please share the output for (grep vmdk *.vmx) from the VM folder.

---------------------------------------------------------------------------------------------------------

Although I am a VMware employee I contribute to VMTN voluntarily (i.e., not in any official capacity)

JohnyBeGood · ‎04-24-2022

@a_p_ wrote:
Sorry if I wasn't quite clear with what I wanted.
Although it looks like the issue can be solved with cloning the VM's virtual disk to the second datastore, there may be an easier way.
What I need is the file listing for the files in the VM's folder, as well as one for all of the other VM's folders/files on the filled up (first) datastore.
The reason why you datastore fills up, is that you seem to keep multiple snapshots for a long time. Please remember that each single snapshot can grow up to its parent virtual disk's provisioned size. Also keep in mind that snapshots are not a replacement for backups!
André

I'm at the point where there's not enough free space to do anything. I was using snapshots before installing various projects ie. nextcloud, SyncThing etc. in case something goes wrong I can revert back, but I guess at some point I left them longer than 72hrs and I'm this situation.

Can you please elaborate on this "What I need is the file listing for the files in the VM's folder, as well as one for all of the other VM's folders/files on the filled up (first) datastore." You want file listing on each VM folder on first datastore?

a_p_ · ‎04-24-2022

>>> You want file listing on each VM folder on first datastore?

Yes, that's what I'm thinking of. This will help to find out how much disk space will be required to consolidate the snapshots, and to find out whether there's an option to free up enough disk space by e.g. temporarily shutting down other VMs.

If that doesn't help, we can clone the Ubuntu VMs virtual disk to the second datastore, and in the same step consolidate all snapshots.

André

JohnyBeGood · ‎04-24-2022

@ksujay wrote:
This is not any ESXi bug. The snapshot file continues to grow in size when it is retained for a longer period. This can cause the snapshot storage location to run out of space and impact the system performance. Do not use a single snapshot for more than 72 hours is what as recommended (https://kb.vmware.com/s/article/1025279)
Are you getting the consolidate option in snapshot manager? If yes, any error messages in the consolidate process?
Also please share the output for (grep vmdk *.vmx) from the VM folder.

---------------------------------------------------------------------------------------------------------
Although I am a VMware employee I contribute to VMTN voluntarily (i.e., not in any official capacity)

I am getting notice to start consolidate process, but when it starts it says not enough space to complete.

Here are output of the grep command:

[root@thinkserver:/vmfs/volumes/5c73314d-b5809724-22f9-000423cb2c48/Ubuntu (1)] grep vmdk *.vmx
scsi0:0.fileName = "Ubuntu (1)-000010.vmdk"
[root@thinkserver:/vmfs/volumes/5c73314d-b5809724-22f9-000423cb2c48/Ubuntu (1)]

JohnyBeGood · ‎04-24-2022

@a_p_ wrote:
>>> You want file listing on each VM folder on first datastore?
Yes, that's what I'm thinking of. This will help to find out how much disk space will be required to consolidate the snapshots, and to find out whether there's an option to free up enough disk space by e.g. temporarily shutting down other VMs.
If that doesn't help, we can clone the Ubuntu VMs virtual disk to the second datastore, and in the same step consolidate all snapshots.
André

I guess that is the easiest option to move Ubuntu to 2nd datastore. I can do that by simply using export option on the 1st datastore and importing into the 2nd?

Is attached export option correct way to do it?

a_p_ · ‎04-24-2022

Yes, Export/Import is yet another option with the benefit that you can test the imported VM before you delete the old one.

André

ksujay · ‎04-24-2022

Can you share the output for the below from the vmfolder?

grep -i DiskLib_IsCombinePossible vmware.log

JohnyBeGood · ‎04-24-2022

@ksujay wrote:
Can you share the output for the below from the vmfolder?
grep -i DiskLib_IsCombinePossible vmware.log

When I run that command I grt blank output?

[root@thinkserver:/vmfs/volumes/5c73314d-b5809724-22f9-000423cb2c48] cd "Ubuntu (1)"
[root@thinkserver:/vmfs/volumes/5c73314d-b5809724-22f9-000423cb2c48/Ubuntu (1)] grep -i  DiskLib_IsCombinePossible vmware.log
[root@thinkserver:/vmfs/volumes/5c73314d-b5809724-22f9-000423cb2c48/Ubuntu (1)]

JohnyBeGod · ‎04-26-2022

Finally got it solved by exporting the image from ESXi and importing it into 2nd datastore. Surprisingly it did not complain during the export process regarding low space.

Thank you all!

All

ESXi 6.7 snapshots have filled my datastore SSD?