Meph79
Enthusiast
Enthusiast

ESXi 7.0.3i - Snapshot delete failed - Out of diskspace

One of our largest VMguests had a snapshot created and then the technician forgot about the snapshot. I found out and tried to delete it only problem is we ran out of space on the datastore. 

The VMguest recide on direct storage and my thought was to move the snapshot to another datastore (iSCSI).. I have searched for articles about this for 2 days now and cannot find any that fits my issue.. The only articles i find is about changing workdir for future snapshots (have tried this for fun and it does work in my test env).. 

When i tried the Delete-All snapshots the datastore had a little more than 6TB free so i thought it would be ok since the snapshot size is 3.3TB.. But now the datastore is completly full and the process failed at 59% .. Now i cannot see the snapshot anymore in the Snapshot Manager but i get the message:
"This virtual machine needs to have its disk consolidated" .. 

Any suggestions how to proceed? I cannot clone it or export it to another datastore since i do not have any with that size available. Thats why i was thinking to move the snapshot to iSCSI storage and then try consolidate.. Is it possible?

Running ESXi 7.0 U3i 
Guest OS has a thin disk with 10.95TB configured.
Local Datastore has capacity of 12.97TB
Snapshot is 3.3TB 
All other machines and ISO files has been moved away from the host. 

ls -lisa:
109052164 3522483200 -rw------- 1 root root 3623893360640 May 13 09:30 Guest1-000001-sesparse.vmdk
113246212 10273060864 -rw-r--r-- 1 root root 12043088297984 May 12 07:26 Guest1-flat.vmdk

df -h
Filesystem Size Used Available Use% Mounted on
VMFS-6 13.0T 12.9T 126.7G 99% /vmfs/volumes/datastore1

 

Tags (1)
Reply
0 Kudos
AnaghB
Enthusiast
Enthusiast

Hello @Meph79 ,

There is 1 snapshot present on the disk which is 000001.vmdk. The snapshot consolidation requires almost 25-30% of the disk size and Snapshot size to complete. 

As the Datastore is 100% full and since it is local Datastore there is no possibility of expanding the Datastore which leaves us with 2 options only which are mentioned below.

1. We need to clone the snapshot disk from Esxi Putty session. This command will clone and create a new disk on other datastore which has enough free space. Post this operation there will only be 1 new disk which will be used and the old disk on local datastore needs to be deleted.

2. We clone the complete VM using Vmware Convertor which will create a copy of the VM. 

Both the options will also consolidate the disks.

As this activity involves risking data please get the command checked before processing.

 

Anagh B
VCIX-DCV6.5, VSAN Specialist
Please mark help full or correct if my answer is use full for you
Reply
0 Kudos
Meph79
Enthusiast
Enthusiast

Hi! Thanks for your answer!

The datastore became full after i ran the snapshot delete all.. So i guess much of the data were comitted to the flat image and made it grow in size.. 

All i have available is a iSCSI volume with 5TB free. I do not think that is enough for a clone. 

My thougt was to move the snapshot to this iSCSI datastore and then try to consolidate the disks again. Is that possible? The VM is currently turned off. 

Reply
0 Kudos
RajeevVCP4
Expert
Expert

Did you try disk consolation 

Rajeev Chauhan
VCIX-DCV6.5/VSAN/VXRAIL
Please mark help full or correct if my answer is use full for you
Reply
0 Kudos
LucianoPatrão

Hi,

This is a snapshot with sesparse, meaning a size bigger than 2Tb.

For you to consolidate, you will need datastore space. Is not possible to do that without datastore space.

If you do not have space in that datastore(or cannot increase the size of that datastore) you can clone the VM to another datastore.

Luciano Patrão

VCP-DCV, VCAP-DCV Design 2023, VCP-Cloud 2023
vExpert vSAN, NSX, Cloud Provider, Veeam Vanguard
Solutions Architect - Tech Lead for VMware / Virtual Backups

________________________________
If helpful Please award points
Thank You
Blog: https://www.provirtualzone.com | Twitter: @Luciano_PT
Reply
0 Kudos
a_p_
Leadership
Leadership

>>> When i tried the Delete-All snapshots the datastore had a little more than 6TB free so i thought it would be ok since the snapshot size is 3.3TB.
That's indeed strange. That should have worked with that much free disk space.

Anyway, there are unfortunately not many options available at this time.
Consolidating - as suggested by another user - is no option due to the low free disk space.
Moving the snapshot file to another datastore is also something that I wouldn't want to do. That was possible with the old snapshot format (delta files), but I'd keep my fingers off the sesparse files.

In the current situation, an option would be to backup, and restore the VM. Which backup application do you use?

André

View solution in original post

Meph79
Enthusiast
Enthusiast

@a_p_ 
Ok, thanks. Then i keep my fingers of the sesparse files :grinning_face_with_smiling_eyes:

Is it possible to Export the machine and then import it back? (With ESXi Host Client, Chrome browser)

We use Veeam as backup but this server im having problem with IS the Veeam server.. LOL... 

I found the answer in another thread.. Answered by YOU... :slightly_smiling_face: .. So i guess thats the way i have to go then.. That will take a LONG time i suspect...

answer by a.p.
"VM based backup application as well as OVF exports use the latest state (snapshot) of the VM.

The backup/OVF basically consolidates the snapshots.
If you want to preserve the snapshots you'll need to manually download the files from the datastore.
André"

Reply
0 Kudos
Meph79
Enthusiast
Enthusiast

@RajeevVCP4 
Yes i did. Nothing happened.. The snapshot file were still there .. It reported it successful after a few minutes only.. But i cannot see anything in any log ... 

Reply
0 Kudos
LucianoPatrão

Until you do not provide space to that datastore, you cant consolidate those disks.

Luciano Patrão

VCP-DCV, VCAP-DCV Design 2023, VCP-Cloud 2023
vExpert vSAN, NSX, Cloud Provider, Veeam Vanguard
Solutions Architect - Tech Lead for VMware / Virtual Backups

________________________________
If helpful Please award points
Thank You
Blog: https://www.provirtualzone.com | Twitter: @Luciano_PT
Reply
0 Kudos
Meph79
Enthusiast
Enthusiast

@a_p_ Export in progress.. speeds around 15MB/s.. 34GB downloaded so far.. 6,41TB free on target that should be enough since about 3-4TB were used on the server in question.. so.. about 3 days to make the backup and then 3 days to copy it back.. really hope this works.. 

Do you have any experiences with exporting this much data?

 

Reply
0 Kudos
a_p_
Leadership
Leadership

Sorry, but I've never exported/imported such a large image.

André

Reply
0 Kudos
Meph79
Enthusiast
Enthusiast

@a_p_  me neither.. :grinning_face_with_smiling_eyes: 

Do you want me to update this thread about the result later? The export is now at 890GB and still running :slightly_smiling_face: A little slower than before at 9MB/s right now, but i expect the speed to vary.. it usually does.. Should have used the OVFtool instead of Chrome though... but too late now :slightly_smiling_face:

Reply
0 Kudos
a_p_
Leadership
Leadership

Yes, an update about the results would be interesting.

André

Reply
0 Kudos
Meph79
Enthusiast
Enthusiast

Been running for 33 hours now. 1,2TB downloaded so far.. Average speed 10,6 MB/s
EDIT: 
62 hours 1,9TB 8.93MB/s 
106 hours 2,7TB 7.41MB/s avg speed.

 

Reply
0 Kudos
Meph79
Enthusiast
Enthusiast

@a_p_ Export seemed to have been succesful.. I'm doing an import now.. it goes a lot faster.. around 230Mbit/s...

Export:
Start: 17/5 10:03
End: 22/5 17:54
End file Size: 3,02 TB
Time hours: 128 hours
Speed: 6,87 MiB/s (57,6Mbit/s) (No VMs running on host)
VMware ESXi 7.0.3 build-20842708

Reply
0 Kudos
Meph79
Enthusiast
Enthusiast

@a_p_ 
Import:
Start: 22/5 21:22
End: 24/5 05:11
OVF file size: 3,02TB
Time hours: 32
Speed: 27,49 MB/s (230 Mbit/s)

Good news! The import was successful!!! Thanks for your support. In total it took 7 days but we do not need to reinstall the server and backupsoftware and everything wich would have taken a few days i guess.. 

Used OVF tool to do the import. It seemed a little slower than using Chrome but i guess more stable and reliable. Chrome did give 400-500Mbit/s in import speed but it was going up and down a lot.. OVFtool kept speed at a constant 230Mbit/s.. 

Reply
0 Kudos
a_p_
Leadership
Leadership

Glad it worked, and thanks for the feedback.

André

Reply
0 Kudos