One of our largest VMguests had a snapshot created and then the technician forgot about the snapshot. I found out and tried to delete it only problem is we ran out of space on the datastore.
The VMguest recide on direct storage and my thought was to move the snapshot to another datastore (iSCSI).. I have searched for articles about this for 2 days now and cannot find any that fits my issue.. The only articles i find is about changing workdir for future snapshots (have tried this for fun and it does work in my test env)..
When i tried the Delete-All snapshots the datastore had a little more than 6TB free so i thought it would be ok since the snapshot size is 3.3TB.. But now the datastore is completly full and the process failed at 59% .. Now i cannot see the snapshot anymore in the Snapshot Manager but i get the message:
"This virtual machine needs to have its disk consolidated" ..
Any suggestions how to proceed? I cannot clone it or export it to another datastore since i do not have any with that size available. Thats why i was thinking to move the snapshot to iSCSI storage and then try consolidate.. Is it possible?
Running ESXi 7.0 U3i
Guest OS has a thin disk with 10.95TB configured.
Local Datastore has capacity of 12.97TB
Snapshot is 3.3TB
All other machines and ISO files has been moved away from the host.
ls -lisa:
109052164 3522483200 -rw------- 1 root root 3623893360640 May 13 09:30 Guest1-000001-sesparse.vmdk
113246212 10273060864 -rw-r--r-- 1 root root 12043088297984 May 12 07:26 Guest1-flat.vmdk
df -h
Filesystem Size Used Available Use% Mounted on
VMFS-6 13.0T 12.9T 126.7G 99% /vmfs/volumes/datastore1
>>> When i tried the Delete-All snapshots the datastore had a little more than 6TB free so i thought it would be ok since the snapshot size is 3.3TB.
That's indeed strange. That should have worked with that much free disk space.
Anyway, there are unfortunately not many options available at this time.
Consolidating - as suggested by another user - is no option due to the low free disk space.
Moving the snapshot file to another datastore is also something that I wouldn't want to do. That was possible with the old snapshot format (delta files), but I'd keep my fingers off the sesparse files.
In the current situation, an option would be to backup, and restore the VM. Which backup application do you use?
André
Hello @Meph79 ,
There is 1 snapshot present on the disk which is 000001.vmdk. The snapshot consolidation requires almost 25-30% of the disk size and Snapshot size to complete.
As the Datastore is 100% full and since it is local Datastore there is no possibility of expanding the Datastore which leaves us with 2 options only which are mentioned below.
1. We need to clone the snapshot disk from Esxi Putty session. This command will clone and create a new disk on other datastore which has enough free space. Post this operation there will only be 1 new disk which will be used and the old disk on local datastore needs to be deleted.
2. We clone the complete VM using Vmware Convertor which will create a copy of the VM.
Both the options will also consolidate the disks.
As this activity involves risking data please get the command checked before processing.
Hi! Thanks for your answer!
The datastore became full after i ran the snapshot delete all.. So i guess much of the data were comitted to the flat image and made it grow in size..
All i have available is a iSCSI volume with 5TB free. I do not think that is enough for a clone.
My thougt was to move the snapshot to this iSCSI datastore and then try to consolidate the disks again. Is that possible? The VM is currently turned off.
Did you try disk consolation
Hi,
This is a snapshot with sesparse, meaning a size bigger than 2Tb.
For you to consolidate, you will need datastore space. Is not possible to do that without datastore space.
If you do not have space in that datastore(or cannot increase the size of that datastore) you can clone the VM to another datastore.
>>> When i tried the Delete-All snapshots the datastore had a little more than 6TB free so i thought it would be ok since the snapshot size is 3.3TB.
That's indeed strange. That should have worked with that much free disk space.
Anyway, there are unfortunately not many options available at this time.
Consolidating - as suggested by another user - is no option due to the low free disk space.
Moving the snapshot file to another datastore is also something that I wouldn't want to do. That was possible with the old snapshot format (delta files), but I'd keep my fingers off the sesparse files.
In the current situation, an option would be to backup, and restore the VM. Which backup application do you use?
André
@a_p_
Ok, thanks. Then i keep my fingers of the sesparse files ![]()
Is it possible to Export the machine and then import it back? (With ESXi Host Client, Chrome browser)
We use Veeam as backup but this server im having problem with IS the Veeam server.. LOL...
I found the answer in another thread.. Answered by YOU...
.. So i guess thats the way i have to go then.. That will take a LONG time i suspect...
answer by a.p.
"VM based backup application as well as OVF exports use the latest state (snapshot) of the VM.
The backup/OVF basically consolidates the snapshots.
If you want to preserve the snapshots you'll need to manually download the files from the datastore.
André"
@RajeevVCP4
Yes i did. Nothing happened.. The snapshot file were still there .. It reported it successful after a few minutes only.. But i cannot see anything in any log ...
Until you do not provide space to that datastore, you cant consolidate those disks.
@a_p_ Export in progress.. speeds around 15MB/s.. 34GB downloaded so far.. 6,41TB free on target that should be enough since about 3-4TB were used on the server in question.. so.. about 3 days to make the backup and then 3 days to copy it back.. really hope this works..
Do you have any experiences with exporting this much data?
Sorry, but I've never exported/imported such a large image.
André
@a_p_ me neither..
Do you want me to update this thread about the result later? The export is now at 890GB and still running
A little slower than before at 9MB/s right now, but i expect the speed to vary.. it usually does.. Should have used the OVFtool instead of Chrome though... but too late now ![]()
Yes, an update about the results would be interesting.
André
Been running for 33 hours now. 1,2TB downloaded so far.. Average speed 10,6 MB/s
EDIT:
62 hours 1,9TB 8.93MB/s
106 hours 2,7TB 7.41MB/s avg speed.
@a_p_ Export seemed to have been succesful.. I'm doing an import now.. it goes a lot faster.. around 230Mbit/s...
Export:
Start: 17/5 10:03
End: 22/5 17:54
End file Size: 3,02 TB
Time hours: 128 hours
Speed: 6,87 MiB/s (57,6Mbit/s) (No VMs running on host)
VMware ESXi 7.0.3 build-20842708
@a_p_
Import:
Start: 22/5 21:22
End: 24/5 05:11
OVF file size: 3,02TB
Time hours: 32
Speed: 27,49 MB/s (230 Mbit/s)
Good news! The import was successful!!! Thanks for your support. In total it took 7 days but we do not need to reinstall the server and backupsoftware and everything wich would have taken a few days i guess..
Used OVF tool to do the import. It seemed a little slower than using Chrome but i guess more stable and reliable. Chrome did give 400-500Mbit/s in import speed but it was going up and down a lot.. OVFtool kept speed at a constant 230Mbit/s..
Glad it worked, and thanks for the feedback.
André
