out of space on a datastore, have big snapshot to commit

[brandilton]

Through a series of bad luck and automated tasks, here's where i'm stuck. looking for advice on best way to fix it.

ESXi server with local datastore

Server A (Blackberry Server, not totally important and have a good backup, worst case I could rebuild without too much pain)

• 30 GB thin provisioned

Server B (Production File server and Production Exchange server):

• 2 Virtual Hard Drives

• Hard Disk 1: 80GB Thin Provisioned

• Hard Disk 2: 500GB Thick Provisioned

• somehow there is a snapshot on this disk that is 196GB (that does have some needed stuff on it)

• Vsphere client does not show that there is a snapshot, but browsing the datastore, it's there

I'm running very low on space (30GB free) on the datastore. I do have a 1TB NAS device (that does NFS) but it's mostly full with other vm backups including the only good backup of server B that i'd prefer not to wipe.

Will this snapshot commit with this little free space? if so, how given that vsphere client shows that there is no snapshot?

Looking to the pros for a good solution/order to tackle issues in. I am of course headed out of town for a week and am hoping the 30GB i have free will last while i'm gone but will keep you posted along the way.

[idle-jam]

You have a very large snapshot and 30GB would not be suffice to commit all. If you could find some temp. free space you could do a storage vmotion to commit the snapshot and storage vmotion back.

iDLE-jAM | VCP 2, VCP 3 & VCP 4

If you found this or any other answer useful please consider the use of the Helpful or correct buttons to award points.

[a_p_ · Accepted solution]

In my opinion there shouldn't be any need of a lot of additional disk space for deleting a snapshot on a thick provisioned disk. To delete the snapshot, create another snapshot on the VM, then open the snapshot manager and select "Delete All". This will commit all snapshots to the base disk, no matter whether they are shown in the snapshot manager or not. Due to the size the deletion my take several hours (depending on your storage speed)

With the one snapshot you have, this is no issue, however if there were multiple snapshots you'd have to be careful, depending on the version/build of ESX(i) installed. Make sure you don't run out of disk space during the commit process of the snapshot, due to e.g. a lot of changes to a thin provisioned VM. To be on the save side, you should do the deletion either while the VM is not utilized much or - even better - while the VM is powered off.

André

[AndreTheGiant]

If you cannot free enough space and if you can take the VM offline... a slow way could be use the Converter to export the VM to a share, delete the VM and then reimport again.

In this way the snapshots are consolidated to the last version.

Andre

[IRIX201110141]

Instead of waste time with deleting a snapshot on a full Datastore i would clone the VMDK into a new one. With this method you consolidate all Snaps into one recend single VMDK.

Open a SSH to your ESX/ESXi and create a folder on your NFS Datastore go into the created folder

1. vmkfstools -i /path/toold/vm/foobar.vmdk ./foobar.vmdk

2. cp /path/toold/vm/foobar.vmx ./

3. Open the VMX and remove the line "sched.swap.derivedName = ...."

4. De-register the old one and register the new one by importing the VMX with the help of the Datastore Browser. After that you can start the VM.

Notes:

• vmkfstools is also available in the ESXi busybox

• The VM have to be shutdown during the clone process

Hints:

• If you dont have a Datastore with enough space you can setup a Laptop together with a external USB Disk and setup Starwinds iSCSI Software within minutes and present a iSCSI Target to your ESX(i).

Regards

Joerg

'Remember if you found this or others answers helpful do not forget to award points by marking an answer as helpful or correct'

[brandilton]

My plan is to use this method.  Is there a way i can also make this disk thin provisioned at the same time (it is currently thick)?  would it be "-d thin"

To clarify the whole thing and because I'm terrible with linux commands, would it be:

1. vmkfstools -i /vmfs/volumes/datastore1/servername/disk2.vmdk /vmfs/volumes/nasserver/servername/disk2new.vmdk -d thin
2. cp /vmfs/volumes/nasserver/servername/disk2new.vmdk /vmfs/volumes/datastore1/servername/disk2.vmdk
3. Open /vmfs/volumes/datastore1/servername/servername.vmx and remove the line "sched.swap.derivedName = ...."
4. open vSphere client:
   1. right click on servername>remove from inventory. 
   2. go to configuration>storage>datastore1 (browse)
   3. browse to /servername/servername.vmx, right click>add to inventory

[a_p_]

Why don't you just shut down the Exchange server and commit the snapshot?

The steps you mentioned will not work this way:

1. You are going to clone the virtual disk to a NAS device. NAS devices are no block devices and therefore the "-d thin" is not useless
   
2. If you go this way you have to specify the snapshot vmdk as the source or you will loose all data on this snapshot. (e.g. vmkfstools -i /<source>/Server_1-000001.vmdk /<target>/newdisk.vmdk
   
3. Before moving the clone back to the source disk you have to delete the "Server_1*.vmdk files in order to free up the disk space you need.
   
4. Instead of using the cp command use the "vmkfstools -i" command again to clone the newdisk.vmdk back to the source Server_1.vmdk disk. In this command you may add the "-d thin" to get a thin provisioned disk on the production datastore
   
5. No need to modify the vmx file (swap), this is only necessary if you would move the vmx file to a different destination.
   
6. After doing all the cloning, open the settings of the VM and replace the "Server_1-000001.vmdk" with "Server_1.vmdk" for the second HDD
   

Be sure to have a current backup in place in case something goes wrong.

Now when you look at all these steps and consider the time it's going to take, wouldn't it be much easier to only delete the snapshot?

André

[brandilton]

If I go with the "create a snapshot, delete all" method mentioned above wouldn't it create 2 snapshots (3 total) since there are 2 virtual disks and one of the disks already has a snapshot?  then what are the concerns?

however if there were multiple snapshots you'd have to be careful, depending on the version/build of ESX(i) installed

I currently have VMware ESXi 4.0.0 build-244038 (showing 8 patches available).  Should i patch first?

[mwpreston]

Hi,

You do have some other options via command line as well...you can attempt to clone those disks with snapshots to another disk located on another datastore...

Look in the vmx or under edit-settings in client to find what the top level snapshot is and run the follwoing

vmkfstools -i toplevelsnapshot <targetdisk>

You may want to think about powering down, cloning all the affected drives, then re-attaching those drives to that vm and powering back up.

Other than doing that, if you wanted to stay live, just storage vmotion some of the other vm's that are running on the same datastore as the affected vm to another datastore to free up some space...

I've always performed the following when faced with multiple snapshots...

1. vmware-cmd <path_to_vmx> hassnapshot

     If this returns 1 then skip to 3.

2. Create another snapshot on top of all them though the client

3. vmware-cmd <path_to_vmx> removesnapshots

4. Grab a coffee and wait....!!

Good luck.

[Virtugirl]

Whenever Ive had this issue, I have always vmoitoned off another VM to make room for the snapshot to commit.  If you can make the space then take another snapshot and delete all.  It will take some time but it will work.

[brandilton]

Is there a way to tell how much space a snapshot will take up so that i can make sure that I have enough space?  there is only one other VM on the datastore that's around 30GB and there's around 30GB free.  So most I can free up is around 60GB, is there a way to see if that will be enough?

[Virtugirl]

Login into the VI Client

Browse the datastore where your VM resides and open the folder containing your VM.

Your snapshot file will be shown as VMName-000001.vmdk.  You should be able to see the size of the file in right side.

[a_p_]

Correct, since you currently only have one snapshot (on disk 2) another 2 snapshots (Consolidate Helper) would be created when running a "Delete All" if the VM is powered on. However these snapshots will start with only a few MB in size and grow depending on the write activity. If you can afford it - and that's what I would highly recommend in your situation - to shut down the VM and do the snapshot consolidation while the VM is powered off, there would be no risk of running out of disk space (except the other VM would rapidly grow in size).

André

[brandilton]

ok, I can make some time over the holidays to shutdown and do the snapshot consolidation.  Can that all be done from VI client or do i need to go to command line, is this the command when powered down?

vmware-cmd <path_to_vmx> removesnapshots

[a_p_]

No need for the command line at all, everything can be done from the vSphere client (GUI).

André

[Virtugirl]

You dont need to use the command line, just use the client.

[mwpreston]

The only reason I mentioned the command line is that is the way I have always done it…

A few years ago I had an issue where I was consolidating some snapshots through the gui, the snapshots were fairly large, and the gui actually reported a timeout and threw an error…

I continued to keep trying to take a snapshot and consolidate it but everything continued to fail. After a talk with Vmware they said that the first time I deleted the snapshots just reported a timeout and was actually still working on the host.

I’ve just always watched the command line since then, because I don’t see those timeout errors and I can see what is going on by listing the vm files every 5 seconds or so…

However, that was a few years ago and some advancements on the ‘phantom’ timeouts probably have been made…

[brandilton]

Thanks all.  I ended up powering down the vm, creating a new snapshot (took maybe 1 minute), deleting all snapshots (took around 12 hours for 236GB to commit).  up and running again.

[DSTAVERT]

So here is a link to a free tool RVtools Returns a lot of information including which VMs have a snapshot.