VMware Cloud Community
friskee
Contributor
Contributor

Error Removing Snapshot

We use vRanger to backup our VM's every Friday. The last two Fridays we have received this error message and out SQL server stops responding!

Remove snapshot

SQLServer

The operation cannot be allowed at the current time because

the virtual machine has a question pending:   'msg.hbacommon.

outofspace:There is no more space for virtual disk

/vmfs/volumes/4e83d0db-81815dcd-0944-001f29600645/SQLAl-

pha64/SQLServer64-000002.vmdk. You might be able to continue

this session by freeing disk space on the relevant volume, and

clicking Retry. Click Cancel to terminate this session. '.

Domain\Administrator

12/12/2013 11:26:29 PM

12/12/2013 11:26:29 PM

12/12/2013 11:26:30 PM

This only started happening after we updated from ESX 4.1 to ESXi5.1. Does anyone know what is happening here?

0 Kudos
3 Replies
continuum
Immortal
Immortal

DO NOT ANSWER THAT QUESTION !!!

If you received this twice this means you have two problems.

1. you are running out of space on a LUN - this is very dangerous and can cost you nerves and valuable data in a good outcome and the whole LUN when you act in a hurry
2. you have a bug in administration setup

When you see any signs of a risk that a LUN may run out of space you MUST ACT.

Ok - that had to be said Smiley Wink

If you have not pushed the button yet - do this:
Find VMs that are not running right now and move them elsewhere to a different datastore - do not delete anything too quickly - I have seen to many bad mistakes in such a scenario ...
Free up some space - how much ?
I would  say give as much space as 2 x size of latest snapshot of the VM that warns + 2 x vRAM of the VM that warns.
Power off other non essential VMs
Try to calm down other running VMs to reduce any diskoperations.
Do not suspend other VMs

Once that is done try the RETRY option and cross your fingers.
Often the damage is already done at this point.

If that still fails do NOT answer again - instead free up more space.

Hopefully you can get out of it now without needing to discard the current state.

Once you are running again - consolidate snapshots. Do not use SnapshotManager to consolidate large snapshots !!! - use vmkfstools instead.

If the VM is damaged it may not start at all or start with an outdated state.
Now do not start the VM again but create a backup of all vmware.logs, all vmdk descriptorfiles, the vmsd and the vmx-file.
Do not assign vmdks to other VMs before you are sure there is no way to fix the VM.
If you have to get data out of vmdks - mount them as non-persistant and use a LiveCD

.Post details if you run into this.

About the administration ...

If ou run vRanger, VEEAM or similar backuptools against VMs that you regard as "mission critical" you NEED a functional warning system when snapshot chains are getting out of control.
Install vcheck or something like that - receive your daily email and read it

If vRanger, VEEAM and friends fail to clean up snapshots they very likely mess up the vmsd file first.
Next snapshotmanager does not display snapshots - the problems no longer looks serious and during the next weeks you will get more and more snapshots until there is no way to  consolidate them safely while the VM is running. A few days or hours later datastore fills up and the data of the last months is lost.
Does not matter - lets restore last snapshot ... oops - last good is months old ....

You do not want to experience this yourself ...

Ulli


________________________________________________
Do you need support with a VMFS recovery problem ? - send a message via skype "sanbarrow"
I do not support Workstation 16 at this time ...

0 Kudos
friskee
Contributor
Contributor

This datastore is only 150GB and it doesn't contain the VM, only a data drive associated with one VM. So it would be hard to clear space. vRanger shouldn't even be backing up this as we have set it to only back up our boot drives where our datastores are. what about increasing the size of the LUN or creating a new larger one and moving the data?

0 Kudos
a_p_
Leadership
Leadership

All image based backup applications create snapshots of VMs, not just individual virtual disks. The only way to avoid snapshots of an individual virtual disk is to set it to "independent-persistent". Anyway, the reason this issue didn't occur with ESXi 4.1, but started with ESXi 5.1 is the fact that VMware changed the default location for snapshots. With ESXi 5.x snapshots of virtual disks are placed in their parent virtual disk's datastore, whereas in earlier versions they were created in the VM's "home" folder (where the .vmx file is located).

So your option is to either increase the datastore, or to follow VMware KB: Changing the location of snapshot delta files for virtual machines in ESXi 5.x

André

0 Kudos