Snapshot issues after running out of space on Virtual Mode RDM

[beagle_jbl]

We have a windows file server that ran out of disk space for it's virtual mode RDM.  The server had several outstanding snapshots (VCB-created).  The LUN was expanded and the serve hung and was rebooted.  Upon reboot, none of the snapshots are loaded - just the original VMDK.  None of the snapshots show up in snapshot manager.

Can the sanpshots be removed (commit all) or is there likely corruption in the last snapshot disk?  Would it be best to remove the last snapshot and remove all.  Has anyone done this to themselves before?

Thanks in advance for any insight you can share.

[ThompsG · Accepted solution]

Hi,

Just wondering if you would mind answering some questions before we tackle this one?

When you say

server that ran out of disk space for it's virtual mode RDM

do you mean the datastore that is hosting the virtual RDM file or the actual RDM volume?

Can you also check your VI client and see what VMDK's the virtual machine thinks it is running off? i.e. servername.vmdk or servername-000001.vmdk

Thanks and kind regards,

Glen

[beagle_jbl]

Hey Glen, thanks for the reply.  The RDM itself ran out of space.  Some of the IT guys were working with VMWare on Monday and modified the VM so that it was only accessing the base VMDK.  In talks Wednesday, according to VMWare it isn't possible for us to recover the data in the delta files from the snapshots.  I've been working with VI for a while and have never run into anything like this.  I'd be curious if anyone had a method for rescuing the data in these files, but I think the decision has been made to restore a previous backup and live with a day or so of data loss.  I've never been a big fan of Virtual Mode RDM's but I am much less so now.  We're implementing a script to check for snapshots older than a few days.  That should help us avoid this issue.

Thanks for your time.

[ThompsG]

Hi,

No I've never been a fan of RDM's either because unless everybody is aware of them you can get yourself into some serious trouble.

Obviously there is no way now to retreive the data from the delta files as these are just block changes. At the time I'm sure you could have rolled the deltas back into the RDM by running vmware-cmd /vmfs/volumes/UUID/VMNAME/VMNAME.vmx removesnapshots. Another option, if you had space, would have been to clone the VM and this would have consolidated all the snapshots - either way it would have been a calculation at how important the delta data is vs how long it would take to consolidate and potentially restore if it went wrong.

Kind regards.

[continuum]

Stupid question: how can a RDM run out of space if it has snapshots ?

You mean the virtual disk seen by the guest was running out of space ?

If you expanded the RDM later the snapshots no longer fit the basedisk.
To fix that you want to expand the snapshots and then consolidate the whole chain into a new vmdk.

Every other approach will make you loose the data created during the time the last snapshots were in use.

> Some of the IT guys were working with VMWare on Monday and modified the  VM so that it was only accessing the base VMDK.

Why on earth did they do that ???

[beagle_jbl]

Yes, it was the Virtual Disk presented to the guest that was out of space.

The LUN was expanded while in snapshot mode (big no-no).  After a short time this caused the VM OS to hang and the system would not reboot.

There are probably a number of things that could have been done at this point, but too many other things happened that caused the data to become unrecoverable.

Lessons Learned:  Monitor VM's that have snapshots and remove if they shouldn't be there.... and NEVER resize a LUN for a Virtual Mode RDM while the VM is in snapshot mode. 

Thanks for your input on this one.