VMware Communities: Recovering from broken snapshot chain?

Up to Discussions in VI: ESX 3.5

This Question is Answered

1 "correct" answer available (10 pts) 1 "helpful" answer available (6 pts)

9 Replies Last post: Mar 7, 2007 1:58 PM by SV

Recovering from broken snapshot chain?

Nov 22, 2006 6:05 PM

jones.matt 26 posts since
Jun 20, 2005

Here's the situation: I have modified the base VMDK of a snapshot set, breaking the entire chain.

Here's how I shot myself in the foot:
1. Shutdown VM
2. Extend base VMDK from 10GB to 30GB
3. Attach base VMDK to utility VM and extend volume using diskpart
4. Shutdown utility VM
5. Fail to boot server because of my bonehead move

Is there a way to recover? I've already restored the server, but I need just a few files I know are after the last snapshot. If I can just recover those files, I will just copy them to the newly restored VM manually.

Thanks for any help or words of advice you can provide!

Reply Re: Recovering from broken snapshot chain? Nov 23, 2006 1:40 AM

mittell 3,095 posts since
Apr 25, 2006

This is a difficult one, the answer is it's going to be quite hard to get it working but you might be able to bodge the VM to accept a snapshot delta if you've restored the base VMDK to it's previous un-modified state.

Take a look at this thread: http://www.vmware.com/community/thread.jspa?threadID=62412&tstart=0

Reply Re: Recovering from broken snapshot chain? Nov 24, 2006 10:07 AM

in response to: mittell

jones.matt 26 posts since
Jun 20, 2005

Any tips on how to restore the base VMDK?

Perhaps reduce the NTFS partition to 10GB, then use vmkfstools to shrink the VMDK? Sounds easy in theory, but I don't know exactly where to start.

Reply Re: Recovering from broken snapshot chain? Nov 24, 2006 10:57 AM

in response to: jones.matt

mittell 3,095 posts since
Apr 25, 2006

Unfortunately if you tried to boot the base VMDK it'll probably have been modified, meaning the snapshot deltas cannot be applied. Youir best bet is probably to restore from backup I'm afraid.

Reply Re: Recovering from broken snapshot chain? Nov 24, 2006 7:06 PM

in response to: mittell

jones.matt 26 posts since
Jun 20, 2005

That's the rub.. BackupExec wasn't catching an embedded MySQL database in a piece of software on that server. The vendor's documentation didn't mention any special backup instructions or considerations, and now I'm screwed. I guess I have some bad news to give Monday morning!

On a related note: Would ESXRanger or VCB been able to properly backup the MySQL database while the VM was running and the database in use?

Reply Re: Recovering from broken snapshot chain? Nov 25, 2006 7:07 AM

in response to: jones.matt

soleblazer 228 posts since
May 25, 2006

Databases are always a tricky lot. Technically ESXranger may have helped you here, but the fact of the matter is any database is going to have issues if you simply copy the files hot, without putting it into a certain mode. Oracle has RMAN, etc.

Your best bet may be to simply write a script that runs mysqldump , and dumps to a localfilesystem, that is a flat file, then Ranger or VCB would backup the running VM and the dump.

Reply Re: Recovering from broken snapshot chain? Nov 25, 2006 2:26 PM

jones.matt 26 posts since
Jun 20, 2005

This only worked because I did not modify the disk in ANY way other than stated above!

Solution:

Boot GPartEd ISO in new VM with plain (wwcs-read180.vmdk) drive attached
Shrink partition back to 10GB

Create blank 10GB drive to hold 10GB partition:
vmkfstools -c 10G -d thick wwcs-read180-10g.vmdk

Copy drive contents using DD, bs and count are set for 10GB drive
dd if=wwcs-read180-flat.vmdk of=wwcs-read180-10g-flat.vmdk bs=1048576 count=10240

Move broken drive out of the way, name new drive to old name:
vmkfstools -E wwcs-read180.vmdk wwcs-read180-30gbad.vmdk
vmkfstools -E wwcs-read180-10g.vmdk wwcs-read180.vmdk

Open snapshot depending on broken VMDK, note parentCID variable:
nano wwcs-read180-000001.vmdk
Example: parentCID=b7c9e128

Open fixed VMDK, set CID to value from parentCID.
nano wwcs-read180.vmdk
Example: CID=b7c9e128

Boot Up

Change SCSI type? No

Hooray!

I'll be doing a FULL backup and reinstalling this application soon on a fresh VM, I don't trust it after all the hackery I've put it though.

Helpful for how to shink a VMDK: http://www.vmware.com/community/message.jspa?messageID=423911

Reply Re: Recovering from broken snapshot chain? Nov 26, 2006 10:38 PM

in response to: jones.matt

jhanekom 1,081 posts since
May 13, 2005

Excellent piece of work! Will keep this as a reference, for sure.

I find the whole multi-level VMDK snapshot-thing very innovative but extremely prone to confusion.

Recently had a customer asign both the original VMDK and one of the snapshots to the same virtual machine. Talk about confusion.

Reply Re: Recovering from broken snapshot chain? Dec 21, 2006 10:32 AM

jones.matt 26 posts since
Jun 20, 2005

I have posted a more official writeup on my blog about this thread.

http://www.gorillapond.com/2006/11/26/foot-wounds/

Reply Re: Recovering from broken snapshot chain? Mar 7, 2007 1:58 PM

SV 317 posts since
Oct 1, 2003 VMware

We just published KB article 1646892 on the topic. Just FYI...

http://kb.vmware.com/kb/1646892

Actions

View as PDF

Up to Discussions in VI: ESX 3.5