Mounting a snapshot delta to recover data

[neogeek83]

I have a VM with 2 virtual drives attached (/Volumes/externalHDD/CentOS6.4_x64.vmwarevm/Virtual Disk.vmdk and /Users/user1/Desktop/extra3.vmdk ). As you can see the extra disk is outside of the VM itself. I made the mistake of making a backup copy of the VM by copying it to another external hard drive that had the same label (/Volumes/externalHDD/CentOS6.4_x64.vmwarevm). Over the last few months I have been working off the original VM, I made a number of snapshots and removed a couple snapshots.

Yesterday, I realized I had cleared off some logs from the VM that weren't captured in any snapshots I had on the main VM... So I thought, no problem, I have the backup. I unplugged the external hdd with the original vm, plugged in the backup HDD, fired up the VM, grabbed the files I needed, then shut it back down. No problems so far, everything came up happily. When I tried to start up the original VM, however, it failed to start, giving me this error in vmware.log:

Worker#1| I120: DISKLIB-CHAIN : "/Users/user1/Desktop/extra3.vmdk" : failed to open (The parent virtual disk has been modified does not match the corresponding parent content ID in the child (18).

At this point I realized that because I had taken the snapshots, and I now have a disk chain for extra3.vmdk inside /Volumes/externalHDD/CentOS6.4_x64.vmwarevm... Looks like there are 2 snapshots:

/Volumes/externalHDD/CentOS6.4_x64.vmwarevm/extra3-000002.vmdk

/Volumes/externalHDD/CentOS6.4_x64.vmwarevm/extra3-000001.vmdk

Before attempting to fix anything, I have backed up both VMs and extra3.vmdk to a 3rd external media. I then did some googling and came across these two pages:

VMware KB:     Resolving the CID mismatch error: The parent virtual disk has been modified since the...

VMDK-Handbook-Basics

I determined my chain is (from mac terminal: head -n 15 extra3xxxxx.vmdk😞

extra3.vmdk (60 GB)	extra3-000002.vmdk (130 MB)	extra3-000001.vmdk (1.7GB)
version=1 encoding="UTF-8" CID=fa8de35e parentCID=ffffffff isNativeShanpshot="no" createType="monothinicSparse" #Extent description RW 209715200 SPARSE "extra3.vmdk"	version=1 encoding="UTF-8" CID=5c11353b parentCID=60a4e855 isNativeShanpshot="no" createType="monothinicSparse" parentFileNameHint="/Users/user1/Desktop/extra3.vmdk" #Extent description RW 209715200 SPARSE "extra3-000002.vmdk"	version=1 encoding="UTF-8" CID=5c11353b <<-- Not sure why this is the same as parent, as file is 1.7GB, but assume isn't an issue? parentCID=5c11353b isNativeShanpshot="no" createType="monothinicSparse" parentFileNameHint="/Users/user1/Desktop/extra3-000002.vmdk" #Extent description RW 209715200 SPARSE "extra3-000001.vmdk"

The parent / child CIDs match up between the snaps, but the base has has a different CID when compared the parentCID of it's first child (in red). I assume this is because some background process on the backup VM must have written to the disk when I started up the backup VM, thinking it had exclusive use of the base vmdk. awesome...

My goal:

  At this point I just want to get as much of the data from the 2 snapshots as I can... There are a bunch of new log files and a GIT repo I would really like to be able to recover from those delta snaps.

Attempt 1:

  So, on the off change it might work, after backing everything up. I tried just changing the parentCID in extra3-000002.vmdk to fa8de35e.  I used vim in mac terminal and pressed ctrl+c to keep entire monolithic vmdk from attempting to be loaded.

Result 1:

   VM booted, cool! But when I tried to access the second HDD, not so happy.

[user1@localhost ~]$ cd /extra3

[user1@localhost extra3]$ ls -l

ls: reading directory .: Input/output error

[user1@localhost extra3]$ dmesg

EXT4-fs error (device sdb1): ext4_find_entry: reading direcotry _#2 offset 0

sd 2:0:1:0: [sdb] Unhandled sense code

sd 2:0:1:0: [sdb] Result: hostbyte=DID OK driverbyte=DRIVER_SENSE

sd 2:0:1:0: [sdb] Sense Key : Medium Error [current]

sd 2:0:1:0: [sdb] Add. Sense: Unrecovered read error

sd 2:0:1:0: [sdb] CDB Read(10): 28 00 00 01 29 08 00 00 00 08 00

So then I figured I could try to fix the filesystem...

[user1@localhost ~]$ sudo umount /extra3

[user1@localhost ~]$ fsck.ext4 -cDfty -C 0 /dev/sdb1

[user1@localhost ~]$ e2fsck /dev/sdb1 -y

[user1@localhost ~]$ sudo mkfs.ext4 -s /dev/sdb1

Didn't help much, "fixed" a bunch of bad blocks, connected most inodes to lost+found. I was able to get into the filesystem after that, but everything had been moved to /extra3/lost+found/_#XXXXXXX (where XXXXX is was the block number) some of them seemed to be files that I could access, but it looked like none of the data written to either of the snapshots was recovered.

Additional Setup Details: 

The host:

     MacBook Pro /w Fusion 5.0.0

The Guest:

     CentOS 6.4 x64

So, can anyone help me get the data off those snapshot delta vmdks?  continuum or a.p., in looking over other vmdk posts, looks like if anyone can help me, one of you are my best chances...

Thanks a bunch in advance!