cycloxr
Contributor
Contributor

Populate Snapshot manager with missing snapshots or revert to one manually?

Jump to solution

Hello,

Is there a way to populate the Snapshot Manager with snapshots that are listed in the datastore but not under Manage Snapshots.  Is there a way to either populate Manage Snapshots or using something like vmkfstools revert to the snapshot manually?

Long story short, I had a hardware failure and had to re-add the datastore to the Linux VM.  I believe before the failure that I was using Portal-000003.vmdk but when I try to start the system with that datastore I get the following error: Failed to power on virtual machine Portal. File system specific implementation of LookupAndOpen[file...

When I try to using Portal.vmdk it boots up fine, but seems to be running code from 2015.   When I look at the Datastore I have several vmdk files:

-rw-------    1 root     root     21308092416 Jun 23  2016 Portal-000001-delta.vmdk

-rw-------    1 root     root           323 Nov 24  2015 Portal-000001.vmdk

-rw-------    1 root     root     117391208448 Nov 24  2015 Portal-000002-delta.vmdk

-rw-------    1 root     root           330 Nov 24  2015 Portal-000002.vmdk

-rw-------    1 root     root       8192512 Apr 19 05:17 Portal-000003-ctk.vmdk

-rw-------    1 root     root     521604673536 Apr 19 03:28 Portal-000003-delta.vmdk

-rw-------    1 root     root           457 Apr 19 04:35 Portal-000003.vmdk

-rw-------    1 root     root       8192512 Apr 18 17:37 Portal-000004-ctk.vmdk

-rw-------    1 root     root     32146173952 Apr 17 03:03 Portal-000004-delta.vmdk

-rw-------    1 root     root           398 Apr 17 03:02 Portal-000004.vmdk

-rw-------    1 root     root       8192512 Apr 19 05:52 Portal-000005-ctk.vmdk

-rw-------    1 root     root     2198843392 Apr 17 03:07 Portal-000005-delta.vmdk

-rw-------    1 root     root           398 Apr 17 03:07 Portal-000005.vmdk

-rw-------    1 root     root       8192512 Apr 18 17:37 Portal-000006-ctk.vmdk

-rw-------    1 root     root     4413435904 Apr 17 03:15 Portal-000006-delta.vmdk

-rw-------    1 root     root           398 Apr 17 03:10 Portal-000006.vmdk

-rw-------    1 root     root       8192512 Apr 18 17:37 Portal-000007-ctk.vmdk

-rw-------    1 root     root     31961624576 Apr 17 07:43 Portal-000007-delta.vmdk

-rw-------    1 root     root           398 Apr 17 07:10 Portal-000007.vmdk

-rw-------    1 root     root     536870912000 Apr 19 05:57 Portal-flat.vmdk

-rw-------    1 root     root          8684 Apr 19 05:57 Portal.nvram

-rw-------    1 root     root           560 Apr 19 05:56 Portal.vmdk

-rw-r--r--    1 root     root            78 Apr 19 05:40 Portal.vmsd

-rwxr-xr-x    1 root     root          3394 Apr 19 05:57 Portal.vmx

-rw-r--r--    1 root     root           367 Apr 18 16:40 Portal.vmxf

Any suggestions on what else I can do to revert back to the latest good snapshot?

Thanks.

0 Kudos
1 Solution

Accepted Solutions
continuum
Immortal
Immortal

That is one part of the task. I agree with your values.
Next thing is to find the original flat.vmdk.
Looks like you renamed the flat.vmdk to *-delta.vmdk.
To check which one it is run
hexdump -n 512 -C *-delta.vmdk
We need to identify the one that does not start with COWD and ends with 00 00 55 AA

Do you need support with a recovery problem ? - send a message via skype "sanbarrow"

View solution in original post

0 Kudos
18 Replies
johncol
VMware Employee
VMware Employee

You can consolidate/remove them like this with vim-cmd VMware Knowledge Base Judging by the size that may take a while...

0 Kudos
cycloxr
Contributor
Contributor

I'm nearly out of disk space though, how much disk space would consolidation take?

0 Kudos
johncol
VMware Employee
VMware Employee

difficult to tell as there is a bit of overhead - could you free up some space as a precaution, svmotion a vm or 2 onto another datastore?

0 Kudos
continuum
Immortal
Immortal

> When I try to using Portal.vmdk it boots up fine, but seems to be running code from 2015.
Do you have older vmware.log files of this VM ?
Stop using the VM immediatly - you obviously assigned not the correct vmdk.
You need to lookup the order of vmdks in older log-files.

Do you need support with a recovery problem ? - send a message via skype "sanbarrow"
0 Kudos
cycloxr
Contributor
Contributor

continuum​, thanks for the response.  Unfortunately the older vmware.log files don't exist any longer.  

The only snapshot file I care about at this point is the Portal-000003-delta.vmdk, is there anyway to recover/extract that data so I can just rebuilt the system and merge the data over?

Currently if I try to mount the Portal-000003.vmdk file I get a "File system specific implementation of LookupAndOpen[file] failed"

Thank you.

0 Kudos
continuum
Immortal
Immortal

> Currently if I try to mount the Portal-000003.vmdk
STOP DOING THAT - this will corrupt the original snapshot chain.
Please connect to the datastore with WinSCP and download all the vmdk-descriptorfiles (the small vmdks without flat / delta extension.
But first of all stop all mount-attempts

Do you need support with a recovery problem ? - send a message via skype "sanbarrow"
0 Kudos
cycloxr
Contributor
Contributor

ok, I have stopped trying to mount anything at the moment.   I have downloaded and copied all vmdk descriptor files to a backup.  Next steps?

Thank you again, really appreciate the replies.

0 Kudos
continuum
Immortal
Immortal

Backup is nice buty first of all I want to see them to check the correct order.
Attach them to your next reply.

Do you need support with a recovery problem ? - send a message via skype "sanbarrow"
0 Kudos
cycloxr
Contributor
Contributor

continuum​, here you go.  Thanks!

0 Kudos
continuum
Immortal
Immortal

Can I assume that all snapshots from 000008 - 000014 were created after the problem occured ?
They are not listed in your original filelist !
I just noticed that you or someone else made a mistake while editing descriptorfiles !
According to your descriptorfiles the basedisk points to "DMZPortal-000005-delta.vmdk"
That makes no sense and shows that a person in panic-mode was trying to fix this problem.
I would suggest that we setup a remote session via Anydesk or Teamviewer - I need to find the original flat.vmdk with hexdump commands ....
We already spend way too much time on this - pleaase call me via Skype so that we can speed this up.
Watching how to fix issues like this will also be a useful lesson for you.
Skype: sanbarrow
Ulli

Do you need support with a recovery problem ? - send a message via skype "sanbarrow"
0 Kudos
cycloxr
Contributor
Contributor

Sounds correct on 000008-000014 being created after the issue occurred.

Also most likely correct on the basedisk as I was in panic mode trying to fix this before talking to you.

I don't have a Skype account.  Can I private message you a Zoom Meeting or Google Hangouts request? 

0 Kudos
continuum
Immortal
Immortal

Never used Zoom Meeting or Google Hangouts for remote support. Anyway - Microsoft makes big progress in messing up Skype - so I need to look into other options anyway.
Send a PM with the details for Zoom - I will be back in about 90 minutes.
I expect that you have a Putty and WinSCP on your admin host.
See you later
Ulli

Do you need support with a recovery problem ? - send a message via skype "sanbarrow"
0 Kudos
cycloxr
Contributor
Contributor

Just sent you message with Zoom meeting details.  If you join when you're available it will notify me. 

0 Kudos
cycloxr
Contributor
Contributor

Also, I checked for child consistency:

[root@ESXi-6-1-104:/vmfs/volumes/4a8a34af-78da8e1c-0000-000000000000/DMZPortal] vmkfstools -e DMZPortal-000007.vmdk

Disk link /vmfs/volumes/4a8a34af-78da8e1c-0000-000000000000/DMZPortal/DMZPortal-000007.vmdk successfully opened.

Disk link /vmfs/volumes/4a8a34af-78da8e1c-0000-000000000000/DMZPortal/DMZPortal-000006.vmdk successfully opened.

Disk link /vmfs/volumes/4a8a34af-78da8e1c-0000-000000000000/DMZPortal/DMZPortal-000005.vmdk successfully opened.

Disk link /vmfs/volumes/4a8a34af-78da8e1c-0000-000000000000/DMZPortal/DMZPortal-000004.vmdk successfully opened.

Disk link /vmfs/volumes/4a8a34af-78da8e1c-0000-000000000000/DMZPortal/DMZPortal-000003.vmdk successfully opened.

Cannot establish parent(/vmfs/volumes/4a8a34af-78da8e1c-0000-000000000000/DMZPortal/DMZPortal-000003.vmdk)-child(/vmfs/volumes/4a8a34af-78da8e1c-0000-000000000000/DMZPortal/DMZPortal-000004.vmdk) relationship : The parent virtual disk has been modified since the child was created. The content ID of the parent virtual disk does not match the corresponding parent content ID in the child (18)

Parent link description

        Name: /vmfs/volumes/4a8a34af-78da8e1c-0000-000000000000/DMZPortal/DMZPortal-000003.vmdk

        Create Type: vmfsSparse

        Capacity: 536870912000

        Content ID: 628596982

        Parent Content ID: 1169256954

Child link description

        Name: /vmfs/volumes/4a8a34af-78da8e1c-0000-000000000000/DMZPortal/DMZPortal-000004.vmdk

        Create Type: vmfsSparse

        Capacity: 536870912000

        Content ID: 2000236308

        Parent Content ID: 119867593

Disk link /vmfs/volumes/4a8a34af-78da8e1c-0000-000000000000/DMZPortal/DMZPortal-000002.vmdk successfully opened.

Disk link /vmfs/volumes/4a8a34af-78da8e1c-0000-000000000000/DMZPortal/DMZPortal-000001.vmdk successfully opened.

Failed to open disk link /vmfs/volumes/4a8a34af-78da8e1c-0000-000000000000/DMZPortal/DMZPortal.vmdk :The file specified is not a virtual disk (15)Disk chain is not consistent : The parent virtual disk has been modified since the child was created. The content ID of the parent virtual disk does not match the corresponding parent content ID in the child (18)

Pulled all the child and parent IDs from the files:

DMZPortal.vmdk

CID=4a247967

parentCID=ffffffff

DMZPortal-000001.vmdk

CID=2cc1f3a9

parentCID=ce59c404

DMZPortal-000002.vmdk

CID=45b171fa

parentCID=2cc1f3a9

DMZPortal-000003.vmdk

CID=2577a0f6

parentCID=45b171fa

DMZPortal-000004.vmdk

CID=77392f14

parentCID=072508c9

DMZPortal-000005.vmdk

CID=f7c0da15

parentCID=77392f14

DMZPortal-000006.vmdk

CID=44954faa

parentCID=f7c0da15

DMZPortal-000007.vmdk

CID=8e9edc7d

parentCID=44954fa

DMZPortal-000008.vmdk

CID=8da068ed

parentCID=8e9edc7d

DMZPortal-000009.vmdk

CID=477eb15d

parentCID=8da068ed

DMZPortal-000010.vmdk

CID=00000000

parentCID=00000000

DMZPortal-000011.vmdk

CID=00000000

parentCID=00000000

DMZPortal-000012.vmdk

CID=bcb85e00

parentCID=477eb15d

DMZPortal-000013.vmdk

CID=79309a7b

parentCID=bcb85e00

DMZPortal-000014.vmdk

CID=dc67d10b

parentCID=79309a7b

DMZPortal-000015.vmdk

CID=dc67d10b

parentCID=dc67d10b

DMZPortal-000016.vmdk

CID=dc67d10b

parentCID=dc67d10b

0 Kudos
cycloxr
Contributor
Contributor

The issue looks pretty evident after looking at this.   I'm in the process of fixing the IDs on a backup but probably wont' make any permanent changes until I speak to you.

0 Kudos
cycloxr
Contributor
Contributor

Looks like this is what it should be: ** = modified

DMZPortal.vmdk

**CID=44954faa

parentCID=ffffffff

DMZPortal-000001.vmdk

CID=2cc1f3a9

parentCID=ce59c404

DMZPortal-000002.vmdk

CID=45b171fa

parentCID=2cc1f3a9

DMZPortal-000003.vmdk

**CID=2cc1f3a9

parentCID=45b171fa

DMZPortal-000004.vmdk

**CID=45b171fa

parentCID=072508c9

DMZPortal-000005.vmdk

**CID=072508c9

parentCID=77392f14

DMZPortal-000006.vmdk

**CID=77392f14

parentCID=f7c0da15

DMZPortal-000007.vmdk

**CID=f7c0da15

parentCID=44954faa

0 Kudos
continuum
Immortal
Immortal

That is one part of the task. I agree with your values.
Next thing is to find the original flat.vmdk.
Looks like you renamed the flat.vmdk to *-delta.vmdk.
To check which one it is run
hexdump -n 512 -C *-delta.vmdk
We need to identify the one that does not start with COWD and ends with 00 00 55 AA

Do you need support with a recovery problem ? - send a message via skype "sanbarrow"

View solution in original post

0 Kudos
cycloxr
Contributor
Contributor

continuum​, thank you for all of your help with resolving this issue!! I greatly appreciate!

I greatly appreciate you taking the time with me to assist in troubleshooting this instance and get things working again!

For others looking for a resolution on this, we basically had to go through each snapshot file and assign new CID as well as clean up other existing config files.  Many lessons learned on this one.  

0 Kudos