Hello,
Is there a way to populate the Snapshot Manager with snapshots that are listed in the datastore but not under Manage Snapshots. Is there a way to either populate Manage Snapshots or using something like vmkfstools revert to the snapshot manually?
Long story short, I had a hardware failure and had to re-add the datastore to the Linux VM. I believe before the failure that I was using Portal-000003.vmdk but when I try to start the system with that datastore I get the following error: Failed to power on virtual machine Portal. File system specific implementation of LookupAndOpen[file...
When I try to using Portal.vmdk it boots up fine, but seems to be running code from 2015. When I look at the Datastore I have several vmdk files:
-rw------- 1 root root 21308092416 Jun 23 2016 Portal-000001-delta.vmdk
-rw------- 1 root root 323 Nov 24 2015 Portal-000001.vmdk
-rw------- 1 root root 117391208448 Nov 24 2015 Portal-000002-delta.vmdk
-rw------- 1 root root 330 Nov 24 2015 Portal-000002.vmdk
-rw------- 1 root root 8192512 Apr 19 05:17 Portal-000003-ctk.vmdk
-rw------- 1 root root 521604673536 Apr 19 03:28 Portal-000003-delta.vmdk
-rw------- 1 root root 457 Apr 19 04:35 Portal-000003.vmdk
-rw------- 1 root root 8192512 Apr 18 17:37 Portal-000004-ctk.vmdk
-rw------- 1 root root 32146173952 Apr 17 03:03 Portal-000004-delta.vmdk
-rw------- 1 root root 398 Apr 17 03:02 Portal-000004.vmdk
-rw------- 1 root root 8192512 Apr 19 05:52 Portal-000005-ctk.vmdk
-rw------- 1 root root 2198843392 Apr 17 03:07 Portal-000005-delta.vmdk
-rw------- 1 root root 398 Apr 17 03:07 Portal-000005.vmdk
-rw------- 1 root root 8192512 Apr 18 17:37 Portal-000006-ctk.vmdk
-rw------- 1 root root 4413435904 Apr 17 03:15 Portal-000006-delta.vmdk
-rw------- 1 root root 398 Apr 17 03:10 Portal-000006.vmdk
-rw------- 1 root root 8192512 Apr 18 17:37 Portal-000007-ctk.vmdk
-rw------- 1 root root 31961624576 Apr 17 07:43 Portal-000007-delta.vmdk
-rw------- 1 root root 398 Apr 17 07:10 Portal-000007.vmdk
-rw------- 1 root root 536870912000 Apr 19 05:57 Portal-flat.vmdk
-rw------- 1 root root 8684 Apr 19 05:57 Portal.nvram
-rw------- 1 root root 560 Apr 19 05:56 Portal.vmdk
-rw-r--r-- 1 root root 78 Apr 19 05:40 Portal.vmsd
-rwxr-xr-x 1 root root 3394 Apr 19 05:57 Portal.vmx
-rw-r--r-- 1 root root 367 Apr 18 16:40 Portal.vmxf
Any suggestions on what else I can do to revert back to the latest good snapshot?
Thanks.
That is one part of the task. I agree with your values.
Next thing is to find the original flat.vmdk.
Looks like you renamed the flat.vmdk to *-delta.vmdk.
To check which one it is run
hexdump -n 512 -C *-delta.vmdk
We need to identify the one that does not start with COWD and ends with 00 00 55 AA
You can consolidate/remove them like this with vim-cmd VMware Knowledge Base Judging by the size that may take a while...
I'm nearly out of disk space though, how much disk space would consolidation take?
difficult to tell as there is a bit of overhead - could you free up some space as a precaution, svmotion a vm or 2 onto another datastore?
> When I try to using Portal.vmdk it boots up fine, but seems to be running code from 2015.
Do you have older vmware.log files of this VM ?
Stop using the VM immediatly - you obviously assigned not the correct vmdk.
You need to lookup the order of vmdks in older log-files.
continuum, thanks for the response. Unfortunately the older vmware.log files don't exist any longer.
The only snapshot file I care about at this point is the Portal-000003-delta.vmdk, is there anyway to recover/extract that data so I can just rebuilt the system and merge the data over?
Currently if I try to mount the Portal-000003.vmdk file I get a "File system specific implementation of LookupAndOpen[file] failed"
Thank you.
> Currently if I try to mount the Portal-000003.vmdk
STOP DOING THAT - this will corrupt the original snapshot chain.
Please connect to the datastore with WinSCP and download all the vmdk-descriptorfiles (the small vmdks without flat / delta extension.
But first of all stop all mount-attempts
ok, I have stopped trying to mount anything at the moment. I have downloaded and copied all vmdk descriptor files to a backup. Next steps?
Thank you again, really appreciate the replies.
Backup is nice buty first of all I want to see them to check the correct order.
Attach them to your next reply.
Can I assume that all snapshots from 000008 - 000014 were created after the problem occured ?
They are not listed in your original filelist !
I just noticed that you or someone else made a mistake while editing descriptorfiles !
According to your descriptorfiles the basedisk points to "DMZPortal-000005-delta.vmdk"
That makes no sense and shows that a person in panic-mode was trying to fix this problem.
I would suggest that we setup a remote session via Anydesk or Teamviewer - I need to find the original flat.vmdk with hexdump commands ....
We already spend way too much time on this - pleaase call me via Skype so that we can speed this up.
Watching how to fix issues like this will also be a useful lesson for you.
Skype: sanbarrow
Ulli
Sounds correct on 000008-000014 being created after the issue occurred.
Also most likely correct on the basedisk as I was in panic mode trying to fix this before talking to you.
I don't have a Skype account. Can I private message you a Zoom Meeting or Google Hangouts request?
Never used Zoom Meeting or Google Hangouts for remote support. Anyway - Microsoft makes big progress in messing up Skype - so I need to look into other options anyway.
Send a PM with the details for Zoom - I will be back in about 90 minutes.
I expect that you have a Putty and WinSCP on your admin host.
See you later
Ulli
Just sent you message with Zoom meeting details. If you join when you're available it will notify me.
Also, I checked for child consistency:
[root@ESXi-6-1-104:/vmfs/volumes/4a8a34af-78da8e1c-0000-000000000000/DMZPortal] vmkfstools -e DMZPortal-000007.vmdk
Disk link /vmfs/volumes/4a8a34af-78da8e1c-0000-000000000000/DMZPortal/DMZPortal-000007.vmdk successfully opened.
Disk link /vmfs/volumes/4a8a34af-78da8e1c-0000-000000000000/DMZPortal/DMZPortal-000006.vmdk successfully opened.
Disk link /vmfs/volumes/4a8a34af-78da8e1c-0000-000000000000/DMZPortal/DMZPortal-000005.vmdk successfully opened.
Disk link /vmfs/volumes/4a8a34af-78da8e1c-0000-000000000000/DMZPortal/DMZPortal-000004.vmdk successfully opened.
Disk link /vmfs/volumes/4a8a34af-78da8e1c-0000-000000000000/DMZPortal/DMZPortal-000003.vmdk successfully opened.
Cannot establish parent(/vmfs/volumes/4a8a34af-78da8e1c-0000-000000000000/DMZPortal/DMZPortal-000003.vmdk)-child(/vmfs/volumes/4a8a34af-78da8e1c-0000-000000000000/DMZPortal/DMZPortal-000004.vmdk) relationship : The parent virtual disk has been modified since the child was created. The content ID of the parent virtual disk does not match the corresponding parent content ID in the child (18)
Parent link description
Name: /vmfs/volumes/4a8a34af-78da8e1c-0000-000000000000/DMZPortal/DMZPortal-000003.vmdk
Create Type: vmfsSparse
Capacity: 536870912000
Content ID: 628596982
Parent Content ID: 1169256954
Child link description
Name: /vmfs/volumes/4a8a34af-78da8e1c-0000-000000000000/DMZPortal/DMZPortal-000004.vmdk
Create Type: vmfsSparse
Capacity: 536870912000
Content ID: 2000236308
Parent Content ID: 119867593
Disk link /vmfs/volumes/4a8a34af-78da8e1c-0000-000000000000/DMZPortal/DMZPortal-000002.vmdk successfully opened.
Disk link /vmfs/volumes/4a8a34af-78da8e1c-0000-000000000000/DMZPortal/DMZPortal-000001.vmdk successfully opened.
Failed to open disk link /vmfs/volumes/4a8a34af-78da8e1c-0000-000000000000/DMZPortal/DMZPortal.vmdk :The file specified is not a virtual disk (15)Disk chain is not consistent : The parent virtual disk has been modified since the child was created. The content ID of the parent virtual disk does not match the corresponding parent content ID in the child (18)
Pulled all the child and parent IDs from the files:
DMZPortal.vmdk
CID=4a247967
parentCID=ffffffff
DMZPortal-000001.vmdk
CID=2cc1f3a9
parentCID=ce59c404
DMZPortal-000002.vmdk
CID=45b171fa
parentCID=2cc1f3a9
DMZPortal-000003.vmdk
CID=2577a0f6
parentCID=45b171fa
DMZPortal-000004.vmdk
CID=77392f14
parentCID=072508c9
DMZPortal-000005.vmdk
CID=f7c0da15
parentCID=77392f14
DMZPortal-000006.vmdk
CID=44954faa
parentCID=f7c0da15
DMZPortal-000007.vmdk
CID=8e9edc7d
parentCID=44954fa
DMZPortal-000008.vmdk
CID=8da068ed
parentCID=8e9edc7d
DMZPortal-000009.vmdk
CID=477eb15d
parentCID=8da068ed
DMZPortal-000010.vmdk
CID=00000000
parentCID=00000000
DMZPortal-000011.vmdk
CID=00000000
parentCID=00000000
DMZPortal-000012.vmdk
CID=bcb85e00
parentCID=477eb15d
DMZPortal-000013.vmdk
CID=79309a7b
parentCID=bcb85e00
DMZPortal-000014.vmdk
CID=dc67d10b
parentCID=79309a7b
DMZPortal-000015.vmdk
CID=dc67d10b
parentCID=dc67d10b
DMZPortal-000016.vmdk
CID=dc67d10b
parentCID=dc67d10b
The issue looks pretty evident after looking at this. I'm in the process of fixing the IDs on a backup but probably wont' make any permanent changes until I speak to you.
Looks like this is what it should be: ** = modified
DMZPortal.vmdk
**CID=44954faa
parentCID=ffffffff
DMZPortal-000001.vmdk
CID=2cc1f3a9
parentCID=ce59c404
DMZPortal-000002.vmdk
CID=45b171fa
parentCID=2cc1f3a9
DMZPortal-000003.vmdk
**CID=2cc1f3a9
parentCID=45b171fa
DMZPortal-000004.vmdk
**CID=45b171fa
parentCID=072508c9
DMZPortal-000005.vmdk
**CID=072508c9
parentCID=77392f14
DMZPortal-000006.vmdk
**CID=77392f14
parentCID=f7c0da15
DMZPortal-000007.vmdk
**CID=f7c0da15
parentCID=44954faa
That is one part of the task. I agree with your values.
Next thing is to find the original flat.vmdk.
Looks like you renamed the flat.vmdk to *-delta.vmdk.
To check which one it is run
hexdump -n 512 -C *-delta.vmdk
We need to identify the one that does not start with COWD and ends with 00 00 55 AA
continuum, thank you for all of your help with resolving this issue!! I greatly appreciate!
I greatly appreciate you taking the time with me to assist in troubleshooting this instance and get things working again!
For others looking for a resolution on this, we basically had to go through each snapshot file and assign new CID as well as clean up other existing config files. Many lessons learned on this one.