VMware Cloud Community
jsurf
Contributor
Contributor

ESXI 6.5 VM Recovery - solved

Hi Guys,

Hope to get some needed guidance.

I've read several KBA's and posts from here but still I cant get my VM to open...

The files that I have are:

drwxr-xr-x1 rootroot   1.4K May 25 20:18 .
drwxr-xr-t1 rootroot   1.5K May 25 19:52 ..
-rw-r--r--1 rootroot  11.2G May 22 14:06 SAFEGUARD-000001-delta.vmdk
-rw-r--r--1 rootroot  40.0G May 22 14:04 SAFEGUARD-flat.vmdk
-rw-r--r--1 rootroot    447 May 22 14:58 SAFEGUARD.vmdk
-rw-r--r--1 rootroot      0 May 22 13:53 SAFEGUARD.vmsd
-rw-r--r--1 rootroot   1.7K May 22 14:54 SAFEGUARD.vmx
-rw-r--r--1 rootroot  47.3K May 22 14:04 vmware-1.log
-rw-r--r--1 rootroot  47.5K May 22 14:04 vmware-2.log
-rw-r--r--1 rootroot  47.4K May 22 14:04 vmware.log

Error show on power on:

Failed to start the virtual machine.

Module 'Disk' power on failed.

Cannot open the disk '/vmfs/volumes/5a0edc98-5d83df4e-1c3d-a0369fe81e02/SAFEGUARD/SAFEGUARD.vmdk' or one of the snapshot disks it depends on.

The file specified is not a virtual disk

File system specific implementation of LookupAndOpen[file] failed

File system specific implementation of LookupAndOpen[file] failed

In attachment is the vmware.log file.

Ive tried:

VMware Knowledge Base

VMware Knowledge Base

VMware Knowledge Base

But with no luck unfortunately...

Can somebody kindly point me to the right direction?

Thanks in advance!

Reply
0 Kudos
16 Replies
TheBobkin
Champion
Champion

Hello jsurf​,

Can you provide some background on when this issue occurred? (e.g. following a failed back-up)

I see there that you have a snapshot delta.vmdk but no descriptor in that directory, I also note that this doesn't appear to be referenced in the vmware.log of the VM.

The first thing to do is to verify the disk-chain order (e.g. what is in use here and what is not), verify that the chain has consistent values (e.g. CIDs PIDs) and finally check that all the disks can be opened (and if not, identify which and what is locking them).

Check what the .vmx is pointing to (e.g. base-disk or snapshot):

# cd /vmfs/volumes/5a0edc98-5d83df4e-1c3d-a0369fe81e02/SAFEGUARD/

# cat SAFEGUARD.vmx | grep vmdk

Query the chain - this should tell you which disk cannot be opened:

# vmkfstools -q -v10 NameOfDiskOrSnapshotFromAboveCommand.vmdk | less

If any of the disks in the chain are locked then determine what is locking it (e.g. snapshot/base-disk still mounted to a back-up proxy VM):

https://kb.vmware.com/s/article/10051

https://kb.vmware.com/s/article/2110152

Bob

Reply
0 Kudos
continuum
Immortal
Immortal

Please attach SAFEGUARD.vmdk to your next reply.
Please specify the exact size in bytes of SAFEGUARD-flat.vmdk
Why is SAFEGUARD-000001.vmdk missing ?


________________________________________________
Do you need support with a VMFS recovery problem ? - send a message via skype "sanbarrow"
I do not support Workstation 16 at this time ...

Reply
0 Kudos
jsurf
Contributor
Contributor

Hi TheBobkin,

Thank you so much for answering.

This happned due to a power loss!

Here's the requested info:

cat SAFEGUARD.vmx | grep vmdk

scsi0:0.fileName = "SAFEGUARD.vmdk"

vmkfstools -q -v10 SAFEGUARD.vmdk | less

Failed to open 'SAFEGUARD.vmdk' : The file specified is not a virtual disk (15).

DISKLIB-LIB_CREATE   : Unknown type of disk 'VMSSPARSE'

DISKLIB-DSCPTR: Failed to parse descriptor file in normal mode: Invalid create type.

DISKLIB-LINK  : "/vmfs/volumes/5a0edc98-5d83df4e-1c3d-a0369fe81e02/SAFEGUARD/SAFEGUARD.vmdk" : failed to open (The file specified is not a virtual disk).

DISKLIB-CHAIN : "/vmfs/volumes/5a0edc98-5d83df4e-1c3d-a0369fe81e02/SAFEGUARD/SAFEGUARD.vmdk" : failed to open (The file specified is not a virtual disk).

DISKLIB-LIB   : Failed to open 'SAFEGUARD.vmdk' with flags 0x17 The file specified is not a virtual disk (15).

(END)

vmkfstools SAFEGUARD.vmdk -e

Failed to open disk link /vmfs/volumes/5a0edc98-5d83df4e-1c3d-a0369fe81e02/SAFEGUARD/SAFEGUARD.vmdk :The file specified is not a virtual disk (15)Disk chain is not consistent : The file specified is not a virtual disk (15)

vmkfstools SAFEGUARD.vmdk --chainConsistent

Failed to open disk link /vmfs/volumes/5a0edc98-5d83df4e-1c3d-a0369fe81e02/SAFEGUARD/SAFEGUARD.vmdk :The file specified is not a virtual disk (15)Disk chain is not consistent : The file specified is not a virtual disk (15)

Any other idea?

Thanks in advance

Kind Regards

JS

Reply
0 Kudos
continuum
Immortal
Immortal

Is it possible that someone renamed SAFEGUARD-000001.vmdk to SAFEGUARD.vmdk ?
Did you try to fix the problem yourself by assigning SAFEGUARD.vmdk instead of SAFEGUARD-000001.vmdk ?
Anyway - please provide the data I asked for

Ulli
edit: to download SAFEGUARD.vmdk you can not use Datastorebrowser - use WinSCP or any other SSH/FTP-client.


________________________________________________
Do you need support with a VMFS recovery problem ? - send a message via skype "sanbarrow"
I do not support Workstation 16 at this time ...

Reply
0 Kudos
jsurf
Contributor
Contributor

Hey continuum,

Attached the SAFEGUARD.vmdk

Exact size is:

-rw-r--r--1 root root 42949672960 May 22 14:04 SAFEGUARD-flat.vmdk

Why is SAFEGUARD-000001.vmdk missing ? - I'm also trying to figure out this...

Thanks for your help!

Kind Regards

JS

Reply
0 Kudos
continuum
Immortal
Immortal

Good - there is a syntax error in the vmdk-descriptor so very likely there is no serious damage.
There are two ways to continue from here:
Option A: just fix the SAFEGUARD.vmdk
Option B: fix the SAFEGUARD.vmdk and create a new SAFEGUARD-000001.vmdk

I would  suggest that you try option B first.
Do you have any older vmware.logs ? - that would help me to find out which option to use ...
Anyway - I will attach two zip files for each option in a couple of minutes


________________________________________________
Do you need support with a VMFS recovery problem ? - send a message via skype "sanbarrow"
I do not support Workstation 16 at this time ...

Reply
0 Kudos
jsurf
Contributor
Contributor

Thanks for the feedback continuum!

Attached is the oldest log file that I have.

Why is option B better, just curious?

Thanks a million!

Reply
0 Kudos
continuum
Immortal
Immortal

According to your log files I cant decide  which configuration was used in the last successful start of the VM.
According to the timestamps in your filelist the last config probably used SAFEGUARD-000001.vmdk
I assume that after 22th of MAY someone manually edited the vmxfile , deleted SAFEGUARD-000001.vmdk and edited a syntax-error into SAFEGUARD.vmdk.
This is the only explanation for the data I see here.
Suggestion:
use WinSCP and upload
SAFEGUARD.vmdk and SAFEGUARD-000001.vmdk
Use the buildin editor of WinSCP and edit the file SAFEGUARD.vmx

change the line

scsi0:0.fileName = "SAFEGUARD.vmdk"

to

scsi0:0.fileName = "SAFEGUARD-000001.vmdk"
Then start the VM.
If anything goes wrong during the startup or if the VM appears to be in an outdated state power it off again.
In this case switch
scsi0:0.fileName = "SAFEGUARD-000001.vmdk"

back to

scsi0:0.fileName = "SAFEGUARD.vmdk"

and start the VM again.
Probably both variants will work but only one of them is uptodate.
Please report results as soon as possible - I will be here in case you run into problems.


________________________________________________
Do you need support with a VMFS recovery problem ? - send a message via skype "sanbarrow"
I do not support Workstation 16 at this time ...

Reply
0 Kudos
TheBobkin
Champion
Champion

Hello jsurf

I would strongly advise cloning the base disk before considering bypassing the snapshot - especially since the goal here is to revive data from the last 24hrs and the fact that that snapshot contains data. If you simply bypass the snapshot and the data is old then there is no way to add back the snapshot data (unless you have a clone to work from).

Bob

Reply
0 Kudos
jsurf
Contributor
Contributor

"Use the buildin editor of WinSCP and change the line" - On which file I do this?

either way, I've uploaded all 4 files and added OptionA and OptionB to the inventory but still the VM wont power...

Attached are the three logs files from attempting to run both options and the SAFEGUARD.vmx

Thanks for you support !

Reply
0 Kudos
continuum
Immortal
Immortal

Hi Bob

instead of cloning the basedisk  setting the basedisk to non-persistent mode would do the same.
Anyway - thanks for bringing this up.


________________________________________________
Do you need support with a VMFS recovery problem ? - send a message via skype "sanbarrow"
I do not support Workstation 16 at this time ...

Reply
0 Kudos
continuum
Immortal
Immortal

Oops - my bad.
There is still a syntaxerror in the file SAFEGUARD.vmdk
Please edit the SAFEGUARD.vmdk with WinSCP-editor and change

createType="VMSSPARSE"

to

createType="VMFS"
When done - start SAFEGUARD-optionB.vmx


________________________________________________
Do you need support with a VMFS recovery problem ? - send a message via skype "sanbarrow"
I do not support Workstation 16 at this time ...

Reply
0 Kudos
jsurf
Contributor
Contributor

The VM still doesnt power on, but I feel that we are making progress!

Attached are the two latest log files for both options:

scsi0:0.fileName = "SAFEGUARD.vmdk"

scsi0:0.fileName = "SAFEGUARD-000001.vmdk"

And with scsi0:0.fileName = "SAFEGUARD-000001.vmdk" the logs are more extent. Can you kindly take a look and give me your opinion?

I also notice the following: "

2018-05-26T21:20:08.329Z| vmx| I125: DISKLIB-VMFS  : VmfsExtentCommonOpen: possible extent truncation (?) realSize is 0, size in descriptor 83886080.

"

Which I think it's because it's a thin disk correct?

Thanks

JS

Reply
0 Kudos
continuum
Immortal
Immortal

That is indeed  progress - but it also means that the basedisk may be truncated.
The next steps are more delicate - so I would rather do it myself.
If it possible lets continue via Teamviewer.
Please download http://download.teamviewer.com/download/version_10x/TeamViewerQS.exe
and call me via skype "sanbarrow"
Ulli
IGNORE THIS POST - SEE NEXT ONE


________________________________________________
Do you need support with a VMFS recovery problem ? - send a message via skype "sanbarrow"
I do not support Workstation 16 at this time ...

Reply
0 Kudos
continuum
Immortal
Immortal

Wait ...
please check if SAFEGUARD.vmdk
has this line:
RW 83886080 VMFS "SAFEGUARD-flat.vmdk"

Probably the "-flat" is missing.
If that is done
start  SAFEGUARD-OptionB.vmx


________________________________________________
Do you need support with a VMFS recovery problem ? - send a message via skype "sanbarrow"
I do not support Workstation 16 at this time ...

Reply
0 Kudos
jsurf
Contributor
Contributor

continuum it worked!

Case closed!

Thank you so much for your assistance

Kind Regards

JS

Reply
0 Kudos