Highlighted
Immortal
Immortal

Locked files - VMFS 6

You probably know the error message: operation impossible since file is locked.
If a VMDK is locked you can not start it inside a VM and you can not copy it.
The knowledgebase basically tells you to stop all processes that access the file and reboot.
This is supposed to fix the problem.
This documentation is not good enough.
In VMFS 6 following this advice may not be enough to release the lock.
This means that you basically lost the file as you can not read it anymore.
Trying to access it via Linux is also impossible at the moment.
Recent recovery third party tools like UFSexplorer also failed to read the vmdks in such a case.
Here is a recent case for your reference .... 
Problem with locked virtual machine after esxi host crash
I have seen a couple of this cases recently so I started to investigate the issue,
Now I may have found a way to recover a VMDK that would other wise be a case for Ontrack or VMware Support.
My procedure requires a direct hexedit of the VMFS heartbeat- section so please do not ask for details at the moment.
I will post details once I know this approach is dafe.
Anyway I believe this is a bug in the way ESXi 6.5 handles the heartbeat section of a VMFS-volume.
This issue should never affect single host environements.
Feel free to contact me via skype if you run into this yourself.
Ulli

Do you need support with a recovery problem ? - call me via skype "sanbarrow"
8 Replies
Highlighted
User Moderator
User Moderator

Good to know whom to contact in such a case!

André

0 Kudos
Highlighted
Contributor
Contributor

continuum​ I have seen this problem, pls help me.

Other way, what's your skype, I will contact you to ask.

0 Kudos
Highlighted
Contributor
Contributor

Hi Ulli

I have the same issue. I moved a VM from a 6.0.0 cluster to a standalone ESXi 6.7U2.

Any help appreciated.

BR

Marc

0 Kudos
Highlighted
Immortal
Immortal

Hi Marc

you were able to move a VM and then have locked files in the new environment ?

thats strange and unexpected ...

please create a header dump and send it myway so that I can look into it.

See Create a VMFS-Header-dump using an ESXi-Host in production | VM-Sickbay

If possible contact me via skype sanbarrow

Ulli

Do you need support with a recovery problem ? - call me via skype "sanbarrow"
0 Kudos
Highlighted
Enthusiast
Enthusiast

Hello,

Have you created a post with the steps to try and fix this?

I've just had this happening in my homelab. After an issue with the storage array, which led to a disk replacement, I now have 5-6 VMs which are inaccesible/orphaned and their vmx is locked by one of the ESXi hosts. Of course I've pretty much tried everything, host reboots, storage array reboot etc. I will not release the lock. Since it's my homelab I'm pretty much open to try anything.

Thank you!

0 Kudos
Highlighted
Immortal
Immortal

This is what I do nowadays:

plan A - safest option :

extract the VMs with Linux

plan B -  smartest option but not for the faint at heart:

patch the .vh.sf file

send from iphone

Do you need support with a recovery problem ? - call me via skype "sanbarrow"
0 Kudos
Highlighted
Contributor
Contributor

Hi Ulli,

Can you explain how to extract the VM with Linux?

Best Regards,
Thiago

0 Kudos
Highlighted
Immortal
Immortal

Easiest option:

connect to the datastore with sshfs in readonly mode  - then use ddrescue against the flat.vmdks

If that does not work - try if you can get the location of the fragments with vmkfstools -p 0 flat.vmdk

If that does not work - try to get the location of the fragments by analysing the VMFS-metadata

If that does not work - find the first fragment with scalpel and hope that the flat.vmdks are allocated in one piece

Do you need support with a recovery problem ? - call me via skype "sanbarrow"
0 Kudos