Can't power on VMs "Unable to access file since it is locked"

[StageCoach20111]

VMs were running fine - then storage system went down.  After storage system came back up I can't power on a number of VMs.  Getting this error when I try to power them on:

Cannot power On vmname on esxhostname.xxxxx.com. Unable to access file <unspecified filename> since it is locked
error

[MauroBonder]

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=10051

Please, don't forget the awarding points for "helpful" and/or "correct" answers.  

Mauro Bonder - Moderator

enjoy and good luck

[StageCoach20111]

I'm working through the steps with much enjoyment and got to the step where I run "tail /var/log/vmkernel" and try to find the MAC address of the host that has the locked file.

(In this case I've identified the locked file as virtualmachinename-9fb5b0a5.vswp file).

I pasted the results below.  Something doesn't look right though.  It says look for the line that says "owner" but that line has e61f13c5ac53

is that really the MAC address that is locking the file? Thanks again

[MauroBonder · Accepted solution]

1. To check for Service Console-based locks on non-ESXi servers, run the command:
   
   lsof | grep <name of locked file>
   
   COMMAND PID USER FD TYPE DEVICE SIZE NODE NAME
   71fd60b6- 3631 root 4r REG 0,9 10737418240 23533 <name of locked file>
   
   Note:  If there is no Service Console process locking the file, you should  receive no printed output. If you receive any results, however, file a  Support Request to identify the process and to determine the root-cause.  If it is a third-party process, however, contact the appropriate vendor  in order to determine the root-cause prior to killing the process ID, as it may occur again in the future.
   
   Stop the process ID and its lock using the kill command. From the above example, the process ID is 3631:
   
   kill 3631
   
   After stopping the process, you can attempt to power on the virtual machine or access the file/resource.
   
   
2. To check if the virtual machine still has a World ID assigned to it, run these commands on all ESX/ESXi hosts:
   
   cd /tmp
   vm-support -x
   
   Available worlds to debug:
   wid=<world id> <name of VM with locked file>
   
   On  the ESX/ESXi host where the virtual machine is still running, kill the  virtual machine, which releases the lock on the file. To kill the  virtual machine, run the command:
   
   vm-support -X <world id>
   
   Where the <world id> is the World ID of the virtual machine with the locked file.
   
   Note: This command takes 5-10 minutes to complete. Answer No to "Can I include a screenshot of the VM", and answer Yes to all subsequent questions.
   
   After stopping the process, you can power on the virtual machine or access the file/resource

Removing the .lck file (NFS Only)

The virtual machine's files may be locked via NFS storage. You can identify this by files denoted with .lck.####  (where #### refers to the World ID that has the file lock) at the end  of the filename. This is an NFS file lock and is only listed when using  the ls -la command as it is hidden file.

Caution: These can be removed safely only if the virtual machine is not running.

Note: VMFS volumes do not have .lck files. The locking mechanism for VMFS volumes is handled within VMFS metadata on the volume.

*****you need to kill the process that is locked, if it does not, you probably will have to restart the VMware host.
I tell you from experience very likely to end up rebooting it.

[StageCoach20111]

If the file that is locked is a .vswp file, what is the impact of just deleting it?

[DSTAVERT]

That would be fine but you can't delete a locked file.

[MauroBonder]

Stage,

Don´t worry, no problems. vswap is re created when virtual machine powered on.

Message was edited by: MauroBonder

[DSTAVERT]

I would try an ESXi host restart if you can't kill the hung process. Follow the KB article. If you can't kill the process or a restart doesn't help I would call VMware Support. There may be some file system corruption or ???

[StageCoach20111]

After restarting ESX host still same problem.  Then deleted locked file - still doesn't work but now there's a new error:

Power On virtual machine
vmbldlnx15
Could not create '/vmfs/volumes/<UUID>/<VMName>/<VMName>.nvram': Failed to
lock the file

[MauroBonder]

Did you there just one host VMware ? If you, you need check if have any processor of this virtual machine running in another host VMware.

Please, don't forget the awarding points for "helpful" and/or "correct" answers. 

Mauro Bonder - Moderator

[DSTAVERT]

I would edit the VMX file and remove the nvram reference.

[StageCoach20111]

Edited vmx file and removed reference to .nvram file.  The VM now boots!!! Mauro and DStavert thanks again.

[DSTAVERT]

Glad you are back running again.

Good luck.