Unable to remove, migrate, etc., VM on ESXi host

[Ashom]

A while back, the server room hosting my vSphere infrastructure suffered cooling loss causing multiple ESXi hosts to shutdown without warning due to excess heat.  Needless to say, the results were chaotic.  While I've recovered/restored most of the systems, one problem seems to persist.  The is a VM that I'm unable to delete, move, etc. due to active file locks.  The option on the .vmx file to "Add to Inventory" under the Datastore is also grayed out.  However, this VM does not show up with a "vim-cmd vmsvc/getallvms" command.  In fact, in the process of investigating the file locks, it only shows up with a "vmkvsitools lsof | grep VM_Name" command and no results return with a "ps | grep VM_Name" command.  The host in question has also been rebooted and upgraded (5.1 Update 2 to Update 3) so a simple restart does not clear the locks.  So, the my question is; how do I get rid of this VM so I can properly re-add it to vCenter?  Any help would be appreciated.

[Alistar]

Hi there, you said:

In fact, in the process of investigating the file locks, it only shows up with a "vmkvsitools lsof | grep VM_Name"

What is the output? Have you tried a plain lsof | grep VM_name ? Also the machine might have been restarted by HA with some other destination host during the emergency situation and then failed again.

[Ashom]

Output of vmkvsitools lsof | grep VM_Name is:

20156     vmx     FILE     4       /vmfs/volumes/DATASTORE/VM_Name/vmware.log

20156     vmx     FILE     83     /vmfs/volumes/DATASTORE/VM_Name/VM_Name-disk1-000002-delta.vmdk

20126     vmx     FILE     84     /vmfs/volumes/DATASTORE/VM_Name/VM_Name-000001-delta.vmdk

etc...

Note - There is a total of 17 files listed.

Given the presents of snapshot-related files (as shown above) and the fact that I run vDP, it is likely that the system was in the middle of a backup at the time of failure.

Output of "lsof | grep VM_Name" is identical to "vmkvsitools lsof | grep VM_Name".  Also, this is on a 2 host datacenter/cluster and the above commands do not return any output when run on the other host.

[Alistar · Accepted solution]

Then simply do a kill -9 20156 and you're all set - this kill the world and will forcibly shut down the virtual machine and your locks will be free

[Ashom]

That did it.  That was the part the KB article (KB 10051) wasn't clear about, i.e. what part of the vmkvsitools command was actually needed.  It just said run the command then run a "ps | grep VM_Name" command without explain the reason for the vmkvsitools command.  Without anything showing up from the "ps" command, I was left wondering.  Thanks for the help!