can't register/add to inventory a vm because of locked file

[swspjcd]

I have several ESXi 5.1 hosts in a cluster. There is one vm that I can't add to inventory. I have followed several kb articles and even removed the ESXi host from the cluster completely after verifying that it was the ESXi host causing the lock by finding the MAC address. I am unable to view the vmware.log file for this virtual machine. I get an "invalid argument" error when trying to cat the vmware.log or the .vmx file. The lock file is "vmname.vmx.lck". I've rebooted and restarted the management agents several times. I'm just not sure where to proceed from here as I've been reading about how to resolve this for about 3 hours and have yet to find anything that works. The contents of the directory contain the following files if this helps at all:

-rw-r--r--	1 root	root	162078 Sep 21 16:06 vmware.log
-rw-r--r--	1 root	root	73 Sep  9 18:33 vmname-63cf5ada.hlog
-rw-------	1 root	root	21474836480 Sep 21 16:51 vmname-flat.vmdk
-rw-------	1 root	root	8684 Sep  9 18:34 vmname.nvram
-rw-------	1 root	root	517 Sep  9 18:33 vmname.vmdk
-rw-r--r--	1 root	root	0 Sep  7 17:50 vmname.vmsd
-rwxr-xr-x	1 root	root	3342 Sep 13 18:18 vmname.vmx
-rw-------	1 root	root	0 Sep  9 18:33 vmname.vmx.lck
-rw-r--r--	1 root	root	262 Sep  7 17:50 vmname.vmxf
-rwxr-xr-x	1 root	root	3341 Sep 13 18:18 vmname.vmx~
-rw-------	1 root	root	37580963840 Sep 21 16:51 vmname_1-flat.vmdk
-rw-------	1 root	root	519 Sep  9 18:33 vmname_1.vmdk

Suggestions?

[rclaveau]

I success to repair something similar with voma :

[root@localhost:~] voma -m vmfs -f fix -d /vmfs/devices/disks/mpx.vmhba39\:C0\:T0\:L0\:2

Checking if device is actively used by other hosts

Running VMFS Checker version 1.2 in fix mode

Initializing LVM metadata, Basic Checks will be done

Phase 1: Checking VMFS header and resource files

   Detected VMFS file system (labeled:'USB_ESXi_MASTER') with UUID:55240f29-a13f992b-3aca-f079595b2257, Version 5:61

   Found stale lock [type 10c00002 offset 6360064 v 124, hb offset 3207168

         gen 27, mode 1, owner 55241293-ff2e34de-acbf-f079595b2257 mtime 158522

         num 0 gblnum 0 gblgen 0 gblbrk 0][Cleared]

Phase 2: Checking VMFS heartbeat region

ON-DISK ERROR: Invalid HB address <65536>[Fixed]

Phase 3: Checking all file descriptors.

   Found stale lock [type 10c00001 offset 30605312 v 12, hb offset 3207168

         gen 27, mode 1, owner 55241293-ff2e34de-acbf-f079595b2257 mtime 119

         num 0 gblnum 0 gblgen 0 gblbrk 0][Cleared]

   Found stale lock [type 10c00001 offset 30607360 v 13, hb offset 3207168

         gen 27, mode 1, owner 55241293-ff2e34de-acbf-f079595b2257 mtime 128

         num 0 gblnum 0 gblgen 0 gblbrk 0][Cleared]

   Found stale lock [type 10c00001 offset 30609408 v 14, hb offset 3207168

         gen 27, mode 1, owner 55241293-ff2e34de-acbf-f079595b2257 mtime 137

         num 0 gblnum 0 gblgen 0 gblbrk 0][Cleared]

   Found stale lock [type 10c00001 offset 30611456 v 15, hb offset 3207168

         gen 27, mode 1, owner 55241293-ff2e34de-acbf-f079595b2257 mtime 146

         num 0 gblnum 0 gblgen 0 gblbrk 0][Cleared]

   Found stale lock [type 10c00001 offset 30613504 v 16, hb offset 3207168

         gen 27, mode 1, owner 55241293-ff2e34de-acbf-f079595b2257 mtime 155

         num 0 gblnum 0 gblgen 0 gblbrk 0][Cleared]

   Found stale lock [type 10c00001 offset 30615552 v 17, hb offset 3207168

         gen 27, mode 1, owner 55241293-ff2e34de-acbf-f079595b2257 mtime 164

         num 0 gblnum 0 gblgen 0 gblbrk 0][Cleared]

   Found stale lock [type 10c00001 offset 30617600 v 18, hb offset 3207168

         gen 27, mode 1, owner 55241293-ff2e34de-acbf-f079595b2257 mtime 173

         num 0 gblnum 0 gblgen 0 gblbrk 0][Cleared]

   Found stale lock [type 10c00001 offset 30619648 v 19, hb offset 3207168

         gen 27, mode 1, owner 55241293-ff2e34de-acbf-f079595b2257 mtime 182

         num 0 gblnum 0 gblgen 0 gblbrk 0][Cleared]

   Found stale lock [type 10c00001 offset 30621696 v 20, hb offset 3207168

         gen 27, mode 1, owner 55241293-ff2e34de-acbf-f079595b2257 mtime 191

         num 0 gblnum 0 gblgen 0 gblbrk 0][Cleared]

   Found stale lock [type 10c00001 offset 30623744 v 25, hb offset 3207168

         gen 27, mode 1, owner 55241293-ff2e34de-acbf-f079595b2257 mtime 105768

         num 0 gblnum 0 gblgen 0 gblbrk 0][Cleared]

   Found stale lock [type 10c00001 offset 30625792 v 22, hb offset 3207168

         gen 27, mode 1, owner 55241293-ff2e34de-acbf-f079595b2257 mtime 209

         num 0 gblnum 0 gblgen 0 gblbrk 0][Cleared]

   Found stale lock [type 10c00001 offset 30627840 v 53, hb offset 3207168

         gen 27, mode 1, owner 55241293-ff2e34de-acbf-f079595b2257 mtime 151471

         num 0 gblnum 0 gblgen 0 gblbrk 0][Cleared]

   Found stale lock [type 10c00001 offset 30629888 v 24, hb offset 3207168

         gen 27, mode 1, owner 55241293-ff2e34de-acbf-f079595b2257 mtime 227

         num 0 gblnum 0 gblgen 0 gblbrk 0][Cleared]

   Found stale lock [type 10c00001 offset 30631936 v 25, hb offset 3207168

         gen 27, mode 1, owner 55241293-ff2e34de-acbf-f079595b2257 mtime 236

         num 0 gblnum 0 gblgen 0 gblbrk 0][Cleared]

   Found stale lock [type 10c00001 offset 30633984 v 26, hb offset 3207168

         gen 27, mode 1, owner 55241293-ff2e34de-acbf-f079595b2257 mtime 245

         num 0 gblnum 0 gblgen 0 gblbrk 0][Cleared]

   Found stale lock [type 10c00001 offset 30636032 v 27, hb offset 3207168

         gen 27, mode 1, owner 55241293-ff2e34de-acbf-f079595b2257 mtime 254

         num 0 gblnum 0 gblgen 0 gblbrk 0][Cleared]

   Found stale lock [type 10c00001 offset 30638080 v 28, hb offset 3207168

         gen 27, mode 1, owner 55241293-ff2e34de-acbf-f079595b2257 mtime 263

         num 0 gblnum 0 gblgen 0 gblbrk 0][Cleared]

   Found stale lock [type 10c00001 offset 30640128 v 29, hb offset 3207168

         gen 27, mode 1, owner 55241293-ff2e34de-acbf-f079595b2257 mtime 288

         num 0 gblnum 0 gblgen 0 gblbrk 0][Cleared]

   Found stale lock [type 10c00001 offset 30642176 v 30, hb offset 3207168

         gen 27, mode 1, owner 55241293-ff2e34de-acbf-f079595b2257 mtime 297

         num 0 gblnum 0 gblgen 0 gblbrk 0][Cleared]

   Found stale lock [type 10c00001 offset 30644224 v 31, hb offset 3207168

         gen 27, mode 1, owner 55241293-ff2e34de-acbf-f079595b2257 mtime 306

         num 0 gblnum 0 gblgen 0 gblbrk 0][Cleared]

   Found stale lock [type 10c00001 offset 30646272 v 34, hb offset 3207168

         gen 27, mode 1, owner 55241293-ff2e34de-acbf-f079595b2257 mtime 146756

         num 0 gblnum 0 gblgen 0 gblbrk 0][Cleared]

   Found stale lock [type 10c00001 offset 30648320 v 33, hb offset 3207168

         gen 27, mode 1, owner 55241293-ff2e34de-acbf-f079595b2257 mtime 324

         num 0 gblnum 0 gblgen 0 gblbrk 0][Cleared]

   Found stale lock [type 10c00001 offset 30650368 v 34, hb offset 3207168

         gen 27, mode 1, owner 55241293-ff2e34de-acbf-f079595b2257 mtime 333

         num 0 gblnum 0 gblgen 0 gblbrk 0][Cleared]

   Found stale lock [type 10c00001 offset 30652416 v 35, hb offset 3207168

         gen 27, mode 1, owner 55241293-ff2e34de-acbf-f079595b2257 mtime 342

         num 0 gblnum 0 gblgen 0 gblbrk 0][Cleared]

   Found stale lock [type 10c00001 offset 30654464 v 36, hb offset 3207168

         gen 27, mode 1, owner 55241293-ff2e34de-acbf-f079595b2257 mtime 351

         num 0 gblnum 0 gblgen 0 gblbrk 0][Cleared]

   Found stale lock [type 10c00001 offset 30656512 v 37, hb offset 3207168

         gen 27, mode 1, owner 55241293-ff2e34de-acbf-f079595b2257 mtime 360

         num 0 gblnum 0 gblgen 0 gblbrk 0][Cleared]

   Found stale lock [type 10c00001 offset 30658560 v 38, hb offset 3207168

         gen 27, mode 1, owner 55241293-ff2e34de-acbf-f079595b2257 mtime 369

         num 0 gblnum 0 gblgen 0 gblbrk 0][Cleared]

   Found stale lock [type 10c00001 offset 30660608 v 39, hb offset 3207168

         gen 27, mode 1, owner 55241293-ff2e34de-acbf-f079595b2257 mtime 378

         num 0 gblnum 0 gblgen 0 gblbrk 0][Cleared]

   Found stale lock [type 10c00001 offset 30662656 v 40, hb offset 3207168

         gen 27, mode 1, owner 55241293-ff2e34de-acbf-f079595b2257 mtime 387

         num 0 gblnum 0 gblgen 0 gblbrk 0][Cleared]

   Found stale lock [type 10c00001 offset 30664704 v 41, hb offset 3207168

         gen 27, mode 1, owner 55241293-ff2e34de-acbf-f079595b2257 mtime 396

         num 0 gblnum 0 gblgen 0 gblbrk 0][Cleared]

   Found stale lock [type 10c00001 offset 30666752 v 42, hb offset 3207168

         gen 27, mode 1, owner 55241293-ff2e34de-acbf-f079595b2257 mtime 405

         num 0 gblnum 0 gblgen 0 gblbrk 0][Cleared]

   Found stale lock [type 10c00001 offset 30668800 v 43, hb offset 3207168

         gen 27, mode 1, owner 55241293-ff2e34de-acbf-f079595b2257 mtime 414

         num 0 gblnum 0 gblgen 0 gblbrk 0][Cleared]

   Found stale lock [type 10c00001 offset 30670848 v 62, hb offset 3207168

         gen 27, mode 1, owner 55241293-ff2e34de-acbf-f079595b2257 mtime 149421

         num 0 gblnum 0 gblgen 0 gblbrk 0][Cleared]

   Found stale lock [type 10c00001 offset 30672896 v 45, hb offset 3207168

         gen 27, mode 1, owner 55241293-ff2e34de-acbf-f079595b2257 mtime 432

         num 0 gblnum 0 gblgen 0 gblbrk 0][Cleared]

   Found stale lock [type 10c00001 offset 30674944 v 46, hb offset 3207168

         gen 27, mode 1, owner 55241293-ff2e34de-acbf-f079595b2257 mtime 441

         num 0 gblnum 0 gblgen 0 gblbrk 0][Cleared]

   Found stale lock [type 10c00001 offset 30681088 v 275, hb offset 3207168

         gen 27, mode 1, owner 55241293-ff2e34de-acbf-f079595b2257 mtime 47703

         num 0 gblnum 0 gblgen 0 gblbrk 0][Cleared]

   Found stale lock [type 10c00001 offset 30695424 v 99, hb offset 3207168

         gen 27, mode 1, owner 55241293-ff2e34de-acbf-f079595b2257 mtime 47767

         num 0 gblnum 0 gblgen 0 gblbrk 0][Cleared]

   Found stale lock [type 10c00001 offset 30697472 v 98, hb offset 3207168

         gen 27, mode 1, owner 55241293-ff2e34de-acbf-f079595b2257 mtime 47760

         num 0 gblnum 0 gblgen 0 gblbrk 0][Cleared]

   Found stale lock [type 10c00001 offset 30838784 v 192, hb offset 3207168

         gen 27, mode 1, owner 55241293-ff2e34de-acbf-f079595b2257 mtime 47689

         num 0 gblnum 0 gblgen 0 gblbrk 0][Cleared]

   Found stale lock [type 10c00001 offset 30840832 v 193, hb offset 3207168

         gen 27, mode 1, owner 55241293-ff2e34de-acbf-f079595b2257 mtime 47701

         num 0 gblnum 0 gblgen 0 gblbrk 0][Cleared]

   Found stale lock [type 10c00001 offset 30842880 v 194, hb offset 3207168

         gen 27, mode 1, owner 55241293-ff2e34de-acbf-f079595b2257 mtime 47712

         num 0 gblnum 0 gblgen 0 gblbrk 0][Cleared]

   Found stale lock [type 10c00001 offset 30844928 v 195, hb offset 3207168

         gen 27, mode 1, owner 55241293-ff2e34de-acbf-f079595b2257 mtime 47736

         num 0 gblnum 0 gblgen 0 gblbrk 0][Cleared]

   Found stale lock [type 10c00001 offset 30846976 v 196, hb offset 3207168

         gen 27, mode 1, owner 55241293-ff2e34de-acbf-f079595b2257 mtime 47751

         num 0 gblnum 0 gblgen 0 gblbrk 0][Cleared]

Phase 4: Checking pathname and connectivity.

Phase 5: Checking resource reference counts.

ON-DISK ERROR: FB inconsistency found: (25,0) allocated in bitmap, but never used

Total Errors Found:           2

Total Errors Fixed:           1

Total Partially Fixed errors: 0

[root@localhost:~] voma -m vmfs -f check -d /vmfs/devices/disks/mpx.vmhba39\:C0\:T0\:L0\:2

Checking if device is actively used by other hosts

Running VMFS Checker version 1.2 in check mode

Initializing LVM metadata, Basic Checks will be done

Phase 1: Checking VMFS header and resource files

   Detected VMFS file system (labeled:'USB_ESXi_MASTER') with UUID:55240f29-a13f992b-3aca-f079595b2257, Version 5:61

Phase 2: Checking VMFS heartbeat region

Phase 3: Checking all file descriptors.

Phase 4: Checking pathname and connectivity.

Phase 5: Checking resource reference counts.

ON-DISK ERROR: FB inconsistency found: (25,0) allocated in bitmap, but never used

Total Errors Found:           1

[root@localhost:~]

Thx Everybody

[porantes]

We ran into similar issue. We resolve it by renaming the vmx.lck file then simply added the VM to inventory. Let me give scenario we ran into,

Power outage occurred all hosts and Dell Equalogic SANs attached to 2 UPS's in same rack. UPS 1 failed, 1 SAN went offline and a switch. started shutting down VM's and the other UPS died. Needles to say the VM's were in process of vmotion I think because of one host was also shut down.

Power returned and exchange would not start. VM said orphaned.  Removed from inventory and tried to add back in and vmx was greyed out. went to 2nd host and tried, vmx was also greyed out. 3rd host i was able to add to inventory but the VM would never load, imported as orphaned no file name. I removed it from inventory on host #3

Browsed the data store on host #3 where exchange was located and renamed the vmx.lck file and then right click the vmx file and added it back into inventory. The VM appeared in inventory with correct naming convention and powered on in safe mode.A couple reboots later and we were back in business.

[2nia]

Set maintenance Mode and reboot host, and then register vm again, no need to delete locked file