VMFS File Lock (Stale?) - Unable to Add VM to Inventory

[rajeshvs]

Hello experts,

One of our ESXi 5.0  Servers crashed recently and the VM it was running from shared storage (FC) is inaccessible now.

We are unable to access any files on the VM Folder, either to download or add to Inventory.

From whatever I could troubleshoot, I could see the file was locked, may be stale lock.

PowerIQ-13.png             PowerIQ-9.png              PowerIQ.vmx

/vmfs/volumes/4ff265e1-1da7beeb-1add-0025b500004f/PowerIQ # vmkfstools -D vmware.log

Lock [type 10c00001 offset 5038080 v 184, hb offset 3588096

gen 843, mode 1, owner 503f7ab8-3b9fd520-8988-0025b500002f mtime 478447 nHld 0 nOvf 0]

Addr <4, 1, 148>, gen 183, links 1, type reg, flags 0, uid 0, gid 0, mode 644

len 266773, nb 1 tbz 0, cow 0, newSinceEpoch 0, zla 1, bs 1048576

/vmfs/volumes/4ff265e1-1da7beeb-1add-0025b500004f/PowerIQ # vmkfstools -D PowerIQ.vmx

Lock [type 10c00001 offset 5015552 v 320, hb offset 3588096

gen 843, mode 1, owner 503f7ab8-3b9fd520-8988-0025b500002f mtime 478458 nHld 0 nOvf 0]

Addr <4, 1, 137>, gen 165, links 1, type reg, flags 0, uid 0, gid 0, mode 100700

len 3384, nb 1 tbz 0, cow 0, newSinceEpoch 0, zla 2, bs 8192

/vmfs/volumes/4ff265e1-1da7beeb-1add-0025b500004f/PowerIQ # vmkfstools -D PowerIQ.vmdk

Lock [type 10c00001 offset 5062656 v 229, hb offset 3579904

gen 1039, mode 0, owner 00000000-00000000-0000-000000000000 mtime 2830 nHld 0 nOvf 0]

Addr <4, 1, 160>, gen 203, links 1, type reg, flags 0, uid 0, gid 0, mode 100600

len 604, nb 0 tbz 0, cow 0, newSinceEpoch 0, zla 4305, bs 8192

/vmfs/volumes/4ff265e1-1da7beeb-1add-0025b500004f/PowerIQ # vmkfstools -D PowerIQ-flat.vmdk

Lock [type 10c00001 offset 4790272 v 1406, hb offset 3588096

gen 843, mode 2, owner 00000000-00000000-0000-000000000000 mtime 752243 nHld 1 nOvf 0]

RO Owner[0] HB Offset 3588096 503f7ab8-3b9fd520-8988-0025b500002f  <<< !!!---------- Crashed ESXi

Addr <4, 1, 27>, gen 31, links 1, type reg, flags 0, uid 0, gid 0, mode 600

len 171798691840, nb 163840 tbz 0, cow 0, newSinceEpoch 0, zla 3, bs 1048576

/vmfs/volumes/4ff265e1-1da7beeb-1add-0025b500004f/PowerIQ #

----

This is the MAC Of the ESX host  - 0025b500002f – which crashed, We have rebuilt it and tried accessing the VMFiles thorough that host too, but getting error accessing it.

/vmfs/volumes/4ff265e1-1da7beeb-1add-0025b500004f/PowerIQ # touch PowerIQ.vmx

touch: PowerIQ.vmx: Invalid argument

/vmfs/volumes/4ff265e1-1da7beeb-1add-0025b500004f/PowerIQ # touch PowerIQ-flat.vmdk

touch: PowerIQ-flat.vmdk: Invalid argument

/vmfs/volumes/4ff265e1-1da7beeb-1add-0025b500004f/PowerIQ # more vmware.log

more: vmware.log: Invalid argument

Appreciate your  help.

Thanks in advance,

Regards,

Rajesh

[DavoudTeimouri]

Hi,

Check ESXi events and find previous host which it was hosting the virtual machine, move it to that and try to power on, it's worked for me.

Also you can check this: http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=10051

[rajeshvs]

Hi Davoud,

Thanks for the response.

I Tried that already, but no luck. When I try to access the VM files from the same old Host , its not allowing me to access it.

There is a lock in the file from the same host, but its stale it seems. Not sure how to release that.

Regards

Rajesh

[DavoudTeimouri]

You're welcome,

If you have enough resources, you can restart your servers one by one and files will be unlocked.

[ScreamingSilenc]

Try restarting your hostd from your vSphere SSH console.

/etc/init.d/hostd restart

/etc/init.d/vpxa restart

[rajeshvs]

Tried, No luck.

[rajeshvs]

/etc/init.d # ./hostd restart

watchdog-hostd: Terminating watchdog process with PID 4913

hostd stopped.

[113176] Begin 'hostd ++min=0,swap,group=hostd /etc/vmware/hostd/config.xml', min-uptime = 60, max-quick-failures = 1, max-total-failures = 1000000, bg_pid_file = ''

hostd started.

/etc/init.d # ./vpxa restart

watchdog-vpxa: Terminating watchdog process with PID 5184

vpxa stopped.

[113343] Begin '/usr/lib/vmware/vpxa/bin/vpxa ++min=0,swap,group=vpxa -D /etc/vmware/vpxa', min-uptime = 60, max-quick-failures = 1, max-total-failures = 1000000, bg_pid_file = ''

/etc/init.d # vmkfstools -D /vmfs/volumes/4ff265e1-1da7beeb-1add-0025b500004f/PowerIQ/vmware.log

Lock [type 10c00001 offset 5038080 v 184, hb offset 3588096

gen 843, mode 1, owner 503f7ab8-3b9fd520-8988-0025b500002f mtime 478447 nHld 0 nOvf 0]

Addr <4, 1, 148>, gen 183, links 1, type reg, flags 0, uid 0, gid 0, mode 644

len 266773, nb 1 tbz 0, cow 0, newSinceEpoch 0, zla 1, bs 1048576

sbin # esxcfg-vmknic -l

Interface  Port Group/DVPort   IP Family IP Address                              Netmask         Broadcast       MAC Address       MTU     TSO MSS   Enabled Type               

vmk0       Management Network  IPv4      10.64.122.30                            255.255.252.0   10.64.123.255   00:25:b5:00:00:2f 1500    65535     true    STATIC             

/sbin #

/sbin # vmkfstools -D /vmfs/volumes/4ff265e1-1da7beeb-1add-0025b500004f/PowerIQ/PowerIQ-flat.vmdk

Lock [type 10c00001 offset 4790272 v 1406, hb offset 3588096

gen 843, mode 2, owner 00000000-00000000-0000-000000000000 mtime 752243 nHld 1 nOvf 0]

RO Owner[0] HB Offset 3588096 503f7ab8-3b9fd520-8988-0025b500002f

Addr <4, 1, 27>, gen 31, links 1, type reg, flags 0, uid 0, gid 0, mode 600

len 171798691840, nb 163840 tbz 0, cow 0, newSinceEpoch 0, zla 3, bs 1048576

============

If i try to copy the file, I get : can't open 'PowerIQ-flat.vmdk': Invalid argument

This VM is not running anywhere else.