Unable to access the file it is locked

[Raghukalloor]

hi ,

This is regarding one of the VM gives the error as "Unable to access the file , it is locked ". cannot able to Power ON . can somebody help me on this.

[ewannema]

Is there another VM using the VMDK file that this machine has defined?

[Raghukalloor]

I don't have another vm which uses the vmdk file. but when i try to power ON the machine it says "Unable to access the file it is locked".

can u suggest me what needs to be done.

[rpartmann]

Hi,

i assume you mean a vmdk file which is locked. sometimes the hostd does that.

try a service mgmt-vmware restart

Reinhard.

ps: Award points if you find answers helpful. Thanks.

[petedr]

Is there a process running that is holding that lock,

try maybe ps -ef |grep 'your vm name'

[wilson94t]

Anyone have the solution to this one? We have this problem today. VC gives the message "Unable to access a file since it is locked"

restarting mgmt-vmware makes no difference. Since the VM is off, we also moved it to another ESX host (cold migration, no relocation of files) and the error follows to the next host. There are mesages about the file being in use. ps shows nothing in use.

ps does not show anything in use.

lsof does not show anything in use.

the vmware.log file shows that the device or resoruce is busy (but i can't see why that would be). I rename the directory VMGUEST with no problem.

LOGS:

From /var/log/vmkernel, you can see what happens each time we try to "power on" the virtual machine called "VMGUEST"

Feb 9 13:25:03 ESXSERVER vmkernel: gen 16770, mode 1, owner 47346501-7cade4a7-bdde-001b78b8b5e4 mtime 657024]

Feb 9 13:25:03 ESXSERVER vmkernel: 53:01:17:06.044 cpu6:1037)FS3: 752: Clearing stale owner for lock[type 10c00001 offset 60389376 v 15, hb offset 3379200

Feb 9 13:25:03 ESXSERVER vmkernel: gen 16770, mode 1, owner 47346501-7cade4a7-bdde-001b78b8b5e4 mtime 657024]

Feb 9 13:25:03 ESXSERVER vmkernel: 53:01:17:06.257 cpu6:1038)FS3: 1714: Checking if lock holders are live for lock [type 10c00001 offset 73555968 v 23, hb offse

t 3366912

Feb 9 13:25:03 ESXSERVER vmkernel: gen 64, mode 1, owner 4769360d-8d6f34ee-e95b-001b78b8b884 mtime 634]

Feb 9 13:25:03 ESXSERVER vmkernel: 53:01:17:06.257 cpu6:1038)FS3: 752: Clearing stale owner for lock[type 10c00001 offset 73555968 v 23, hb offset 3366912

Feb 9 13:25:03 ESXSERVER vmkernel: gen 64, mode 1, owner 4769360d-8d6f34ee-e95b-001b78b8b884 mtime 634]

Feb 9 13:25:42 ESXSERVER vmkernel: 53:01:17:45.761 cpu6:1038)World: vm 1320: 693: Starting world vmkping with flags 4

Feb 9 13:25:53 ESXSERVER vmkernel: 53:01:17:56.928 cpu0:1320)World: vm 1320: 3867: Killing self with status=0x0:Success

Feb 9 13:43:39 ESXSERVER vmkernel: 53:01:35:42.986 cpu4:1038)World: vm 1321: 693: Starting world vmware-vmx with flags 4

Feb 9 13:43:40 ESXSERVER vmkernel: 53:01:35:43.458 cpu4:1321)World: vm 1322: 693: Starting world vmm0:VMGUEST with flags 8

Feb 9 13:43:40 ESXSERVER vmkernel: 53:01:35:43.459 cpu4:1321)Sched: vm 1322: 4836: adding 'vmm0:VMGUEST': group 'host/user/pool1': cpu: shares=-3 min=0 max=-

1

Feb 9 13:43:40 ESXSERVER vmkernel: 53:01:35:43.459 cpu4:1321)Sched: vm 1322: 4849: renamed group 67 to vm.1321

Feb 9 13:43:40 ESXSERVER vmkernel: 53:01:35:43.459 cpu4:1321)Sched: vm 1322: 4863: moved group 67 to be under group 13

Feb 9 13:43:40 ESXSERVER vmkernel: 53:01:35:43.466 cpu4:1321)Swap: vm 1322: 1426: extending swap to 4194304 KB

Feb 9 13:43:40 ESXSERVER vmkernel: 53:01:35:43.604 cpu6:1321)FS3: 1714: Checking if lock holders are live for lock [type 10c00001 offset 73408512 v 32, hb offse

t 3976192

Feb 9 13:43:40 ESXSERVER vmkernel: gen 75280, mode 1, owner 47694b12-8f4f98a3-8c87-001b78b8b572 mtime 2406535]

Feb 9 13:43:44 ESXSERVER vmkernel: 53:01:35:47.687 cpu6:1321)Sched: vm 1322: 727: name='vmm0:VMGUEST'

Feb 9 13:43:44 ESXSERVER vmkernel: 53:01:35:47.687 cpu6:1321)CpuSched: vm 1322: 15654: zombified unscheduled world: runState=NEW

Feb 9 13:43:44 ESXSERVER vmkernel: 53:01:35:47.687 cpu6:1321)World: vm 1322: 2209: deathPending set; world not running, scheduling reap

Feb 9 13:43:44 ESXSERVER vmkernel: 53:01:35:47.955 cpu6:1321)World: vm 1321: 3867: Killing self with status=0x0:Success

Feb 9 13:46:19 ESXSERVER vmkernel: 53:01:38:22.988 cpu1:1038)World: vm 1323: 693: Starting world vmkping with flags 4

Feb 9 13:46:24 ESXSERVER vmkernel: 53:01:38:27.163 cpu7:1323)World: vm 1323: 3867: Killing self with status=0x0:Success

Feb 9 14:50:05 ESXSERVER vmkernel: 53:02:42:08.885 cpu0:1038)Config: 384: "Enabled" = 1, Old Value: 1, (Status: 0x0)

-

---

From vmware.log for the virtual machine VMGUEST:

Feb 09 14:51:38.992: vmx| VMXVmdb_LoadRawConfig: Loading raw config

Feb 09 14:51:38.997: vmx| DISK: OPEN scsi0:0 '/vmfs/volumes/47346508-67714175-6d82-001b78b8b99e/VMGUEST/VMGUEST.vmdk' persistent R[(null)]

Feb 09 14:51:39.000: vmx| DISKLIB-VMFS : "/vmfs/volumes/47346508-67714175-6d82-001b78b8b99e/VMGUEST/VMGUEST-flat.vmdk" : failed to open (1048585): AIOMgr_

Open failed. Type 3

Feb 09 14:51:39.000: vmx| DISKLIB-DSCPTR: Failed to open extents for descriptor file in normal mode

Feb 09 14:51:39.001: vmx| DISKLIB-LINK : "/vmfs/volumes/47346508-67714175-6d82-001b78b8b99e/VMGUEST/VMGUEST.vmdk" : failed to open (Device or resource bu

sy).

Feb 09 14:51:39.001: vmx| DISKLIB-CHAIN : "/vmfs/volumes/47346508-67714175-6d82-001b78b8b99e/VMGUEST/VMGUEST.vmdk" : failed to open (Device or resource bu

sy).

Feb 09 14:51:39.001: vmx| DISKLIB-LIB : Failed to open '/vmfs/volumes/47346508-67714175-6d82-001b78b8b99e/VMGUEST/VMGUEST.vmdk' with flags 0xa (Device o

r resource busy).

Feb 09 14:51:39.001: vmx| DISK: Cannot open disk "/vmfs/volumes/47346508-67714175-6d82-001b78b8b99e/VMGUEST/VMGUEST.vmdk": Device or resource busy (104858

5).

Feb 09 14:51:39.001: vmx| DISK: Failed to open disk '/vmfs/volumes/47346508-67714175-6d82-001b78b8b99e/VMGUEST/VMGUEST.vmdk' : Device or resource busy (10

48585) 3191

Feb 09 14:51:39.001: vmx| Msg_Post: Error

Feb 09 14:51:39.001: vmx| http://msg.disk.noBackEnd Cannot open the disk '/vmfs/volumes/47346508-67714175-6d82-001b78b8b99e/VMGUEST/VMGUEST.vmdk' or one of the

snapshot disks it depends on.

Feb 09 14:51:39.001: vmx| http://msg.disk.configureDiskError Reason: Device or resource busy.----

---

[ctfoster]

You may want to repost this as a new item. It will get lost at the end of an answered thread.

[wilson94t]

or maybe i'll just answer mysef here....

Symptom: A vmdk becomes locked.

Solution:

Check /var/log/vmkernel for a line which looks like:

vmkernel: gen 75280, mode 1, owner 47694b12-8f4f98a3-8c87-001b78b8b572 mtime 2406535]

From this line, observe the last field of the GUID: 001b78b8b572

This is the MAC address: 00:1b:78:b8:b5:72

If you have something like BladeLogic:

Create a text file of the hosts in the cluster - from BladeLogic:

vi //@/temp/clustername.txt

ESXHost1

ESXHost2

ESXHost3

...

ESXHost12

Next, use runcmd in parallel to check all ESX hosts for this MAC address quickly. In this case, we'll look for the MAC suffix of 72.

runcmd -h //@/temp/clustername.txt -p 11 -V -NH -e ifconfig |grep HW |grep 72 and in 2 seconds, get the answer.

If you don't have a tool like that, simply check each of your ESX servers for the MAC found in /var/log/vmkernel.

you can do this by logging into each console and typing: ifconfig |grep HW |grep 72 , or click around virtual center for a while....

Once the host is discovered, a couple possibilities are available.

vmkfstools -D /path/vmguest-flat.ext may be able to release the lock (but its not likely)

when this doesn't work (because it's not going to - if it does, play the lotto) put esx into maint mode.

After ESX is in maint mode, delete the file from another ESX host while you wait for the rebooting ESX to return.

The locks will now be released.

Its also possible to do this with the vmkfstools -L lunreset - but you want to be sure what you're doing there. This can cause problems with your virtual machines. (see man vmkfstools for more info about the dangers). In our experience, this will not solve the problem. You'll have to reboot - which, in our minds, is totally unacceptable. If anyone has a better way, I'd love to hear it.

[Engelsman]

Can you try removing and re-adding the vm to your inventory?

This worked for me a couple of times.

[depping]

I had the same issue a couple of weeks ago and ended up rebooting all esx hosts one by one... could not locate which host locked the file.

Duncan

My virtualisation blog: