VMware Cloud Community
sporty
Contributor
Contributor
Jump to solution

"Unable to access a file since it is locked"

I can't access a "Win XP SP2 -flat.vmdk" file on a SAN shared LUN in our ESX 3.5 infrastructure (3 ESX servers). I created a new VM. Attached this file to this VM. I started this VM and got this error message:

Unable to access a file since it is locked

When I copy the locked file to another location, I have this error message:

root@esx3s01 Win XP SP2# cp "Win XP SP2-flat.vmdk" /vmfs/volumes/ESX_SAN_Storage03/WinXP_SP2/

cp: cannot open `Win XP SP2-flat.vmdk' for reading: Device or resource busy

How do I know why it is locked? I ran ps ax |grep "Win XP SP2" on all our hosts. Didn't find anything. I can't afford restarting my hosts at the moment.

Reply
0 Kudos
1 Solution

Accepted Solutions
ctfoster
Expert
Expert
Jump to solution

Found this on the community board. - It'll give you some idea what host is holding the file open.

Sometimes a file or set of files in a VMFS become locked and any attempts to edit them or delete will give a device or resource busy error, even though the vm associated with the files is not running. If the vm is running then you would need to stop the vm to manipulate the files. If you know that the vm is stopped then you need to find the ESX server that has the files locked and then stop the process that is locking the file(s).

  • 1. Logon to the ESX host where the VM was last known to be running.

  • 2. vmkfstools -D /vmfs/volumes/path/to/file to dump information on the file into /var/log/vmkernel

  • 3. less /var/log/vmkernel and scroll to the bottom, you will see output like below:

  • a. Nov 29 15:49:17 vm22 vmkernel: 2:00:15:18.435 cpu6:1038)FS3: 130: <START vmware-16.log>

  • b. Nov 29 15:49:17 vm22 vmkernel: 2:00:15:18.435 cpu6:1038)Lock [type 10c00001 offset 30439424 v 21, hb offset 4154368

  • c. Nov 29 15:49:17 vm22 vmkernel: gen 66493, mode 1, owner 46c60a7c-94813bcf-4273-0017a44c7727 mtime 8781867] ß Bold type added to number for emphasis.

  • d. Nov 29 15:49:17 vm22 vmkernel: 2:00:15:18.435 cpu6:1038)Addr <4, 588, 7>, gen 20, links 1, type reg, flags 0x0, uid 0, gid 0, mode 644

  • e. Nov 29 15:49:17 vm22 vmkernel: 2:00:15:18.435 cpu6:1038)len 23973, nb 1 tbz 0, zla 2, bs 65536

  • f. Nov 29 15:49:17 vm22 vmkernel: 2:00:15:18.435 cpu6:1038)FS3: 132: <END vmware-16.log>

  • 4. The owner of the lock is on line 3c, the last part is all you need, in this case 0017a44c7727

  • 5. esxcfg-info | grep -i 'system uuid' | awk -F '-' '{print $NF}' will display the system uuid of the esx server. You need to run the esxcfg-info command on each esx server in the cluster to discover the owner.

  • 6. When you find the ESX server that matches the uuid owner, logon to that ESX server and run the command: ps -elf|grep vmname where vmname is the problem vm. Example output below:

  • a. 4 S root 7570 1 0 65 -10 - 435 schedu Nov27 ? 00:00:02 /usr/lib/vmware/bin/vmkload_app /usr/lib/vmware/bin/vmware-vmx -ssched.group=host/user/pool2 -@ pipe=/tmp/vmhsdaemon-0/vmxf7fb85ef5d8b3522;vm=f7fb85ef5d8b3522 /vmfs/volumes/470e25b6-37016b37-a2b3-001b78bedd4c/iu-lsps-vstest/iu-lsps-vstest.vmx0

  • 7. Since there is a process running, pid 7570 in the example, you need to kill it by following steps 5-12 on stopping a VM above

  • 8. Once the kill is complete the files should be released.

View solution in original post

Reply
0 Kudos
13 Replies
ctfoster
Expert
Expert
Jump to solution

I'll make the obvious point that the file looks like it's in use by another active VM.

Was the file part of another VM ? Which VM did you search the process list for - the name of the new one or the old one ?

Also you should use the vmdktools -i command to clone the vmdk around rather than cp - since cp can lead to frgmentation of the file and possible performance issues.

Reply
0 Kudos
sporty
Contributor
Contributor
Jump to solution

I searched for "Win XP SP2", the name of the old and the new Vms.

The file was a part of a VM (a template, to be more precise). I wasn't able to deploy a VM from this template and desided to remove it from inventory hoping it will "unlock" the file.

However the file is still locked and I don't know how to figure out what process is holding it. It might be anything on any ESX host.

I know vmdktools is a better way of handling vmfs files but now I just want to "unlock" this file ASAP. It's our only Windows XP template.

If I run vmkfstools, I have the following error message:

# vmkfstools -i "Win XP SP2-flat.vmdk" /vmfs/volumes/ESX_SAN_Storage03/WinXP_SP2/Win-XP-SP2-flat.vmdk

DiskLib_Check() failed for source disk Device or resource busy (1048585).

Reply
0 Kudos
ctfoster
Expert
Expert
Jump to solution

Found this on the community board. - It'll give you some idea what host is holding the file open.

Sometimes a file or set of files in a VMFS become locked and any attempts to edit them or delete will give a device or resource busy error, even though the vm associated with the files is not running. If the vm is running then you would need to stop the vm to manipulate the files. If you know that the vm is stopped then you need to find the ESX server that has the files locked and then stop the process that is locking the file(s).

  • 1. Logon to the ESX host where the VM was last known to be running.

  • 2. vmkfstools -D /vmfs/volumes/path/to/file to dump information on the file into /var/log/vmkernel

  • 3. less /var/log/vmkernel and scroll to the bottom, you will see output like below:

  • a. Nov 29 15:49:17 vm22 vmkernel: 2:00:15:18.435 cpu6:1038)FS3: 130: <START vmware-16.log>

  • b. Nov 29 15:49:17 vm22 vmkernel: 2:00:15:18.435 cpu6:1038)Lock [type 10c00001 offset 30439424 v 21, hb offset 4154368

  • c. Nov 29 15:49:17 vm22 vmkernel: gen 66493, mode 1, owner 46c60a7c-94813bcf-4273-0017a44c7727 mtime 8781867] ß Bold type added to number for emphasis.

  • d. Nov 29 15:49:17 vm22 vmkernel: 2:00:15:18.435 cpu6:1038)Addr <4, 588, 7>, gen 20, links 1, type reg, flags 0x0, uid 0, gid 0, mode 644

  • e. Nov 29 15:49:17 vm22 vmkernel: 2:00:15:18.435 cpu6:1038)len 23973, nb 1 tbz 0, zla 2, bs 65536

  • f. Nov 29 15:49:17 vm22 vmkernel: 2:00:15:18.435 cpu6:1038)FS3: 132: <END vmware-16.log>

  • 4. The owner of the lock is on line 3c, the last part is all you need, in this case 0017a44c7727

  • 5. esxcfg-info | grep -i 'system uuid' | awk -F '-' '{print $NF}' will display the system uuid of the esx server. You need to run the esxcfg-info command on each esx server in the cluster to discover the owner.

  • 6. When you find the ESX server that matches the uuid owner, logon to that ESX server and run the command: ps -elf|grep vmname where vmname is the problem vm. Example output below:

  • a. 4 S root 7570 1 0 65 -10 - 435 schedu Nov27 ? 00:00:02 /usr/lib/vmware/bin/vmkload_app /usr/lib/vmware/bin/vmware-vmx -ssched.group=host/user/pool2 -@ pipe=/tmp/vmhsdaemon-0/vmxf7fb85ef5d8b3522;vm=f7fb85ef5d8b3522 /vmfs/volumes/470e25b6-37016b37-a2b3-001b78bedd4c/iu-lsps-vstest/iu-lsps-vstest.vmx0

  • 7. Since there is a process running, pid 7570 in the example, you need to kill it by following steps 5-12 on stopping a VM above

  • 8. Once the kill is complete the files should be released.

Reply
0 Kudos
sporty
Contributor
Contributor
Jump to solution

Thanks a lot! I have found the VM that was locking the file. The file that supposed to be a template file was used as a primary partition for one of the Windows XP VM machines.

I will be having a serious talk with a guy who did this Smiley Happy

Reply
0 Kudos
ctfoster
Expert
Expert
Jump to solution

Go easy - he thought he was doing you a favour ! Thanks for the points.

Reply
0 Kudos
john_gallucci
Enthusiast
Enthusiast
Jump to solution

There are several posts similar to this, but I found this solution to be the only one that worked!

I found the easiest way to find dead processes is to put the ESX host into maintenance mode first and then look around (that way you dont accidentally kill a good VM).

No reboot required for me, putting the host into maintenance mode released the locks and all was good.

Thanks for the post!

Reply
0 Kudos
burdweiser
Enthusiast
Enthusiast
Jump to solution

What happens if the owner is all zeros?

Reply
0 Kudos
wally
Enthusiast
Enthusiast
Jump to solution

I have the same issue:

May 13 04:35:36 esx24 vmkernel: gen 213, mode 0, owner 00000000-00000000-0000-000000000000 mtime 6130337

Did you find a solution to the 'all zeros' ?

Reply
0 Kudos
mih
Contributor
Contributor
Jump to solution

I know this is an old post, but anyone got a solution to the owner being all zeros ?

Reply
0 Kudos
fannonland
Contributor
Contributor
Jump to solution

I am trying to follow your steps but when I enter in vmkfstools -D /vmfs/volumes/path/to/file and trying to use .vmdk or .vmx I get

Could not get the dump information for 'my virtual machin' (rv -1)

Error: Inappropriate ioctl for device

Anyone know what that means?

Reply
0 Kudos
iaco
Contributor
Contributor
Jump to solution

i had a case with a all zero owner

i tried running vmkfstools from other esx hosts that had access to the same luns until i found one ESX that reported a valid owner

Reply
0 Kudos
nancy31
Enthusiast
Enthusiast
Jump to solution

Been experiencing the same problem.  And I hope someone could give an up to date advice.  Thanks in advance http://imagicon.info/cat/5-59/1.gif

Reply
0 Kudos
iaco
Contributor
Contributor
Jump to solution

as i've said in my previous post i've been able to successfully identify a 00000000-00000000-0000-000000000000 owner by running vmkfstools on another esx.

full details here:

http://pleasedonttouchthescreen.blogspot.com/2011/09/how-to-unlock-vmdk-file.html

Reply
0 Kudos