Template vmdk locked

[imclaren]

Hi,

I tried to clone a VM from a template today, which I've done several times before, but today it didn't work (it said the vmdk file couldn't be opened). I converted it to a VM and tried to power it on, and it came up saying that it was locked.

Does anyone know how to clear the lock?

Thanks,

Iain

[opbz]

sounds like it might be in use at the moment...

How many ESX servers are accessing this template? If its just one then go to the folder where its located and run

lsof

it should tell you what process is using this vmdk once you identify it you should be able to kill that process and then try using the template again

[imclaren]

Hi,

The VM is on a SAN, which is accessible by 4 ESX servers - three 3.5 ones, and a 3i one. I've tried the lsof command on the 'thick' servers, which drew a blank. I'm not sure if I can do the same on the 3i server?

Cheers,

Iain

[atbnet]

Hi Ive come across this before, this should fix the locked file.

Sometimes a file or set of files in a VMFS become locked and any attempts to edit them or delete will give a device or resource busy error, even though the vm associated with the files is not running. If the vm is running then you would need to stop the vm to manipulate the files. If you know that the vm is stopped then you need to find the ESX server that has the files locked and then stop the process that is locking the file(s).

1. Logon to the ESX host where the VM was last known to be running.

2. vmkfstools -D /vmfs/volumes/path/to/file to dump information on the file into /var/log/vmkernel

3. less /var/log/vmkernel and scroll to the bottom, you will see output like below:

a. Nov 29 15:49:17 vm22 vmkernel: 2:00:15:18.435 cpu6:1038)FS3: 130: <START vmware-16.log>

b. Nov 29 15:49:17 vm22 vmkernel: 2:00:15:18.435 cpu6:1038)Lock [type 10c00001 offset 30439424 v 21, hb offset 4154368

c. Nov 29 15:49:17 vm22 vmkernel: gen 66493, mode 1, owner 46c60a7c-94813bcf-4273-0017a44c7727 mtime 8781867]  Bold type added to number for emphasis.

d. Nov 29 15:49:17 vm22 vmkernel: 2:00:15:18.435 cpu6:1038)Addr <4, 588, 7>, gen 20, links 1, type reg, flags 0x0, uid 0, gid 0, mode 644

e. Nov 29 15:49:17 vm22 vmkernel: 2:00:15:18.435 cpu6:1038)len 23973, nb 1 tbz 0, zla 2, bs 65536

f. Nov 29 15:49:17 vm22 vmkernel: 2:00:15:18.435 cpu6:1038)FS3: 132: <END vmware-16.log>

4. The owner of the lock is on line 3c, the last part is all you need, in this case 0017a44c7727

5. esxcfg-info | grep -i 'system uuid' | awk -F '-' '{print $NF}' will display the system uuid of the esx server. You need to run the esxcfg-info command on each esx server in the cluster to discover the owner.

6. When you find the ESX server that matches the uuid owner, logon to that ESX server and run the command: ps -elf|grep vmname where vmname is the problem vm. Example output below:

a. 4 S root 7570 1 0 65 -10 - 435 schedu Nov27 ? 00:00:02 /usr/lib/vmware/bin/vmkload_app /usr/lib/vmware/bin/vmware-vmx -ssched.group=host/user/pool2 -@ pipe=/tmp/vmhsdaemon-0/vmxf7fb85ef5d8b3522;vm=f7fb85ef5d8b3522 /vmfs/volumes/470e25b6-37016b37-a2b3-001b78bedd4c/iu-lsps-vstest/iu-lsps-vstest.vmx0

7. Since there is a process running, pid 7570 in the example, you need to kill it by following steps 5-12 on stopping a VM above

8. Once the kill is complete the files should be released.

Andy, VMware Certified Professional (VCP),

If you found this information useful please award points using the buttons at the top of the page accordingly.

[imclaren]

Hi,

Thanks for this. I think the last host that it was running on was the 3i host...which seems to use /var/log/messages.

I've tried the vmkfstools -D command on all the hosts, and each report the owner as being 00000000-00000000-0000-000000000000.

I've even tried rebooting the 3i host, but the VC stil thinks the files are locked.

Cheers,

Iain

[imclaren]

Right...I've now rebooted all my ESX hosts, and the file's still locked

I'm not that bothered - I can always do the template again (assuming I can delete it), but I'd like to know what's caused it / how to fix it in case it ever happens to a VM.

Cheers,

Iain

[imclaren]

Hi,

I'm re-visiting this problem to see if I can cure it. I tried doing vmkfstools -D <path_to_file>, and checking the vmkernel log, which gives:

Jan 27 12:03:02 smca-edi-esx4 vmkernel: 63:01:47:30.038 cpu1:1041)FS3: 130: <START W2003template-flat.vmdk>

Jan 27 12:03:02 smca-edi-esx4 vmkernel: 63:01:47:30.038 cpu1:1041)Lock [type 10c00001 offset 30201856 v 107, hb offset 3721728

Jan 27 12:03:02 smca-edi-esx4 vmkernel: gen 20, mode 1, owner 492bca71-b8f7da90-accf-001b24a2e178 mtime 1083]

Jan 27 12:03:02 smca-edi-esx4 vmkernel: 63:01:47:30.038 cpu1:1041)Addr <4, 40, 27>, gen 28, links 1, type reg, flags 0x0, uid 0, gid 0, mode 600

Jan 27 12:03:02 smca-edi-esx4 vmkernel: 63:01:47:30.038 cpu1:1041)len 25769803776, nb 24576 tbz 10391, zla 3, bs 1048576

Jan 27 12:03:02 smca-edi-esx4 vmkernel: 63:01:47:30.038 cpu1:1041)FS3: 132: <END W2003template-flat.vmdk>

I can't do anything with the VM, in terms of powering it on, deploying a VM from it (when it's set as a template), copy it (using cp or cloning it). I've not tried deleting it, but I don't really want to as I won't have understood the problem.

All ESX hosts have been bounced, but still the lock remains!

Any further suggestions?

Thanks,

Iain

[imclaren]

The line in the log -001b24a2e178 - looks like a MAC address. The closest match I can find (at the moment) is one of my ESX blades, but it is 00:1B:24:A2:E8:A4...

Edit: 00:1B:24:A2:E1:78 is a MAC address from one of my other blades. I'll try shutting it down and see what happens...

[imclaren]

Ah-haaaaa! I've sussed it :smileysilly:

I migrated the VMs off the host with that MAC address, and re-checked the log - the lock had moved to another host.

So...I checked the last VM that I'd created from the template, and lo and behold it was using the vmdk file in the template folder instead of its own folder.

I dunno if something went screwy when it was deployed or what, but at least I've got to the bottom of it. Interestingly the folder of the deployed VM contains a vmdk file who's name is the name of the VM. Maybe I just did something wrong!!! ?:|

Cheers,

Iain

[Erik_Zandboer · Accepted solution]

This is an issue I have seen before. Sometimes when you deploy a VM from a template, the VM actually gets linked to the TEMPLATEs virtual disk instead of his own virtual disk. Everything apperas to run well, right until you:

1) delete the deployed VM and find your template disk is gone;

2) You try to deploy another template and find the virtual disk is locked somehow (being your issue)

So, if a template is locked, always look for any (recently deployed) VM that is using that disk instead of his own.

[imclaren]

Thanks for that. Good to know it (probably) wasn't operator-error!

[aldikan]

Great info,

I actually encountered this before as well,

I thought it must be just a glitch, was wondering fow a while how in the world did my VM got a vmdk from the template..., but since it was a single occurence, did not investigate further..

Alex

[imclaren]

What I also found was that the other virtual hard disk file in my deployed VM that was actually the hard disk file which should have been used by the new VM - so it was effectively an unused copy of my template's hard disk file (but had the name of the new VM - rsa-test-flat.vmdk)

So I was able to move that back to the template's folder, rename it and edit the vmdk descriptor file to re-assign it to the template.