Unable to access a file since it is locked - Something to look for

[djflux]

Good day,

I just wanted to post this information for others that might be having the same issue.

We had created a template on our new VI3 system (ESX 3.5 update 1 and VC 2.5 Update 1). We deployed almost 10 VMs from this template with no problems. When we tried to deploy another VM from the template VC popped up the error that it couldn't find the VMDK file associated with the template. Looking through vmware.log there was an error that stated it couldn't open the VMDK flat file because "Device or resource busy."

Thinking it could possibly be a template issue I converted the template to a VM and tried to power it on. I received the "Unable to access a file since it is locked" error. I tried almost everything I knew of to resolve the issue - migrating the VMs off the host and rebooting it, moving all the VMs back and rebooting our second host, cloning the template, copying the disk via vmkfstools - all failed because the datastore thought the file was in use.

I also followed the procedure in this document: http://communities.vmware.com/docs/DOC-2290 I even called VMware support and they were stumped as well. BTW, that is not a knock on VMware Support - they were very helpful and extremely knowledgeable and tried a lot of things I didn't know of.

Well as a last resort I was going to remove the datastore on which the template resided and recreate it. When I tried to remove it I received an error that stated that the datastore was in use, even though I migrated all of the VMs off the datastore and there were no running VMs using the datastore - or so I thought.

I removed the problem template/VM from the VC inventory. As a last resort I stopped presenting the LUN to one of the ESX hosts. I then tried to VMotion all of my other VMs to the host on which I had removed the problem datastore to reboot the host again. When I tried to VMotion the VMs the validaton failed stating the other host didn't have a datastore on which a VMDK that was attached to it resided. Looking at the settings of the VM that the validation process was barking about, it's virtual hard disk file was pointing to the template's VMDK instead of it's own.

Basically the deployment of the template to that VM failed. It create a new VMDK. but the templating process just pointed the new VM's virtual disk to the template's VMDK file. We think this was a result of trying the "Experimental" feature in the deployment wizard that allows one to edit the VM hardware before templating. We tried to increase the size of the VMDK file for the new VM. We think that choosing that option blew up the deployment process.

We have a ticket open with VMware support and they are calling us back on Monday to check on the results of our issue. We will let them know that we think this editing virtual hardware during the deployment process hosed up the template.

When VMware says that feature is "experimental," believe it. At the very least don't try to expand the templates VMDK when deploying from a template.

So if you receive the "Unable to access a file since it is locked" when powering on a VM, or receive an error about can't find the file when trying to deploy a new VM from a template, check ALL your VM's vmx files to make sure that one of them is not actually using that template/VMs VMDK file.

It took almost a full day's work to find this one. Hope this info helps someone.

Enjoy,

Flux.

[sjohn777]

Thank you for this post. I had the same issue and also found that trying to expand the templates VMDK when deploying from a template caused this issue. Caused me a few hours of grief until I ran across your post.

[Vmwared]

wow...thanks for the lead... spending hours on the phone with vmware proved useless...

youre right i used the experiemental feature and it hijacked my template image instead...how ghetto is that

[phil_riley]

I encountered this problem, but with your note, have been able to fix it in ~60 seconds.

On the affected VM, download the .vmx file to a local machine where you can use notepad, textpad, etc.

Towards the top of the file, you will notice the drive mappings and SCSI IDs used by the VM that are pointing to the wrong files. At the bottom of the file, you will notice the correct SCSI ID and path pointing to the VM's correct swap file location. Cut and paste the entire path and file name and replace the bogus path in the drive entry(ies) at the top of the file.

(To be clear, only paste the appropriate amount of filename, not replacing the entire thing, but you should be able to rebuild the proper paths with the information from the swapfile path.)

Save, upload back to the VM directory, power on. VM is now useable again without re-cloning.

-Phil

[kurtwest]

I have this same problem. Any answer from VMWare support?

[phil_riley]

I was able to fix my issue with the steps described.

I did not involve VMWare support with my fix.

[dnsadministrato]

Phil,

Thanks for the info, I would have spent hours working on this problem, with your info had it fixed in a matter of minutes.

Harold.

[eazyAdm]

Hi @all,

if I understood the first thread right, the problem is the vmdk or -flat.vmdk and not the .svwp file.

To fix vswp issues there are a lot of dirty methods available, move the file, delete the file or just chance the file in der .vmx.

But if the -flat.vmdk is locked, you will not have any chance to unlock the file with the available tools.

You can check it with touch: "touch /vmfs/volumes/VM-Store/guest_folder/guest-flat.vmdk"

if you get no output, everything is ok, but if you get a message like "invalid argument" the file ist locked.

We run in this problem after a crash of 1 ESX Server. After the Server comes up again we had about 30 locked files in the Store, we tried to restart all the ESX Server, hide the LUN and so on but nothing worked.

In our case the Problem is that the VMFS3 does not update the timestamp in dthe VMFS-Meta-data, normaly a file gets unlocked if the timestamp is older then 3 Seconds. In our case the Timestamp is older, but nothing happens.

bye

eazy

[tkutil]

I just ran into the same problem. Thanks for the post. I was troubleshooting this for a couple of days.

What other methods are there available for resizing a disk from a template since the "experimental" is a bit tricky?

[djflux]

eazyAdm,

I ran into this issue while deploying a VM from a template and checking the box on the last page of the deployment wizard that says something to the effect of edit virtual machine hardware (Experimental) and then trying to modify the VMDK for the new machine deployed from the template.

[djflux]

tkutil,

Depends on the operating system installed in the template. From the command line of one of your ESX hosts you can use vmkfstools to expand the VMDK:

vmkfstools -X50G mynewmachine.vmdk

will expand your VMDK to 50GB. That will only expand the underlying virtual disk - not the filesystem on the virtual disk.

For Windows VMs we use Windows PE as a loading mechanism and we can PXE boot the VM and use diskpart to extend the filesystem even on the system partition (i.e. the C:\ drive) of a Windows VM. One could also use BartPE I would assume.

Here's our process to extend to 50GB for a Windows-based OS:

. Deploy new VM from template

. vmkfstools -X50G newmachine.vmdk

. Boot to WinPE and get to a command prompt

. Run diskpart and run the following commands at the diskpart> prompt

. select disk 0

. select partition 1

. extend

For Linux, it would depend on the distro. On RedHat/Fedora-based distros with the root filesystem on a Linux LVM logical volume the process would be basically the same first 2 steps above then:

. Create a new physical partition with fdisk

. Change the partition type to Linux LVM (partition type 8e)

. Boot into Linux

. pvcreate on new partition

. vgextend onto new physical partition

. lvextend current logical volume

. resize2fs the current filesystem on the logical volume to the new size.

I leave other OSs as an exercise for the reader

Hope this helps.

EDIT: you have to boot to PE or some other offline OS to extend the system partition on Windows because you're not supposed to be able to extend the system partition.

[eazyAdm]

Hi,

i think resizing is a offtopic ... so i don't matter!

@DJflux:I can't see any paralells between your problem and mine.

You are trying to resize VMDK if i understand right. The initial Problem was that no access to the -flat.vmdk was possible, no dd, no cat nothing read/write lock.

For resizing: 'man vmkfstool' there you can find nearly everything possible. If not, try again here.

Sorry

bye

eazy

[NYSDHCR]

Thanks! Just wanted to let you know I stumbled upon your posting and it resolved my issue also! [http://communities.vmware.com/message/1210914|m-1210914]

:^{0 Gotta love these forums! :}0

-John-

[peterlyttle]

Great post thanks -

Heres an easy way to search all the vmx files -

from within the /vmfs/volumes - replace Template with something from the problem vmdk this will list all the vmx files that it finds Template in

find . -iname '*vmx' | xargs grep 'Template' -sl

[mierdaan]

This bug still exists in ESXi/vCenter 5.  Opened a case, #12177367605.

/vmfs/volumes/4eb18412-e8f5bca4-1d84-0050562e0c88/newVM1 # vmware -l

VMware ESXi 5.0.0 GA

/vmfs/volumes/4eb18412-e8f5bca4-1d84-0050562e0c88/newVM1 # pwd

/vmfs/volumes/Shared/newVM1

/vmfs/volumes/4eb18412-e8f5bca4-1d84-0050562e0c88/newVM1 # ls -lh

-rw------- 1 root root 2.0G May 23 12:08 newVM1-flat.vmdk

-rw------- 1 root root 8.5k May 23 12:08 newVM1.nvram

-rw------- 1 root root 514 May 23 12:09 newVM1.vmdk

-rw-r--r-- 1 root root 0 May 23 12:09 newVM1.vmsd

-rwxr-xr-x 1 root root 2.7k May 23 12:09 newVM1.vmx

-rw-r--r-- 1 root root 261 May 23 12:09 newVM1.vmxf

/vmfs/volumes/4eb18412-e8f5bca4-1d84-0050562e0c88/newVM1 # ls ../templateVM1/ -lh

-rw------- 1 root root 2.0G May 23 12:01 templateVM1-flat.vmdk

-rw------- 1 root root 8.5k May 23 12:01 templateVM1.nvram

-rw------- 1 root root 496 May 23 12:09 templateVM1.vmdk

-rw-r--r-- 1 root root 0 May 23 11:42 templateVM1.vmsd

-rwxr-xr-x 1 root root 2.8k May 23 12:01 templateVM1.vmtx

-rw-r--r-- 1 root root 266 May 23 11:42 templateVM1.vmxf

-rw-r--r-- 1 root root 121.6k May 23 12:01 vmware.log

/vmfs/volumes/4eb18412-e8f5bca4-1d84-0050562e0c88/newVM1 # cat newVM1.vmx | grep -i vmdk

scsi0:0.fileName = "/vmfs/volumes/4eb18412-e8f5bca4-1d84-0050562e0c88/templateVM1/templateVM1.vmdk"

/vmfs/volumes/4eb18412-e8f5bca4-1d84-0050562e0c88/newVM1 # cat ../templateVM1/templateVM1.vmtx | grep -i vmdk

scsi0:0.fileName = "templateVM1.vmdk"

[HartmannA]

Was spending a lot of time on it before I found this post. Thanks!

[khitrenovich]

To whom it may concern - some time ago i described a way to save both the VM and the template it was deployed from.

http://technotes.khitrenovich.com/deployment-vm-template-fails-vmdk-locked-error