13 Replies Latest reply on Sep 12, 2011 12:52 AM by iaco

    "Unable to access a file since it is locked"

    sporty Novice

      I can't access a "Win XP SP2 -flat.vmdk" file on a SAN shared LUN in our ESX 3.5 infrastructure (3 ESX servers). I created a new VM. Attached this file to this VM. I started this VM and got this error message:

       

      Unable to access a file since it is locked

       

       

       

      When I copy the locked file to another location, I have this error message:

       

       

       

      root@esx3s01 Win XP SP2# cp "Win XP SP2-flat.vmdk" /vmfs/volumes/ESX_SAN_Storage03/WinXP_SP2/

      cp: cannot open `Win XP SP2-flat.vmdk' for reading: Device or resource busy

       

       

       

      How do I know why it is locked? I ran ps ax |grep "Win XP SP2" on all our hosts. Didn't find anything. I can't afford restarting my hosts at the moment.

        • 1. Re: "Unable to access a file since it is locked"
          ctfoster Master

          I'll make the obvious point that the file looks like it's in use by another active VM.

          Was the file part of another VM ? Which VM did you search the process list for - the name of the new one or the old one ?

           

           

          Also you should use the vmdktools -i command to clone the vmdk around rather than cp - since cp can lead to frgmentation of the file and possible performance issues.

          • 2. Re: "Unable to access a file since it is locked"
            sporty Novice

            I searched for "Win XP SP2", the name of the old and the new Vms.

             

            The file was a part of a VM (a template, to be more precise). I wasn't able to deploy a VM from this template and desided to remove it from inventory hoping it will "unlock" the file.

             

             

             

            However the file is still locked and I don't know how to figure out what process is holding it. It might be anything on any ESX host.

             

             

             

            I know vmdktools is a better way of handling vmfs files but now I just want to "unlock" this file ASAP. It's our only Windows XP template.

             

             

            If I run vmkfstools, I have the following error message:

             

             

            # vmkfstools -i "Win XP SP2-flat.vmdk" /vmfs/volumes/ESX_SAN_Storage03/WinXP_SP2/Win-XP-SP2-flat.vmdk

            DiskLib_Check() failed for source disk Device or resource busy (1048585).

            • 3. Re: "Unable to access a file since it is locked"
              ctfoster Master

               

              Found this on the community board. - It'll give you some idea what host is holding the file open.

               

               

              Sometimes a file or set of files in a VMFS become locked and any attempts to edit them or delete will give a device or resource busy error, even though the vm associated with the files is not running.  If the vm is running then you would need to stop the vm to manipulate the files.  If you know that the vm is stopped then you need to find the ESX server that has the files locked and then stop the process that is locking the file(s).

               

               

               

               

              • 1. Logon to the ESX host where the VM was last known to be running.

              • 2. vmkfstools -D /vmfs/volumes/path/to/file to dump information on the file into /var/log/vmkernel

              • 3. less /var/log/vmkernel and scroll to the bottom, you will see output like below:

              • a. Nov 29 15:49:17 vm22 vmkernel: 2:00:15:18.435 cpu6:1038)FS3: 130: <START vmware-16.log>

              • b. Nov 29 15:49:17 vm22 vmkernel: 2:00:15:18.435 cpu6:1038)Lock [type 10c00001 offset 30439424 v 21, hb offset 4154368

              • c. Nov 29 15:49:17 vm22 vmkernel: gen 66493, mode 1, owner 46c60a7c-94813bcf-4273-0017a44c7727 mtime 8781867] ß Bold type added to number for emphasis.

              • d. Nov 29 15:49:17 vm22 vmkernel: 2:00:15:18.435 cpu6:1038)Addr <4, 588, 7>, gen 20, links 1, type reg, flags 0x0, uid 0, gid 0, mode 644

              • e. Nov 29 15:49:17 vm22 vmkernel: 2:00:15:18.435 cpu6:1038)len 23973, nb 1 tbz 0, zla 2, bs 65536

              • f. Nov 29 15:49:17 vm22 vmkernel: 2:00:15:18.435 cpu6:1038)FS3: 132: <END vmware-16.log>

              • 4. The owner of the lock is on line 3c, the last part is all you need, in this case 0017a44c7727

              • 5. esxcfg-info | grep -i 'system uuid' | awk -F '-' '{print $NF}' will display the system uuid of the esx server. You need to run the esxcfg-info command on each esx server in the cluster to discover the owner.

              • 6. When you find the ESX server that matches the uuid owner, logon to that ESX server and run the command: ps -elf|grep vmname where vmname is the problem vm. Example output below:

              • a. 4 S root 7570 1 0 65 -10 - 435 schedu Nov27 ? 00:00:02 /usr/lib/vmware/bin/vmkload_app /usr/lib/vmware/bin/vmware-vmx -ssched.group=host/user/pool2 -@ pipe=/tmp/vmhsdaemon-0/vmxf7fb85ef5d8b3522;vm=f7fb85ef5d8b3522 /vmfs/volumes/470e25b6-37016b37-a2b3-001b78bedd4c/iu-lsps-vstest/iu-lsps-vstest.vmx0

              • 7. Since there is a process running, pid 7570 in the example, you need to kill it by following steps 5-12 on stopping a VM above

              • 8. Once the kill is complete the files should be released.

               

              • 4. Re: "Unable to access a file since it is locked"
                sporty Novice

                 

                Thanks a lot! I have found the VM that was locking the file. The file that supposed to be a template file was used as a primary partition for one of the Windows XP VM machines.

                 

                 

                I will be having a serious talk with a guy who did this

                 

                 

                • 5. Re: "Unable to access a file since it is locked"
                  ctfoster Master

                  Go easy - he thought he was doing you a favour !   Thanks for the points.

                  • 6. Re: "Unable to access a file since it is locked"
                    john.gallucci Enthusiast

                     

                    There are several posts similar to this, but I found this solution to be the only one that worked!

                     

                     

                    I found the easiest way to find dead processes is to put the ESX host into maintenance mode first and then look around (that way you dont accidentally kill a good VM).

                     

                     

                    No reboot required for me, putting the host into maintenance mode released the locks and all was good.

                     

                     

                    Thanks for the post!

                    • 7. Re: "Unable to access a file since it is locked"
                      burdweiser Hot Shot
                      vExpert

                      What happens if the owner is all zeros?

                      • 8. Re: "Unable to access a file since it is locked"
                        wally Hot Shot

                        I have the same issue:

                         

                        May 13 04:35:36 esx24 vmkernel: gen 213, mode 0, owner 00000000-00000000-0000-000000000000 mtime 6130337

                         

                        Did you find a solution to the 'all zeros' ?

                        • 9. Re: "Unable to access a file since it is locked"
                          mih Novice

                          I know this is an old post, but anyone got a solution to the owner being all zeros ?

                          • 10. Re: "Unable to access a file since it is locked"
                            fannonland Lurker

                            I am trying to follow your steps but when I enter in vmkfstools -D /vmfs/volumes/path/to/file and trying to use .vmdk or .vmx I get

                             

                            Could not get the dump information for 'my virtual machin' (rv -1)

                            Error: Inappropriate ioctl for device

                             

                            Anyone know what that means?

                            • 11. Re: "Unable to access a file since it is locked"
                              iaco Novice

                              i had a case with a all zero owner

                              i tried running vmkfstools from other esx hosts that had access to the same luns until i found one ESX that reported a valid owner

                              • 12. Re: "Unable to access a file since it is locked"
                                nancy31 Novice

                                Been experiencing the same problem.  And I hope someone could give an up to date advice.  Thanks in advance http://imagicon.info/cat/5-59/1.gif

                                • 13. Re: "Unable to access a file since it is locked"
                                  iaco Novice

                                  as i've said in my previous post i've been able to successfully identify a 00000000-00000000-0000-000000000000 owner by running vmkfstools on another esx.

                                  full details here:

                                  http://pleasedonttouchthescreen.blogspot.com/2011/09/how-to-unlock-vmdk-file.html