4 Replies Latest reply on Oct 10, 2018 4:01 AM by ConorOH

    Disk Consolidation File Lock (vSphere 5.5)

    ConorOH Lurker

      Hello,

       

      I'm currently facing an issue in which two VMs require disk consolidation after being backed up by Veeam but a file lock remains on the files. I get the message "An error occurred while consolidating disks: msg.fileio.lock." when I try to consolidate. When looking at the host servers ESXI logs I can see this error:

       

      AIOGNRC: Failed to open '/vmfs/volumes/*Datastore*/*VM*/*VM*-flat.vmdk' : Failed to lock the file (40003) (0x2013).

      AIOMgr_OpenWithRetry: Descriptor file '/vmfs/volumes/*Datastore*/*VM*/*VM*-flat.vmdk' locked (try 2)

       

      From what I've seen elsewhere a common solution to this is to vMotion the VM in question to a new datastore. Unfortunately this is going to be complicated for me because we don't have the full license to allow a powered on migration, and one of the affected VMs is the VCenter Appliance itself. So if I power down the Appliance to try and perform the migration I will lose access to the vSphere console and the individual host won't be able to perform the move on its own.

       

      So my questions are:

      Is there an alternative to the Storage vMotion to unlock the files?

      Is there a better way of doing the migration? I have seen that Veeam's Quick Migration might work but again I don't know how that will interact with the vCenter Appliance being the one moved.

       

      Any advice would be appreciated.

        • 1. Re: Disk Consolidation File Lock (vSphere 5.5)
          a.p. Guru
          Community WarriorsvExpertUser Moderators

          What sometimes happens is that a backup doesn't properly complete, and a backup server/proxy still has the VM's virtual disk attached, thus locking the VM's .vmdk file.

          Please check whether this is the case, and - if the backup job appears to be completed - manually detach the hot-added .vmdk from the backup proxy.

           

          André

          • 2. Re: Disk Consolidation File Lock (vSphere 5.5)
            ConorOH Lurker

            Hi Andre, thanks for your quick reply.

             

            I had a look and there was a hot added disk attached for some other VM that doesn't exist anymore, but the VMs I'm having issues with were not attached there. I removed that old one anyway. Any other ideas?

             

            Conor

            • 3. Re: Disk Consolidation File Lock (vSphere 5.5)
              a.p. Guru
              vExpertUser ModeratorsCommunity Warriors

              If the virtual disk isn't mounted anymore (except for the original VM), it may be worth rebooting the Veeam Backup server.

              If this doesn't help, run

               

              vmkfstools -D /vmfs/volumes/datastreo/folder/vmname-flat.vmdk

               

              to see whether the output shows a MAC address for a lock owner.

               

              André

              vmkfstools -D

              • 4. Re: Disk Consolidation File Lock (vSphere 5.5)
                ConorOH Lurker

                Thank you Andre! That command pointed me to the solution. Resolved now, I'll detail my steps below for others.

                 

                I had already tried powering off Veeam to try release the lock which did nothing, so I ran the command suggested on the ESX host of the VM:

                vmkfstools -D /vmfs/volumes/datastreo/folder/vmname-flat.vmdk

                This showed a read-only lock was in place on the file from the MAC address of the ESX host, here's some of the result it showed:

                RO Owner[0] HB Offset XXX XXX-XXX-XXX-MAC Address

                 

                I used this command to find which process was holding the lock:

                lsof |grep "./vCenter Appliance-flat.vmdk"

                It showed me two processes:

                21930594    vmx                  FILE                      102  /vmfs/volumes/.../vCenter Appliance-flat.vmdk

                9807336    vpxa-worker          FILE                      39  /vmfs/volumes/.../vCenter Appliance-flat.vmdk

                 

                vpxa is the process that deals with snapshots and other managements tasks, so I made sure no snapshots, vMotions etc were running and then restart the process with this command:

                /etc/init.d/vpxa restart

                 

                I reran lsof command to check for locks, and now only vmx was holding the file. Consolidation then worked.

                 

                Hope that helps others