VMware Cloud Community
ConorOH
Contributor
Contributor
Jump to solution

Disk Consolidation File Lock (vSphere 5.5)

Hello,

I'm currently facing an issue in which two VMs require disk consolidation after being backed up by Veeam but a file lock remains on the files. I get the message "An error occurred while consolidating disks: msg.fileio.lock." when I try to consolidate. When looking at the host servers ESXI logs I can see this error:

AIOGNRC: Failed to open '/vmfs/volumes/*Datastore*/*VM*/*VM*-flat.vmdk' : Failed to lock the file (40003) (0x2013).

AIOMgr_OpenWithRetry: Descriptor file '/vmfs/volumes/*Datastore*/*VM*/*VM*-flat.vmdk' locked (try 2)

From what I've seen elsewhere a common solution to this is to vMotion the VM in question to a new datastore. Unfortunately this is going to be complicated for me because we don't have the full license to allow a powered on migration, and one of the affected VMs is the VCenter Appliance itself. So if I power down the Appliance to try and perform the migration I will lose access to the vSphere console and the individual host won't be able to perform the move on its own.

So my questions are:

Is there an alternative to the Storage vMotion to unlock the files?

Is there a better way of doing the migration? I have seen that Veeam's Quick Migration might work but again I don't know how that will interact with the vCenter Appliance being the one moved.

Any advice would be appreciated.

1 Solution

Accepted Solutions
a_p_
Leadership
Leadership
Jump to solution

If the virtual disk isn't mounted anymore (except for the original VM), it may be worth rebooting the Veeam Backup server.

If this doesn't help, run

vmkfstools -D /vmfs/volumes/datastreo/folder/vmname-flat.vmdk

to see whether the output shows a MAC address for a lock owner.

André

vmkfstools -D

View solution in original post

Reply
0 Kudos
4 Replies
a_p_
Leadership
Leadership
Jump to solution

What sometimes happens is that a backup doesn't properly complete, and a backup server/proxy still has the VM's virtual disk attached, thus locking the VM's .vmdk file.

Please check whether this is the case, and - if the backup job appears to be completed - manually detach the hot-added .vmdk from the backup proxy.

André

Reply
0 Kudos
ConorOH
Contributor
Contributor
Jump to solution

Hi Andre, thanks for your quick reply.

I had a look and there was a hot added disk attached for some other VM that doesn't exist anymore, but the VMs I'm having issues with were not attached there. I removed that old one anyway. Any other ideas?

Conor

Reply
0 Kudos
a_p_
Leadership
Leadership
Jump to solution

If the virtual disk isn't mounted anymore (except for the original VM), it may be worth rebooting the Veeam Backup server.

If this doesn't help, run

vmkfstools -D /vmfs/volumes/datastreo/folder/vmname-flat.vmdk

to see whether the output shows a MAC address for a lock owner.

André

vmkfstools -D

Reply
0 Kudos
ConorOH
Contributor
Contributor
Jump to solution

Thank you Andre! That command pointed me to the solution. Resolved now, I'll detail my steps below for others.

I had already tried powering off Veeam to try release the lock which did nothing, so I ran the command suggested on the ESX host of the VM:

vmkfstools -D /vmfs/volumes/datastreo/folder/vmname-flat.vmdk

This showed a read-only lock was in place on the file from the MAC address of the ESX host, here's some of the result it showed:

RO Owner[0] HB Offset XXX XXX-XXX-XXX-MAC Address

I used this command to find which process was holding the lock:

lsof |grep "./vCenter Appliance-flat.vmdk"

It showed me two processes:

21930594    vmx                  FILE                      102  /vmfs/volumes/.../vCenter Appliance-flat.vmdk

9807336    vpxa-worker          FILE                      39  /vmfs/volumes/.../vCenter Appliance-flat.vmdk

vpxa is the process that deals with snapshots and other managements tasks, so I made sure no snapshots, vMotions etc were running and then restart the process with this command:

/etc/init.d/vpxa restart

I reran lsof command to check for locks, and now only vmx was holding the file. Consolidation then worked.

Hope that helps others Smiley Happy