File lock owned by disconnected (down) vmnic

[dra60n]

Hi,

While troubleshooting snapshot consolidation issues (VM powered off, unable to start, consolidation fails due to a file lock) I discovered that MAC address of the vmnic that owns the lock belongs to the adapted that is disconnected (not assigned to any vswitches, port groups, etc.) How can I remove this lock?

The host runs esxi 6.0 and is connected to the vCenter 6.0

Thanks

[continuum]

Please show the output of

vmkfstools -D file-in-question

Please also read Please read and help .... Bug or feature in VMFS 6

and check if you have one of the new locktypes

[dra60n]

Thank for your response.

The output of vmkfstools -D command:

Lock [type 10c00001 offset 258791424 v 126720, hb offset 3637248

gen 21, mode 1, owner 5bd21c39-c4e2347e-7de9-b496910a6344 mtime 47682325

num 0 gblnum 0 gblgen 0 gblbrk 0]

Addr <4, 354, 43>, gen 126713, links 1, type reg, flags 0, uid 0, gid 0, mode 600

len 16986112, nb 17 tbz 0, cow 0, newSinceEpoch 17, zla 1, bs 1048576

[continuum]

Which file is locked ? - if you are lucky it is one of the non essential files like vmx,vmsd, vmdk-descriptor ...

Did you check if have one of the new lock types ?

[dra60n]

The "delta.vmdk" file is locked

I don't think I have any of the new lock types but I'm not sure if I'm interpreting the output of the command correctly.

od -t x4 .vh.sf | grep -c "abcdef0"

2048

od -t x4 .vh.sf | grep -c "abcdef01"

2041

od -t x4 .vh.sf | grep -c "abcdef02"

7

od -t x4 .vh.sf | grep -c "abcdef03"

0

od -t x4 .vh.sf | grep -c "abcdef04"

0

[continuum]

I assume you already tried the suggestions in the Knowledgebase.

Is the delta-file so large that you cant simply discard it ? - yes I know - lame suggestion but the other ones I have to offer are not trivial ...

I often can work around locks by mounting the VMFS readonly from Linux - that sometimes helps.

Another more radical approach is the procedure I described here:

Locked files with VMFS 6 – updated for ESXi 6.5 and higher | VM-Sickbay

But that version should only be attempted if you have a full backup of the datastore - so this is no solution for all cases.

If you want - send a dump of the vmfs-metadata - see

Create a VMFS-Header-dump using an ESXi-Host in production | VM-Sickbay​

I would need some extra info like inode number of the affected delta so call me via skype if you want to try that.

The good news is that you do not have one of the new mistery locks.
And if you have VMware support - please file a complaint about the missing documentation on this topic.

[dra60n]

Yes, I tried the suggestions in the Knowledgebase.

What do you mean "discard it"?

The weird thing is that there are 87 "delta" files and the same number of "ctk" files in the VM folder on the datastore.

[continuum]

Not weird at all - this is a normal part of the learning curve.

When a  vSphere environments starts to use automatic backups the admins usually assume that this reduces the daily work and increases the reliabilty of the complete setup.

Big mistake: once you start to use automatic backups your daily routine has to be adjusted. You MUST monitor the backup tool daily and fix problems asap.

If you neglect that check you will sooner or later run into a scenario like this: snapshots pile up, fill the datastore until the situation ends ugly and requires some serious work ....

If you are lucky you can get away with discarding the very last snapshot and then consolidate the other 86 snapshots in one or two steps with vmkfstools.

Dont hope for an easy solution to release the lock.

The required documentation is not available and the workarounds that I can offer are risky.

So first step is to provide a list of all vmdk-files including date and size and full names.

You need to do that via WinSCP or putty - no chance to do that with the WebUI only.

Then run vmkfstools -D against all vmdks to find out if the complete chain is locked or if only the last snapshots are affected.

For troubleshooting the following files are required:

vmx

vmsd

all vmware.logs

all vmdk-files that do NOT have delta, sesparse. ctk or flat in the name.

If you want help with releasing the locks ... I am on strike at the moment to protest against the missing documentation so at the moment only support via skype is available.

Ulli