VMware ESXi 7.0.0 build-15843807: -flat.vmdk files are read locked when a snapshot is taken

[DanielJGF]

We have detected that on ESXi 7.0.0 build-15843807 -flat.vmdk files are locked for reading even though the VM is running on top of a snapshot.

[root@wxpmk:~] cat /vmfs/volumes/data/CentOS-7.2/CentOS-7.2.vmsd

.encoding = "UTF-8"

snapshot.lastUID = "13"

snapshot.current = "13"

snapshot0.uid = "13"

snapshot0.filename = "CentOS-7.2-Snapshot13.vmsn"

snapshot0.displayName = "1"

snapshot0.createTimeHigh = "369657"

snapshot0.createTimeLow = "387876330"

snapshot0.numDisks = "1"

snapshot0.disk0.fileName = "CentOS-7.2.vmdk"

snapshot0.disk0.node = "scsi0:0"

snapshot.numSnapshots = "1"

[root@wxpmk:~] dd if=/vmfs/volumes/data/CentOS-7.2/CentOS-7.2-flat.vmdk ibs=1 skip=0 count=1

dd: can't open '/vmfs/volumes/data/CentOS-7.2/CentOS-7.2-flat.vmdk': Device or resource busy

Nonetheless when we query the file lock info, the file appears as: lockMode: Read-Only

vmfsfilelockinfo -p /vmfs/volumes/data/CentOS-7.2/CentOS-7.2-flat.vmdk

vmfsfilelockinfo Version 2.0

Looking for lock owners on "CentOS-7.2-flat.vmdk"

"CentOS-7.2-flat.vmdk" is locked in Read-Only mode by host having mac address ['00:0c:29:d5:ea:36']

Trying to use information from VMFS Heartbeat

Host owning the lock on file is 192.168.3.240, lockMode : Read-Only

Total time taken : 4.1909195680054836 seconds.

We don't know if this is the intended behaviour or it is a bug, as there seems to be some contradictions in what we could find out so far.

[DanielJGF]

Forget my last comment, it was quite innocent. I revisited the .vh.sf documentation. Was thinking more on some eventual variable to be set in the .vmx file that would prevent the -flat.vmdk files from being locked, as the observed behaviour from part of my client would leave a door open to that.

I spent some hours playing around with the .vmx files provided by my client, no joy so far. Maybe the VMs that allow access to these files are just behaving in a buggy way themselves. In any case, being this lock something done on purpose, any still open gap would soon be closed as well.

[continuum]

2 Workarounds:

to read a flat.vmdk when dd if=name-flat.vmdk bs=1M ... fails

1. use a helper VM and create a protective snapshot added to the original basedisk.

2. dont access the vmdk-file - acess the device

instead of

dd if=name-flat.vmdk bs=1 count=4096 skip=0  | hexdump -C #fails

run

dd if=/dev/disks/mpx.vmhba0:C0:T4:L0:2 bs=1 count=4096 skip=86991962112| hexdump -C #works

[DanielJGF]

Thank you for your kind answer Continuum.

Yes, I know that. There is in fact an easier way to accomplish the same. Just get a list of the virtual disk extents by running:

vmkfstools -t0 name.vmdk

That will return a list of the extents compounding the Virtual Disk mapped to the offsets in the device.

That method has nonetheless a huge caveat. The device is tuned to only allow reading a byte at a time. That makes reading sizes above some MBs something unfeasible, unless you don't mind waiting some hours to read some big chunk of data.

[continuum]

> The device is tuned to only allow reading a byte at a time.
Sorry - I cant follow you here.
When I use this approach to extract vmdks I typically use dd with a blocksize of 1 MB

[DanielJGF]

As per the tests I have performed so far, any attempt to read the device with a block size larger than 1 byte on ESXi 7.0.0 always returned zero.

I will perform some more thorough tests and post my results, there's always the possibility that I was doing something wrong or I misinterpreted the output.

[continuum]

I get the offsets of the fragments with vmkfstools -p 0 flat.vmdk

Then I divide the output values by 1048576 to get the values for dd in MBs

[DanielJGF]

You are right!. This is definitely the way to go.

Ironically this is the first approach I took, nonetheless I think that I forgot that ESXi dd was quite limited and most probably was passing some unit abbreviation instead of the block size in bare units. Switching from Linux to ESXi all the time can make you fall in stupid errors.

[DanielJGF]

Solved!

Well, I guess this is the proof that there isn't really an intention to block access to data from the shell and that my first approach to the matter was a bit paranoid in the end.

Thank you so much!, I was blinded by my own prejudice, I would have wasted a lot more time until I would have eventually returned to read the disk in a right way.

Nonetheless, as the song says, it's never Wasted Time, you always learn something along the way.