Just some more good arguments on top of the table. I believe whoever is taking this decissions has not thought about it twice, nonetheless he's probably smarter than me; thus he might have thought about it thrice, now the thing is that the vast majority of sysadmins will just do it once.
Just a curious thing to add: some of our customers who are trying our software on ESXi 7.0.0 report that the blocking doesn't happen everytime. In some cases they are indeed able to access the -flat.vmdk file and back it up.
Looks like vmkfstools -i is one way to work around the lock.
Did you find a second way to work around the lock ?
That sounds promising - please investigate those cases and keep us updated.
I am looking into releasing the locks by editing the heartbeat-section. But that will be a workaround which is not suitable for regular backup jobs.
Not yet, running vmkfstools on some .vmdk file to then send a SIGSTOP to the process seems too awkward to be taken into consideration, at least to me (update: tried, it didn't work).
I will try to find out why those files are blocked in a seemingly random way. I will share all that I can find here, nonetheless you seem to have a deeper knowledge of ESXi than I do.
> I am looking into releasing the locks by editing the heartbeat-section. But that will be a workaround which is not suitable for regular backup jobs.
No hope on that road : no chance to manipulate the locks while the VM is running. Tried it but both the device and the .vh.sf are readonly for commands like dd.
What about releasing the locks before switching the VM on?. If this is to be done in the hearbeat.section, I believe it won't be valid for clustered scenarios, but could very well serve spare ESXi hosts systems.
Also it's worth noting that the physical disks can be read one byte at a time, which is useless for backups, but may be useful for other situations.
I was able to get some .vmx files from the VMs that are being backed up as well as the ones that are not.
I suspect that some variable in the .vmx file can be the reason why some -flat.vmdk files are not being locked.
I will perform a comparative study on this.
Forget my last comment, it was quite innocent. I revisited the .vh.sf documentation. Was thinking more on some eventual variable to be set in the .vmx file that would prevent the -flat.vmdk files from being locked, as the observed behaviour from part of my client would leave a door open to that.
I spent some hours playing around with the .vmx files provided by my client, no joy so far. Maybe the VMs that allow access to these files are just behaving in a buggy way themselves. In any case, being this lock something done on purpose, any still open gap would soon be closed as well.
to read a flat.vmdk when dd if=name-flat.vmdk bs=1M ... fails
1. use a helper VM and create a protective snapshot added to the original basedisk.
2. dont access the vmdk-file - acess the device
dd if=name-flat.vmdk bs=1 count=4096 skip=0 | hexdump -C #fails
dd if=/dev/disks/mpx.vmhba0:C0:T4:L0:2 bs=1 count=4096 skip=86991962112| hexdump -C #works
Thank you for your kind answer Continuum.
Yes, I know that. There is in fact an easier way to accomplish the same. Just get a list of the virtual disk extents by running:
vmkfstools -t0 name.vmdk
That will return a list of the extents compounding the Virtual Disk mapped to the offsets in the device.
That method has nonetheless a huge caveat. The device is tuned to only allow reading a byte at a time. That makes reading sizes above some MBs something unfeasible, unless you don't mind waiting some hours to read some big chunk of data.
> The device is tuned to only allow reading a byte at a time.
Sorry - I cant follow you here.
When I use this approach to extract vmdks I typically use dd with a blocksize of 1 MB
As per the tests I have performed so far, any attempt to read the device with a block size larger than 1 byte on ESXi 7.0.0 always returned zero.
I will perform some more thorough tests and post my results, there's always the possibility that I was doing something wrong or I misinterpreted the output.
I get the offsets of the fragments with vmkfstools -p 0 flat.vmdk
Then I divide the output values by 1048576 to get the values for dd in MBs
You are right!. This is definitely the way to go.
Ironically this is the first approach I took, nonetheless I think that I forgot that ESXi dd was quite limited and most probably was passing some unit abbreviation instead of the block size in bare units. Switching from Linux to ESXi all the time can make you fall in stupid errors.
Well, I guess this is the proof that there isn't really an intention to block access to data from the shell and that my first approach to the matter was a bit paranoid in the end.
Thank you so much!, I was blinded by my own prejudice, I would have wasted a lot more time until I would have eventually returned to read the disk in a right way.
Nonetheless, as the song says, it's never Wasted Time, you always learn something along the way.