Could you migrate the VM to another hypervisor? That fixes the locked error sometimes.
Also creating a snapshot and then consolidate fixes it. Sometimes ...
To start of with, check if any of the VM disks are mounted on the backup proxy VM. If yes, unmount them unless there is an active backup running for the VM.
If this does not help, you need to navigate to the VM folder from the host on which the VM is currently running (folders if the VM disks spans across multiple folders/datastores) and run <touch *>. For the flat vmdk's that report as busy, run <vmkfstools -D "flat-file_name"> to determine the lock owner. Then, follow the KB VMware Knowledge Base to identify the MAC address of the host holding the lock and reboot the host in case it is a stale lock.
You might want to investigate if there are any lock on any of the vm's file.
Refer KB : VMware Knowledge Base
Regarding your query for VMware Knowledge Base your understanding is correct . However depending on the size of all deltas vmdk please make sure that you have enough space on the destination datastore where you are cloning the vmdk to.
Clone the virtual hard disk from its current snapshot delta point using the vmkfstools -i command.
# vmkfstools -i /vmfs/volumes/Storage1/examplevm/examplevm-000003.vmdk /vmfs/volumes/Storage2/examplevm_clone.vmdk
You see output similar to:
Destination disk format: VMFS thick
Cloning disk '/vmfs/volumes/Storage1 (3)/examplevm/examplevm-000003.vmdk'...
Clone: 100% done.
@aleex42 - Cannot do that unfortunatley, as only single host, cannot migrate. But will try to create a new snapshot and remove
@SupreetK --- We use HPE VMX Explorer for vm backups but I do not see that anything is mounted there, but will double check
I have looked at that article you linked, which I did in my initial post as well, but I am at a loss on what I am looking for in the logs, it says to look at last consolidate/power-on process, but not sure how that is worded in the logs. But have had rebooted the host prior to trying to consolidate
@concord0007 -- ok good to know. I may need to do that ultimately if i cannot find why I cannot do it normally
Follow the steps from my comment and refer to that article just to know how to identify the MAC address of the locking host.
Supreek -- Again, I do not understand what I am looking for in the logs files. I have attached the log file as well.
As part of the article says to do this vmkfstools -D vm-disk-flat.vmdk and here is what i get
gen 153, mode 0, owner 00000000-00000000-0000-000000000000 mtime 43352
vmware.log.zip 49.1 K
For file lock issues, looking into the logs is of no use. Can you run the below command and paste the output?
grep -i vmdk /vmfs/volumes/datastore/VM-Folder/VM-name.vmx
Here is the output.
scsi0:0.fileName = "servername-000025.vmdk"
scsi0:1.fileName = "servername_1-000023.vmdk"
Cool. From the host on which the VM is registered, navigate to the VM folder and run <touch *>. Prior to doing this, power off the VM. Once powered off, run <touch *>. If the flat disks are shown as 'Device or resource busy', run <vmkfstools -D "flat-file_name"> and share the output.
-> What you can do is, go inside the VM folder from command line and run below command:
for a in $(ls -1);do echo $a; vmkfstools -D $a; done
Above command will give you lock details for all the files. Check for the VMDK files and it would show the MAC address.
Pick up that MAC address and you can find the host. Once you are able to identify the host, just restart the management agents on the host.
" Please consider marking this answer "correct" or "helpful" if you think your question have been answered correctly."
I will try this. I will have to wait till i can scheduled to shutdown the vm. it will be in the evening
I wanted to give an update, as I personally hate seeing posts not completed
I was able to fix this by making a clone of the HDD1(out of 2) for the main server, remove the old vmdk and snapshots associated with that disk.
Then it was able to consolidate the 2nd disk as normal.
I would have liked to continue the troubleshooting, but I was getting low on space on the datastore and was worried that if I did not have enough room to clone I would be really stuck
Want to say thanks for the help the others have provided