Disk consolidation issues

[tpoulter]

Hi,

So I am in an situation that for some reason we have a lot of deltas and the server needed disk consolidation done.

But when I go to start it it gives me the following error

  "An error occurred while consolidating disks: msg.fileio.lock."

Now I did see the following article https://kb.vmware.com/s/article/2107795  but am a bit lost on how to the steps

Also I did find this following article as well https://kb.vmware.com/s/article/1027876  that I could clone the latest delta file to a new vmdk

Would this clone be in essence the latest vmdk as if I did do the consolidation?

The other issue is the amount of free space, which is currently at about 500GB. and the machine consists of  2 hdd, 1 - 147GB and 1 - 500gb

I forgot to mention this is on ESXI 5.5 as well

Thanks

[aleex42]

Could you migrate the VM to another hypervisor? That fixes the locked error sometimes.

Also creating a snapshot and then consolidate fixes it. Sometimes ...

[SupreetK]

To start of with, check if any of the VM disks are mounted on the backup proxy VM. If yes, unmount them unless there is an active backup running for the VM.

If this does not help, you need to navigate to the VM folder from the host on which the VM is currently running (folders if the VM disks spans across multiple folders/datastores) and run <touch *>. For the flat vmdk's that report as busy, run <vmkfstools -D "flat-file_name"> to determine the lock owner. Then, follow the KB VMware Knowledge Base to identify the MAC address of the host holding the lock and reboot the host in case it is a stale lock.

Cheers,

Supreet

[concord0007]

You might want to investigate if there are any lock on any of the vm's file. 

Refer KB : VMware Knowledge Base

Regarding your query for VMware Knowledge Base your understanding is correct . However depending on the size of all deltas vmdk please make sure that you have enough space on the destination datastore where you are cloning the vmdk to.

Clone the virtual hard disk from its current snapshot delta point using the vmkfstools -i command.

# vmkfstools -i /vmfs/volumes/Storage1/examplevm/examplevm-000003.vmdk /vmfs/volumes/Storage2/examplevm_clone.vmdk

You see output similar to:

Destination disk format: VMFS thick
Cloning disk '/vmfs/volumes/Storage1 (3)/examplevm/examplevm-000003.vmdk'...
Clone: 100% done.

[tpoulter]

@aleex42​ - Cannot do that unfortunatley, as only single host, cannot migrate. But will try to create a new snapshot and remove

@SupreetK --- We use HPE VMX Explorer for vm backups but I do not see that anything is mounted there, but will double check

I have looked at that article you linked, which I did in my initial post as well, but I am at a loss on what I am looking for in the logs, it says to look at last consolidate/power-on process, but not sure how that is worded in the logs. But have had rebooted the host prior to trying to consolidate

@concord0007  -- ok good to know. I may need to do that ultimately if i cannot find why I cannot do it normally

[SupreetK]

Follow the steps from my comment and refer to that article just to know how to identify the MAC address of the locking host.

Cheers,

Supreet

[tpoulter]

Supreek -- Again, I do not understand what I am looking for in the logs files.    I have attached the log file as well.

As part of the article says to do this  vmkfstools -D vm-disk-flat.vmdk and here is what i get 

gen 153, mode 0, owner 00000000-00000000-0000-000000000000 mtime 43352

[SupreetK]

For file lock issues, looking into the logs is of no use. Can you run the below command and paste the output?

grep -i vmdk /vmfs/volumes/datastore/VM-Folder/VM-name.vmx

Cheers,

Supreet

[tpoulter]

Here is the output.

scsi0:0.fileName = "servername-000025.vmdk"

scsi0:1.fileName = "servername_1-000023.vmdk"

[SupreetK]

Cool. From the host on which the VM is registered, navigate to the VM folder and run <touch *>. Prior to doing this, power off the VM. Once powered off, run <touch *>. If the flat disks are shown as 'Device or resource busy', run <vmkfstools -D "flat-file_name"> and share the output.

Cheers,

Supreet

[saurabhheda]

Hi tpoulter​

-> What you can do is, go inside the VM folder from command line and run below command:

for a in $(ls -1);do echo $a; vmkfstools -D $a; done

Above command will give you lock details for all the files. Check for the VMDK files and it would show the MAC address.

Pick up that MAC address and you can find the host. Once you are able to identify the host, just restart the management agents on the host.

services.sh restart

" Please consider marking this answer "correct" or "helpful" if you think your question have been answered correctly."

Thanks,

Saurabh

[tpoulter]

I will try this. I will have to wait till i can scheduled to shutdown the vm. it will be in the evening

[tpoulter]

I wanted to give an update, as I personally hate seeing posts not completed

I was able to fix this by making a clone of the HDD1(out of 2) for the main server, remove the old vmdk and snapshots associated with that disk.

Then it was able to consolidate the 2nd disk as normal.

I would have liked to continue the troubleshooting, but I was getting low on space on the datastore and was worried that if I did not have enough room to clone I would be really stuck

Want to say thanks for the help the others have provided