12 Replies Latest reply on Jul 23, 2018 5:39 AM by tpoulter

    Disk consolidation issues

    tpoulter Novice

      Hi,

       

      So I am in an situation that for some reason we have a lot of deltas and the server needed disk consolidation done.

      But when I go to start it it gives me the following error

        "An error occurred while consolidating disks: msg.fileio.lock."

       

      Now I did see the following article https://kb.vmware.com/s/article/2107795  but am a bit lost on how to the steps

       

      Also I did find this following article as well https://kb.vmware.com/s/article/1027876  that I could clone the latest delta file to a new vmdk

      Would this clone be in essence the latest vmdk as if I did do the consolidation?

       

      The other issue is the amount of free space, which is currently at about 500GB. and the machine consists of  2 hdd, 1 - 147GB and 1 - 500gb

       

      I forgot to mention this is on ESXI 5.5 as well

       

       

      Thanks

        • 1. Re: Disk consolidation issues
          aleex42 Enthusiast
          vExpert

          Could you migrate the VM to another hypervisor? That fixes the locked error sometimes.

           

          Also creating a snapshot and then consolidate fixes it. Sometimes ...

          • 2. Re: Disk consolidation issues
            SupreetK Master

            To start of with, check if any of the VM disks are mounted on the backup proxy VM. If yes, unmount them unless there is an active backup running for the VM.

            If this does not help, you need to navigate to the VM folder from the host on which the VM is currently running (folders if the VM disks spans across multiple folders/datastores) and run <touch *>. For the flat vmdk's that report as busy, run <vmkfstools -D "flat-file_name"> to determine the lock owner. Then, follow the KB VMware Knowledge Base to identify the MAC address of the host holding the lock and reboot the host in case it is a stale lock.

             

            Cheers,

            Supreet

            • 3. Re: Disk consolidation issues
              concord0007 Lurker
              VMware Employees

              You might want to investigate if there are any lock on any of the vm's file. 

              Refer KB : VMware Knowledge Base

               

              Regarding your query for VMware Knowledge Base your understanding is correct . However depending on the size of all deltas vmdk please make sure that you have enough space on the destination datastore where you are cloning the vmdk to.

              Clone the virtual hard disk from its current snapshot delta point using the vmkfstools -i command.

               

              # vmkfstools -i /vmfs/volumes/Storage1/examplevm/examplevm-000003.vmdk /vmfs/volumes/Storage2/examplevm_clone.vmdk

               

              You see output similar to:

               

              Destination disk format: VMFS thick
              Cloning disk '/vmfs/volumes/Storage1 (3)/examplevm/examplevm-000003.vmdk'...
              Clone: 100% done.

              • 4. Re: Disk consolidation issues
                tpoulter Novice

                Cannot do that unfortunatley, as only single host, cannot migrate. But will try to create a new snapshot and remove

                 

                @SupreetK --- We use HPE VMX Explorer for vm backups but I do not see that anything is mounted there, but will double check

                 

                I have looked at that article you linked, which I did in my initial post as well, but I am at a loss on what I am looking for in the logs, it says to look at last consolidate/power-on process, but not sure how that is worded in the logs. But have had rebooted the host prior to trying to consolidate

                 

                @concord0007  -- ok good to know. I may need to do that ultimately if i cannot find why I cannot do it normally

                • 5. Re: Disk consolidation issues
                  SupreetK Master

                  Follow the steps from my comment and refer to that article just to know how to identify the MAC address of the locking host.

                   

                  Cheers,

                  Supreet

                  • 6. Re: Disk consolidation issues
                    tpoulter Novice

                    Supreek -- Again, I do not understand what I am looking for in the logs files.    I have attached the log file as well.

                     

                    As part of the article says to do this  vmkfstools -D vm-disk-flat.vmdk and here is what i get 

                    gen 153, mode 0, owner 00000000-00000000-0000-000000000000 mtime 43352

                    • 7. Re: Disk consolidation issues
                      SupreetK Master

                      For file lock issues, looking into the logs is of no use. Can you run the below command and paste the output?

                       

                      grep -i vmdk /vmfs/volumes/datastore/VM-Folder/VM-name.vmx

                       

                      Cheers,

                      Supreet

                      • 8. Re: Disk consolidation issues
                        tpoulter Novice

                        Here is the output.

                         

                         

                        scsi0:0.fileName = "servername-000025.vmdk"

                        scsi0:1.fileName = "servername_1-000023.vmdk"

                        • 9. Re: Disk consolidation issues
                          SupreetK Master

                          Cool. From the host on which the VM is registered, navigate to the VM folder and run <touch *>. Prior to doing this, power off the VM. Once powered off, run <touch *>. If the flat disks are shown as 'Device or resource busy', run <vmkfstools -D "flat-file_name"> and share the output.

                           

                          Cheers,

                          Supreet

                          • 10. Re: Disk consolidation issues
                            saurabhheda Lurker
                            vExpert

                            Hi tpoulter

                             

                            -> What you can do is, go inside the VM folder from command line and run below command:

                             

                            for a in $(ls -1);do echo $a; vmkfstools -D $a; done

                             

                            Above command will give you lock details for all the files. Check for the VMDK files and it would show the MAC address.

                             

                            Pick up that MAC address and you can find the host. Once you are able to identify the host, just restart the management agents on the host.

                             

                            services.sh restart

                             

                            " Please consider marking this answer "correct" or "helpful" if you think your question have been answered correctly."

                             

                            Thanks,

                            Saurabh

                            • 11. Re: Disk consolidation issues
                              tpoulter Novice

                              I will try this. I will have to wait till i can scheduled to shutdown the vm. it will be in the evening

                              • 12. Re: Disk consolidation issues
                                tpoulter Novice

                                I wanted to give an update, as I personally hate seeing posts not completed

                                 

                                I was able to fix this by making a clone of the HDD1(out of 2) for the main server, remove the old vmdk and snapshots associated with that disk.

                                Then it was able to consolidate the 2nd disk as normal.

                                 

                                I would have liked to continue the troubleshooting, but I was getting low on space on the datastore and was worried that if I did not have enough room to clone I would be really stuck

                                Want to say thanks for the help the others have provided