14 Replies Latest reply on Aug 11, 2018 6:51 AM by InSPa

    Problem with locked virtual machine after esxi host crash

    SchmidA2 Novice

      Hi all,

       

      a few days ago my ESXi host crashed because of an unknown reason.

      I had to reinstall my ESXi host (on same hardware) from scratch but I was able to restore configuration from a previous backup.

       

      But unfortunately there is one corrupt vm. Can't register / start / stop this vm with new re-installed ESXi host.

      I'm also not able to copy the harddisk *.vmdk from that vm to a different datastore.

       

      It seems that this vm still has a lock from the previous installed esxi host.

      I can't delete the <vm-name>.vmx.lck file manually because of insuffizient rights. (0600 - rw----..)

       

      With "vmkfstools -D <vm-name>.vmx" I confirmed the still active lock.

      Output:

      Lock [type 10c00001 offset 40050688 v 3927, hb offset 3244032

      gen 1489, mode 1, owner 5b4d60da-c1f01488-2a46-MACADDR mtime 2303

      num 0 gblnum 0 gblgen 0 gblbrk 0]

      Addr <4, 11, 1>, gen 2, links 1, type reg, flags 0, uid 0, gid 0, mode 100755

      len 3410, nb 0 tbz 0, cow 0, newSinceEpoch 0, zla 4305, bs 65536

       

      But this information is not really helpful because that ESXi host doesn't exists anymore.

       

      Any ideas how I could get this vm to work again? It would be ok if I could just create a new vm and choose as target harddisk the still available *.vmdk file...

      Many thanks in advance for you help.

       

      esxi-crash.png

       

      Regards,

      Schmidi

        • 1. Re: Problem with locked virtual machine after esxi host crash
          SupreetK Master

          Please try the below steps, in order -

           

          1) Right click the invalid VM entry and remove it from inventory. Check if this has released the lock.

          2) If only the vmx file is locked, you can create a new VM and attach the existing disk.

          3) If you are unable to remove the invalid entry, clear the contents of the vminventory.xml file -

          • Take a backup - <cp /etc/vmware/hostd/vmInventory.xml /etc/vmware/hostd/vmInventory.xml.backup>
          • Clear the contents - <cat /dev/null > /etc/vmware/hostd/vmInventory.xml>
          • Restart the services - </sbin/services.sh restart>
          • Register the VM to the inventory

           

          Please consider marking this answer as "correct" or "helpful" if you think your questions have been answered.

           

          Cheers,

          Supreet

          • 2. Re: Problem with locked virtual machine after esxi host crash
            SchmidA2 Novice

            Thanks for your reply. I already registered and removed the vm from inventory several times. (through UI)

            But that doesn't seem to release the lock. I also tried your suggestion to clear the vmInventory.xml but the problem still exists.

             

            The most important problem is that both files (*.vmdk - VMFS6 formatted and *.vmx) have a lock. Otherwise I would just create a new vm and attach the old *.vmdk, but this is also not working because of the existing lock of the *.vmdk file... I'm also not able to copy or move this file.

             

            Any other suggestions? I really need to get the data from this *.vmdk file.

            • 3. Re: Problem with locked virtual machine after esxi host crash
              SupreetK Master

              Are you sure that the lock is from the host that has been decommissioned already? The decommissioned host is in powered state right?

               

              Cheers,

              Supreet

              • 4. Re: Problem with locked virtual machine after esxi host crash
                SchmidA2 Novice

                yes, I'm sure. There is only one host in my environment.

                Yes, the host is in powered state. Other vm's from before the crash are running without any problems.

                 

                Just one from has this problem with locked files...

                 

                --------------------------------------------------------------------------

                 

                Is there any other way where I could copy or get read access to the *.vmdk from the locked virtual machine / files.

                I would like to copy the file to a different datastore or external drive so that I can re-create a new VM with the already existing disk...

                 

                Already tried the following:

                - using FTP program

                - using WinSCP --> Error code: 4

                - using ESXi/vSphere web interface

                - mount the datastore disk in a different linux system. --> Tried to mount the VMFS6 formated disk in Linux Ubuntu with vmfs-tools. But this is also not working for VMFS6.

                 

                All these ways failed because of the still locked *.vmx and *.vmdk files. And I am not able to get rid of that lock

                Any ideas?

                • 5. Re: Problem with locked virtual machine after esxi host crash
                  continuum Guru
                  User ModeratorsvExpertCommunity Warriors

                  > Tried to mount the VMFS6 formated disk in Linux Ubuntu with vmfs-tools. But this is also not working for VMFS6.

                  My next suggestion would be:

                  use a Linux helper VM while running ESXi as usual.From the Linux VM mount the complete datastore via sshfs in READONLY mode.
                  Mount another datastore in Writeable mode.
                  Use ddrescue to copy the flat.vmdk to the datastore that you can write to.
                  Use normal copy for the small files.
                  If that also fails try to get the mapping of the locked flat.vmdk via
                  vmkfstools -p 0 name-flat.vmdk > mapping-file.txt
                  If that works we can create a dd-script to extract the flat.vmdk in a raw mode that does not check for locks.
                  Are you still booted into Linux ?
                  If you dump the first 1536 MB of the VMFSvolume in question to a file with a command like
                  dd if=/dev/sdXY of=/tmp/schmid.1536 bs=1M count=1536
                  and provide the dump I can suggest the best next steps.
                  Feel free to call me via skype - ich spreche deutsch ...

                  • 6. Re: Problem with locked virtual machine after esxi host crash
                    SchmidA2 Novice

                    Hi continuum,

                     

                    many thanks for your reply.

                    I followed all of your steps with no success :-(

                     

                    1. Mounted datastore via sshfs (with and without readonly mode | parameter: -o ro)

                    Error message:

                    There was an error copying the file into ...

                    Error opening file: Operation not permitted

                     

                    2. Tried to get the mapping. But this was also failing.

                    Error message:

                    Could not open file...

                     

                    3. Uploaded the VMFS-Header-Dump here: <removed wrong link>

                    Many thanks again for your time. Hope you get some more details from that dump...

                    • 7. Re: Problem with locked virtual machine after esxi host crash
                      continuum Guru
                      Community WarriorsvExpertUser Moderators

                      Hi
                      just checked the dump - the file is good but I need to know which flat.vmdk you need.
                      Please specify the name of the directory and the name of the flat.vmdk.
                      If possible call me via skype - I probably have follow-up questions.

                       

                      Ulli

                      • 8. Re: Problem with locked virtual machine after esxi host crash
                        SchmidA2 Novice

                        Tried to register for a skype account but they like to know a lot of information.

                        And I need to verify that I am an adult via credit card number. But I don't like to give them that information...

                         

                        I need this *.vmdk:

                        /vmfs/volumes/5866e0f5-376e9c9e-5768-d05099c0f4ac/HO-APP-SRV/HO-APP-SRV-flat.vmdk

                         

                        Thanks in advance.

                        • 9. Re: Problem with locked virtual machine after esxi host crash
                          continuum Guru
                          Community WarriorsvExpertUser Moderators

                          Are you sure you created the dump for the correct datastore ?
                          I dont see the directory HO-APP-SRV
                          I see for example HO-VCSA.

                          Check your private messages.
                          Ulli

                          • 10. Re: Problem with locked virtual machine after esxi host crash
                            continuum Guru
                            Community WarriorsvExpertUser Moderators

                            Got  the correct dump now.
                            Can you explain the file HO-APPSRV-flat.vmdk you created or last modified yesterday.
                            There is no descriptor-vmdk for this one.
                            Is this the result of an accident ?

                            1 person found this helpful
                            • 11. Re: Problem with locked virtual machine after esxi host crash
                              SchmidA2 Novice

                              Hi continuum,

                               

                              yes, this was an result of an accident during mounting, copying, etc...

                              This file can be safely ignored.

                               

                              I'm not at home at the moment. I will try to call you in about 2 hours.

                              I already prepared a bootable usb-stick with the iso image provided via pm.

                               

                              Many thanks in advance for your time and assistant.

                               

                              Regards

                              • 12. Re: Problem with locked virtual machine after esxi host crash
                                SchmidA2 Novice

                                The datastore is back and running after two live remote sessions together with continuum.

                                 

                                It was a very strange and deep problem. But continuum was able to find a for me magic solution.

                                He solved the problem very fast. Amazing guy with a great knowledge about VMWARE products!

                                 

                                I can absolut suggest to contact him if you have any problems with restoring files from your datastore.

                                 

                                Thanks again for your assistance.

                                 

                                Regards

                                • 13. Re: Problem with locked virtual machine after esxi host crash
                                  continuum Guru
                                  User ModeratorsCommunity WarriorsvExpert

                                  For anybody else with a similar problem please also read
                                  Locked files - VMFS 6
                                  Ulli

                                  • 14. Re: Problem with locked virtual machine after esxi host crash
                                    InSPa Lurker

                                    I wanted to thank you for how tremendously useful this thread was!!

                                     

                                    A sudden crash left every vmx and vmdk on one datastore locked, and hours of searching only to find that none of the current tools seem to actually fix stale locks - just let you know the mode and who holds the lock, I came back to this thread.

                                     

                                    Pulling the first 1.5G from the datastore as you mentioned, and working with a hex editor, I was able to use that information to clear the mode and lock entity in a single find and replace. You were so tremendously helpful pointing the right direction with this thread!!