11 Replies Latest reply on Apr 4, 2020 6:17 AM by continuum

    Unable to start VM

    grob115 Hot Shot

      Hello

       

      I have ran into a rather scary situation.  For some reasons my VM have become non responsive.  So I decided to restart it by issuing the "reset" command.  Unfortunately this resulted in the VM not being able to start up.  I tried rebooting the ESXi server and am still given the following message when attempting to start the VM.

      Failed to start the virtual machine.
      Module Disk power on failed. 
      Cannot open the disk '/vmfs/volumes/557dc2e2-2c11eba8-1494-00259066ac58/<VM Name>/<VM Name>.vmdk' or one of the snapshot disks it depends on. 
      22 (Invalid argument)
      

       

      The screen shot looks like the image attached named "PowerOnError.jpg".

       

      Even more puzzling is the ESXi shell isn't able to find the .vmx file.  Note I tried to list content of the directory it attempts to include the .vmx file but isn't able to.  In fact before the reboot it was included in the output of this command, in addition to showing something unable to stat file or something like that.  And if I tried to copy it it'd say Invalid Argument.  If I do something like 'strace cp <VM Name>.vmx <dest>' it'd show the error happened during the Linux system call 'open' operation with returned value EINVAL -1.

      [root@esxi6prim:/vmfs/volumes/557dc2e2-2c11eba8-1494-00259066ac58/<VM Name>] ls -ltr
      ls: ./<VM Name>.vmx: No such file or directory
      total 52833280
      -rw-r--r--    1 root     root             0 Oct  4  2015 <VM Name>.vmsd
      -rw-r--r--    1 root     root        179092 Oct  9  2015 vmware-1.log
      -rw-r--r--    1 root     root        177929 Oct 14  2015 vmware-2.log
      -rw-r--r--    1 root     root     409599604 Nov 24 07:26 vmware-3.log
      -rw-------    1 root     root           524 Nov 24 09:55 <VM Name>.vmdk
      -rw-------    1 root     root     53687091200 Feb 17 12:52 <VM Name>-flat.vmdk
      -rw-------    1 root     root          8684 Mar 28 04:42 <VM Name>.nvram
      -rw-r--r--    1 root     root        270087 Mar 28 04:42 vmware.log
      

       

      If I try to copy the directory to another location it'd say the following.  Note it says the <VM Name>-flat.vmdk can't be opened due to an Invalid argument despite it's shown as having a size of 53687091200 on the above 'ls' output? 53687091200

      [root@esxi6prim:/vmfs/volumes/557dc2e2-2c11eba8-1494-00259066ac58] cp -rp '<VM Name>' '<VM Name>.orig.copi
      ed.after.reboot'
      cp: can't stat '<VM Name>/<VM Name>.vmx': No such file or directory
      cp: can't open '<VM Name>/<VM Name>-flat.vmdk': Invalid argument
      

       

      I'm seriously out of ideas here.  Any suggestions on how to fix this?

       

      Thanks.

        • 1. Re: Unable to start VM
          a.p. Guru
          vExpertCommunity WarriorsUser Moderators

          This looks like an issue with the VMFS locks. Although there are several posts with hints on the Internet, you may try to reach continuum via Skype. He's an expert when it comes to such issues.


          André

          • 2. Re: Unable to start VM
            bewe Expert

            ok the vmx is missing in this directory - but if you are able to start the vm (but fails) there must be a vmx in another place.

            no lockfile is visible (*,lck)

            the vmsd should not be 0 bytes

            is the vm spreaded on different datastores ?

             

            edit the vm and have a look on "vm-options->general options" there you find the entry for "vm config file"

            also see the actual vmware.log for errors

            • 3. Re: Unable to start VM
              Blaze4up Novice
              vExpert

              Sometimes it could help to completly migrate the vm to another host and datastore.

               

              Or as your error suggest: or one of the snapshot disks it depends on.

              If the vm has multiple snapshots remove or consolidate them

               

              If the vms isn't responsive then and it's running you could kill it by using esxicli vm process kill

              A vm could be shown as powered off, and still be a process on the hypervisor, checking this is never a loss.

               

              Hope some of the solutions will work for you

              • 4. Re: Unable to start VM
                continuum Guru
                vExpertCommunity Warriors

                > Note it says the <VM Name>-flat.vmdk can't be opened due to an Invalid argument despite it's shown as having a size of 53687091200 on the above 'ls' output?

                Can you open the flat.vmdk with hexdump -C name-flat.vmdk | less ?

                I assume that the flat.vmdk has I/O errors ...

                • 5. Re: Unable to start VM
                  grob115 Hot Shot

                  Hi just tried but am seeing errors like this.

                  [root@esxi6prim:/vmfs/volumes/557dc2e2-2c11eba8-1494-00259066ac58/<VM name>] ls -ltr
                  ls: ./<VM name>.vmx: No such file or directory
                  total 52833280
                  -rw-r--r--    1 root     root             0 Oct  4  2015 <VM name>.vmsd
                  -rw-r--r--    1 root     root        179092 Oct  9  2015 vmware-1.log
                  -rw-r--r--    1 root     root        177929 Oct 14  2015 vmware-2.log
                  -rw-r--r--    1 root     root     409599604 Nov 24 07:26 vmware-3.log
                  -rw-------    1 root     root           524 Nov 24 09:55 <VM name>.vmdk
                  -rw-------    1 root     root     53687091200 Feb 17 12:52 <VM name>-flat.vmdk
                  -rw-------    1 root     root          8684 Mar 28 04:42 <VM name>.nvram
                  -rw-r--r--    1 root     root        270087 Mar 28 04:42 vmware.log
                  [root@esxi6prim:/vmfs/volumes/557dc2e2-2c11eba8-1494-00259066ac58/<VM name>]
                  [root@esxi6prim:/vmfs/volumes/557dc2e2-2c11eba8-1494-00259066ac58/<VM name>] hexdump -C '<VM name>-flat.vmdk' | less
                  [root@esxi6prim:/vmfs/volumes/557dc2e2-2c11eba8-1494-00259066ac58/<VM name>] hexdump -C '<VM name>-flat.vmdk'
                  hexdump: <VM name>-flat.vmdk: Invalid argument
                  [root@esxi6prim:/vmfs/volumes/557dc2e2-2c11eba8-1494-00259066ac58/<VM name>] which hexdump
                  /bin/hexdump
                  [root@esxi6prim:/vmfs/volumes/557dc2e2-2c11eba8-1494-00259066ac58/<VM name>] ls -ltr '<VM name>-flat.vmdk'
                  -rw-------    1 root     root     53687091200 Feb 17 12:52 <VM name>-flat.vmdk
                  [root@esxi6prim:/vmfs/volumes/557dc2e2-2c11eba8-1494-00259066ac58/<VM name>] cp '<VM name>-flat.vmdk' ../.
                  cp: can't open '<VM name>-flat.vmdk': Invalid argument
                  

                   

                  I'm somewhat lost atm.  I have taken a backup by copying out the directory and noticed now the backup itself is missing this <VM name>-flat.vmdk file in the backed up directory, despite it is showing a sizable 53687091200 bytes in the original directory.  I suspect during the copy it was having the same issue with "Invalid argument".

                   

                  A strace shows a copy operation failed during the system call 'open' with a returned error EINVAL which is Invalid Argument.

                  [root@esxi6prim:/vmfs/volumes/557dc2e2-2c11eba8-1494-00259066ac58/<VM name>] strace cp '<VM name>-flat.vmdk' ../.
                  execve("/bin/cp", ["cp", "<VM name>-fl"..., "../."], [/* 18 vars */]) = 0
                  [ Process PID=202994 runs in 32 bit mode. ]
                  brk(0)                                  = 0x1f000000
                  uname({sys="VMkernel", node="esxi6prim", ...}) = 0
                  access("/etc/ld.so.preload", R_OK)      = -1 ENOENT (No such file or directory)
                  open("/etc/ld.so.cache", O_RDONLY)      = -1 ENOENT (No such file or directory)
                  open("/lib/tls/libm.so.6", O_RDONLY)    = -1 ENOENT (No such file or directory)
                  stat64("/lib/tls", 0xffb11728)          = -1 ENOENT (No such file or directory)
                  open("/lib/libm.so.6", O_RDONLY)        = 3
                  read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0\3\0\1\0\0\0\2004\0\0004\0\0\0"..., 512) = 512
                  fstat64(3, {st_mode=S_IFREG|0555, st_size=169969, ...}) = 0
                  mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xffffffffffb12000
                  mmap2(NULL, 147584, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x259c000
                  mmap2(0x25bf000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x22) = 0x25bf000
                  close(3)                                = 0
                  open("/lib/libpam.so.0", O_RDONLY)      = 3
                  read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0\3\0\1\0\0\0p\27\0\0004\0\0\0"..., 512) = 512
                  fstat64(3, {st_mode=S_IFREG|0555, st_size=43568, ...}) = 0
                  mmap2(NULL, 46380, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x25c1000
                  mmap2(0x25cc000, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0xa) = 0x25cc000
                  close(3)                                = 0
                  open("/lib/libpam_misc.so.0", O_RDONLY) = 3
                  read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0\3\0\1\0\0\0\300\n\0\0004\0\0\0"..., 512) = 512
                  fstat64(3, {st_mode=S_IFREG|0555, st_size=8596, ...}) = 0
                  mmap2(NULL, 11376, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x25cd000
                  mmap2(0x25cf000, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x1) = 0x25cf000
                  close(3)                                = 0
                  open("/lib/libc.so.6", O_RDONLY)        = 3
                  read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0\3\0\1\0\0\0\300\214\1\0004\0\0\0"..., 512) = 512
                  fstat64(3, {st_mode=S_IFREG|0555, st_size=1638750, ...}) = 0
                  mmap2(NULL, 1452696, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x25d0000
                  mmap2(0x272d000, 12288, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x15c) = 0x272d000
                  mmap2(0x2730000, 10904, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x2730000
                  close(3)                                = 0
                  open("/lib/libdl.so.2", O_RDONLY)       = 3
                  read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0\3\0\1\0\0\0@\n\0\0004\0\0\0"..., 512) = 512
                  fstat64(3, {st_mode=S_IFREG|0555, st_size=13168, ...}) = 0
                  mmap2(NULL, 12408, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x2733000
                  mmap2(0x2735000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x1) = 0x2735000
                  close(3)                                = 0
                  mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xffffffffffb13000
                  mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xffffffffffb14000
                  set_thread_area(0xffb11b94)             = 0
                  mprotect(0x2735000, 4096, PROT_READ)    = 0
                  mprotect(0x272d000, 8192, PROT_READ)    = 0
                  mprotect(0x25bf000, 4096, PROT_READ)    = 0
                  mprotect(0x259a000, 4096, PROT_READ)    = 0
                  getuid32()                              = 0
                  stat64("<VM name>-flat.vmdk", {st_mode=S_IFREG|0600, st_size=53687091200, ...}) = 0
                  stat64("../.", {st_mode=S_IFDIR|S_ISVTX|0755, st_size=4620, ...}) = 0
                  brk(0)                                  = 0x1f000000
                  brk(0x1f021000)                         = 0x1f021000
                  stat64("<VM name>-flat.vmdk", {st_mode=S_IFREG|0600, st_size=53687091200, ...}) = 0
                  lstat64(".././<VM name>-flat.vmdk", 0xffb11a90) = -1 ENOENT (No such file or directory)
                  open("<VM name>-flat.vmdk", O_RDONLY|O_LARGEFILE) = -1 EINVAL (Invalid argument)
                  write(2, "cp: can't open '<VM name>"..., 75cp: can't open '<VM name>-flat.vmdk': Invalid argument
                  ) = 75
                  exit_group(1)                           = ?
                  [root@esxi6prim:/vmfs/volumes/557dc2e2-2c11eba8-1494-00259066ac58/<VM name>]
                  

                   

                  Would it be possible to do a screen share over the weekend?

                  • 6. Re: Unable to start VM
                    grob115 Hot Shot

                    I have other hosts but none of them are ESXi and the process isn't running anymore as I have rebooted the machine.

                    • 7. Re: Unable to start VM
                      grob115 Hot Shot

                      The problem is I am unable to start this VM.  And in fact the original VM shown on the ESXi thick client isn't there anymore.

                      I was searching online and came across a suggestion to remove the VM, in order to try adding it back from the datastore with "Add to Inventory".

                      The problem was either the .vmx file wasn't there or it was there but it didn't work.

                      The VM isn't spreaded on different datastores.

                      I can't do the edit VM operation and check "vm-options->general options" anymore as the VM was removed from inventory.

                      • 8. Re: Unable to start VM
                        vFouad Hot Shot
                        VMware EmployeesvExpert

                        If you are sure the VM is not registered;

                         

                        Rebuild the vmx file you should have enough data in the vmware.log files in that folder to save yourself:

                        VMware Knowledge Base

                         

                        Try register the VM manually via the CLI: VMware Knowledge Base

                        vim-cmd solo/registervm /vmfs/volumes/datastore_name/VM_directory/VM_name.vmx

                         

                        Then try power if on from either the gui or cli:

                        VMware Knowledge Base

                        vim-cmd vmsvc/getallvms |grep <vm name>

                        vim-cmd vmsvc/power.on <vmid>

                        • 9. Re: Unable to start VM
                          grob115 Hot Shot

                          Thanks.  I just did the first step but a few observations and am stuck.

                          • The VMware article isn't viewable even are logging in for some reasons.  I searched on Google and loaded the cached version but it has a pop up blocking the view.  I then searched around a bit more and found the following which seems to have the same instructions.

                          http://www.aionsolution.com/it-support/vmware/recover-vmx-from-log-file/

                          • I was successfully able to create the new .sh file but there appears to be a duplicate of the content so I removed one of the copies.
                          • I do not understand why this needs to be a shell script because it has the same type of content as other working VMs' .vmx files with the exception of having the following extra line at the top.
                          #!/usr/bin/vmware
                          
                          • When I ran the script I was given the following error.  Assuming the VM is named as "Corp PROD - Atlassian PRIM" the script is named as "Corp PROD - Atlassian PRIM.sh"
                          [root@esxi6prim]./Corp PROD - Atlassian PRIM.sh
                          -sh: ./Corp: not found
                          [root@esxi6prim]./Corp\ PROD\ -\ Atlassian\ PRIM.sh
                          vmware: Invalid arguments
                          vmware      -v, --version         Get version information for vmkernel
                                      -l, --level           Get release level information
                                      -h, --help            Print this menu
                          

                           

                          Questions:

                          1. Why is it a script and wouldn't the content be usable as is if I rename it as a "Corp PROD - Atlassian PRIM.vmx"?
                          2. Does VMware generally speaking have problems with VM names with spaces?
                          3. Why am I unable to do anything with the *flat.vmdk file despite being able to see it being there?  I suppose this is the actual content of the VM and is the most important file, and if it's unusable we are screwed, right?
                          [root@esxi6prim]ls -ltr 'Corp PROD - Atlassian PRIM-flat.vmdk'
                          -rw-------    1 root     root     53687091200 Feb 17 12:52 Corp PROD - Atlassian PRIM-flat.vmdk
                          [root@esxi6prim]
                          [root@esxi6prim]cp -p 'Corp PROD - Atlassian PRIM-flat.vmdk' ../.
                          cp: can't open 'Corp PROD - Atlassian PRIM-flat.vmdk': Invalid argument
                          [root@esxi6prim]
                          [root@esxi6prim]pwd
                          /vmfs/volumes/datastore1/Corp PROD - Atlassian PRIM
                          [root@esxi6prim]
                          [root@esxi6prim]cp -p 'Corp PROD - Atlassian PRIM-flat.vmdk' /vmfs/volumes/datastore1/.
                          cp: can't open 'Corp PROD - Atlassian PRIM-flat.vmdk': Invalid argument
                          [root@esxi6prim]
                          

                           

                          If I tried to create the vmx file something strange happens as if the vmx file is still there and can't be overwritten despite it isn't shown by 'ls'.

                          [root@esxi6prim]cp 'Corp PROD - Atlassian PRIM.sh' 'Corp PROD - Atlassian PRIM.vmx'
                          cp: can't create 'Corp PROD - Atlassian PRIM.vmx': File exists
                          [root@esxi6prim]ls -lt
                          ls: ./Corp PROD - Atlassian PRIM.vmx: No such file or directory
                          rtotal 52833288
                          -rwxr-xr-x    1 root     root          6781 Apr  4 04:34 Corp PROD - Atlassian PRIM.sh
                          -rw-r--r--    1 root     root        270087 Mar 28 04:42 vmware.log
                          -rw-------    1 root     root          8684 Mar 28 04:42 Corp PROD - Atlassian PRIM.nvram
                          -rw-------    1 root     root     53687091200 Feb 17 12:52 Corp PROD - Atlassian PRIM-flat.vmdk
                          -rw-------    1 root     root           524 Nov 24 09:55 Corp PROD - Atlassian PRIM.vmdk
                          -rw-r--r--    1 root     root     409599604 Nov 24 07:26 vmware-3.log
                          -rw-r--r--    1 root     root        177929 Oct 14  2015 vmware-2.log
                          -rw-r--r--    1 root     root        179092 Oct  9  2015 vmware-1.log
                          -rw-r--r--    1 root     root             0 Oct  4  2015 Corp PROD - Atlassian PRIM.vmsd
                          

                           

                          'ls' uses the syscall 'lstat64' and is able to read the stats from the file.  Anything like 'cp' or 'less' require the syscall 'open' and that's when it'd return an error.  The actual error shown when I 'strace' it is the following:

                          open("Corp PROD - Atlassian PRIM-flat.vmdk", O_RDONLY|O_LARGEFILE) = -1 EINVAL (Invalid argument)
                          

                           

                          Looks like there are corruption on the VMFS volume.  Yikes!

                          [root@esxi6prim:/var/log] grep -i corrupt /scratch/log/vmkernel.log
                          2020-04-04T04:28:08.827Z cpu6:346397)WARNING: DLX: 921: Volume 557dc2e2-2c11eba8-1494-00259066ac58 ("datastore1") might be damaged on the disk. Corrupt lock detected at offset 82253824
                          2020-04-04T04:28:49.702Z cpu3:346432)WARNING: DLX: 921: Volume 557dc2e2-2c11eba8-1494-00259066ac58 ("datastore1") might be damaged on the disk. Corrupt lock detected at offset 82253824
                          2020-04-04T04:29:47.035Z cpu9:346463)WARNING: DLX: 921: Volume 557dc2e2-2c11eba8-1494-00259066ac58 ("datastore1") might be damaged on the disk. Corrupt lock detected at offset 82253824
                          2020-04-04T05:02:45.627Z cpu11:347827)WARNING: DLX: 921: Volume 557dc2e2-2c11eba8-1494-00259066ac58 ("datastore1") might be damaged on the disk. Corrupt lock detected at offset 82253824
                          2020-04-04T05:03:21.536Z cpu2:347830)WARNING: DLX: 921: Volume 557dc2e2-2c11eba8-1494-00259066ac58 ("datastore1") might be damaged on the disk. Corrupt lock detected at offset 82253824
                          2020-04-04T05:05:29.728Z cpu9:347938)WARNING: DLX: 921: Volume 557dc2e2-2c11eba8-1494-00259066ac58 ("datastore1") might be damaged on the disk. Corrupt lock detected at offset 82253824
                          2020-04-04T05:05:45.786Z cpu12:347966)WARNING: DLX: 921: Volume 557dc2e2-2c11eba8-1494-00259066ac58 ("datastore1") might be damaged on the disk. Corrupt lock detected at offset 82253824
                          2020-04-04T05:16:08.766Z cpu5:348326)WARNING: DLX: 921: Volume 557dc2e2-2c11eba8-1494-00259066ac58 ("datastore1") might be damaged on the disk. Corrupt lock detected at offset 82253824
                          

                           

                          [root@esxi6prim:/var/log] less vmkwarning.log
                          2020-03-23T14:38:38.731Z cpu11:5371123)WARNING: [lockAddr 82253824] Invalid lock addr 4728680755771736379
                          2020-03-23T14:38:38.731Z cpu11:5371123)WARNING: [lockAddr 4728680755771736379] Invalid state: Owner 000a6b57-00000000-0000-000000000000 mode 0 numHolders 0 gblNumHolders 1501603542
                          2020-03-23T14:38:38.731Z cpu11:5371123)WARNING: DLX: 921: Volume 557dc2e2-2c11eba8-1494-00259066ac58 ("datastore1") might be damaged on the disk. Corrupt lock detected at offset 82253824
                          2020-03-23T14:38:38.731Z cpu11:5371123)WARNING: [type 81a4 offset 4728680755771736379 v 16, hb offset 4728680757079534805
                          2020-03-23T14:38:38.737Z cpu13:5371124)WARNING: Invalid lock address 7308814929491424288
                          2020-03-23T14:38:38.737Z cpu13:5371124)WARNING: [lockAddr 7308814929491424288] Invalid lock type 0x6e692065
                          2020-03-23T14:38:38.737Z cpu13:5371124)WARNING: [lockAddr 81836032] Invalid lock addr 7308814929491424288
                          2020-03-23T14:38:38.737Z cpu13:5371124)WARNING: [lockAddr 7308814929491424288] Invalid lock numHolders 1629513843
                          2020-03-23T14:38:38.737Z cpu13:5371124)WARNING: [lockAddr 7308814929491424288] Invalid lock mode 2036425843
                          2020-03-23T14:38:38.737Z cpu13:5371124)WARNING: DLX: 921: Volume 557dc2e2-2c11eba8-1494-00259066ac58 ("datastore1") might be damaged on the disk. Corrupt lock detected at offset 81836032
                          2020-03-23T14:38:38.737Z cpu13:5371124)WARNING: [type 6e692065 offset 7308814929491424288 v 7310842101425793127, hb offset 7526207048542349161
                          2020-03-23T14:38:39.131Z cpu19:5756607)WARNING: Invalid lock address 4728680755771736379
                          2020-03-23T14:38:39.131Z cpu19:5756607)WARNING: [lockAddr 4728680755771736379] Invalid lock type 0x81a4
                          2020-03-23T14:38:39.131Z cpu19:5756607)WARNING: [lockAddr 82253824] Invalid lock addr 4728680755771736379
                          2020-03-23T14:38:39.131Z cpu19:5756607)WARNING: [lockAddr 4728680755771736379] Invalid state: Owner 000a6b57-00000000-0000-000000000000 mode 0 numHolders 0 gblNumHolders 1501603542
                          2020-03-23T14:38:39.131Z cpu19:5756607)WARNING: DLX: 921: Volume 557dc2e2-2c11eba8-1494-00259066ac58 ("datastore1") might be damaged on the disk. Corrupt lock detected at offset 82253824
                          2020-03-23T14:38:39.131Z cpu19:5756607)WARNING: [type 81a4 offset 4728680755771736379 v 16, hb offset 4728680757079534805
                          2020-03-23T14:38:39.137Z cpu10:5756609)WARNING: Invalid lock address 7308814929491424288
                          2020-03-23T14:38:39.137Z cpu10:5756609)WARNING: [lockAddr 7308814929491424288] Invalid lock type 0x6e692065
                          2020-03-23T14:38:39.137Z cpu10:5756609)WARNING: [lockAddr 81836032] Invalid lock addr 7308814929491424288
                          2020-03-23T14:38:39.137Z cpu10:5756609)WARNING: [lockAddr 7308814929491424288] Invalid lock numHolders 1629513843
                          2020-03-23T14:38:39.137Z cpu10:5756609)WARNING: [lockAddr 7308814929491424288] Invalid lock mode 2036425843
                          2020-03-23T14:38:39.137Z cpu10:5756609)WARNING: DLX: 921: Volume 557dc2e2-2c11eba8-1494-00259066ac58 ("datastore1") might be damaged on the disk. Corrupt lock detected at offset 81836032
                          2020-03-23T14:38:39.137Z cpu10:5756609)WARNING: [type 6e692065 offset 7308814929491424288 v 7310842101425793127, hb offset 7526207048542349161
                          2020-03-23T14:38:39.531Z cpu5:5756602)WARNING: Invalid lock address 4728680755771736379
                          2020-03-23T14:38:39.531Z cpu5:5756602)WARNING: [lockAddr 4728680755771736379] Invalid lock type 0x81a4
                          2020-03-23T14:38:39.531Z cpu5:5756602)WARNING: [lockAddr 82253824] Invalid lock addr 4728680755771736379
                          2020-03-23T14:38:39.531Z cpu5:5756602)WARNING: [lockAddr 4728680755771736379] Invalid state: Owner 000a6b57-00000000-0000-000000000000 mode 0 numHolders 0 gblNumHolders 1501603542
                          2020-03-23T14:38:39.531Z cpu5:5756602)WARNING: DLX: 921: Volume 557dc2e2-2c11eba8-1494-00259066ac58 ("datastore1") might be damaged on the disk. Corrupt lock detected at offset 82253824
                          2020-03-23T14:38:39.531Z cpu5:5756602)WARNING: [type 81a4 offset 4728680755771736379 v 16, hb offset 4728680757079534805
                          2020-03-23T14:38:39.537Z cpu23:4729105)WARNING: Invalid lock address 7308814929491424288
                          2020-03-23T14:38:39.537Z cpu23:4729105)WARNING: [lockAddr 7308814929491424288] Invalid lock type 0x6e692065
                          2020-03-23T14:38:39.537Z cpu23:4729105)WARNING: [lockAddr 81836032] Invalid lock addr 7308814929491424288
                          2020-03-23T14:38:39.537Z cpu23:4729105)WARNING: [lockAddr 7308814929491424288] Invalid lock numHolders 1629513843
                          2020-03-23T14:38:39.537Z cpu23:4729105)WARNING: [lockAddr 7308814929491424288] Invalid lock mode 2036425843
                          
                          • 10. Re: Unable to start VM
                            a.p. Guru
                            vExpertUser ModeratorsCommunity Warriors

                            Depending on the ESXi version, Checking Metadata Consistency with VOMA may help.

                             

                            André

                            • 11. Re: Unable to start VM
                              continuum Guru
                              Community WarriorsvExpert

                              Ok ...

                              make sure lsof does not list "Corp PROD - Atlassian PRIM-flat.vmdk|
                              Try vmkfstools -p 0 "Corp PROD - Atlassian PRIM-flat.vmdk" > /tmp/flat.map

                               

                              If you get /tmp/flat,map - send the file to me.

                              If you dont get /tmp/flat.map

                               

                              we need to do more.
                              Download http://sanbarrow.com/livecds/moa64-nogui/MOA64-nogui-incl-src-111014-efi.iso

                              upload it to an unaffected datastore.

                              Create Ubuntu64 VM with 5gb ram and 2 vcpus.

                              Make sure it gets a nic that allows to talk to your ESXi.

                               

                              I expect that you have putty, winscp and teamviewer/anydesk on your admin host.

                              We will very likely need to clone via Linux .....