VMware Cloud Community
grob115
Enthusiast
Enthusiast

Unable to start VM

Hello

I have ran into a rather scary situation.  For some reasons my VM have become non responsive.  So I decided to restart it by issuing the "reset" command.  Unfortunately this resulted in the VM not being able to start up.  I tried rebooting the ESXi server and am still given the following message when attempting to start the VM.

Failed to start the virtual machine.

Module Disk power on failed.

Cannot open the disk '/vmfs/volumes/557dc2e2-2c11eba8-1494-00259066ac58/<VM Name>/<VM Name>.vmdk' or one of the snapshot disks it depends on.

22 (Invalid argument)

The screen shot looks like the image attached named "PowerOnError.jpg".

Even more puzzling is the ESXi shell isn't able to find the .vmx file.  Note I tried to list content of the directory it attempts to include the .vmx file but isn't able to.  In fact before the reboot it was included in the output of this command, in addition to showing something unable to stat file or something like that.  And if I tried to copy it it'd say Invalid Argument.  If I do something like 'strace cp <VM Name>.vmx <dest>' it'd show the error happened during the Linux system call 'open' operation with returned value EINVAL -1.

[root@esxi6prim:/vmfs/volumes/557dc2e2-2c11eba8-1494-00259066ac58/<VM Name>] ls -ltr

ls: ./<VM Name>.vmx: No such file or directory

total 52833280

-rw-r--r--    1 root     root             0 Oct  4  2015 <VM Name>.vmsd

-rw-r--r--    1 root     root        179092 Oct  9  2015 vmware-1.log

-rw-r--r--    1 root     root        177929 Oct 14  2015 vmware-2.log

-rw-r--r--    1 root     root     409599604 Nov 24 07:26 vmware-3.log

-rw-------    1 root     root           524 Nov 24 09:55 <VM Name>.vmdk

-rw-------    1 root     root     53687091200 Feb 17 12:52 <VM Name>-flat.vmdk

-rw-------    1 root     root          8684 Mar 28 04:42 <VM Name>.nvram

-rw-r--r--    1 root     root        270087 Mar 28 04:42 vmware.log

If I try to copy the directory to another location it'd say the following.  Note it says the <VM Name>-flat.vmdk can't be opened due to an Invalid argument despite it's shown as having a size of 53687091200 on the above 'ls' output? 53687091200

[root@esxi6prim:/vmfs/volumes/557dc2e2-2c11eba8-1494-00259066ac58] cp -rp '<VM Name>' '<VM Name>.orig.copi

ed.after.reboot'

cp: can't stat '<VM Name>/<VM Name>.vmx': No such file or directory

cp: can't open '<VM Name>/<VM Name>-flat.vmdk': Invalid argument

I'm seriously out of ideas here.  Any suggestions on how to fix this?

Thanks.

11 Replies
a_p_
Leadership
Leadership

This looks like an issue with the VMFS locks. Although there are several posts with hints on the Internet, you may try to reach continuum​ via Skype. He's an expert when it comes to such issues.


André

berndweyand
Expert
Expert

ok the vmx is missing in this directory - but if you are able to start the vm (but fails) there must be a vmx in another place.

no lockfile is visible (*,lck)

the vmsd should not be 0 bytes

is the vm spreaded on different datastores ?

edit the vm and have a look on "vm-options->general options" there you find the entry for "vm config file"

also see the actual vmware.log for errors

Reply
0 Kudos
Blaze4up
Enthusiast
Enthusiast

Sometimes it could help to completly migrate the vm to another host and datastore.

Or as your error suggest: or one of the snapshot disks it depends on.

If the vm has multiple snapshots remove or consolidate them

If the vms isn't responsive then and it's running you could kill it by using esxicli vm process kill

A vm could be shown as powered off, and still be a process on the hypervisor, checking this is never a loss.

Hope some of the solutions will work for you

Reply
0 Kudos
continuum
Immortal
Immortal

> Note it says the <VM Name>-flat.vmdk can't be opened due to an Invalid argument despite it's shown as having a size of 53687091200 on the above 'ls' output?

Can you open the flat.vmdk with hexdump -C name-flat.vmdk | less ?

I assume that the flat.vmdk has I/O errors ...


________________________________________________
Do you need support with a VMFS recovery problem ? - send a message via skype "sanbarrow"
I do not support Workstation 16 at this time ...

Reply
0 Kudos
grob115
Enthusiast
Enthusiast

Hi just tried but am seeing errors like this.

[root@esxi6prim:/vmfs/volumes/557dc2e2-2c11eba8-1494-00259066ac58/<VM name>] ls -ltr

ls: ./<VM name>.vmx: No such file or directory

total 52833280

-rw-r--r--    1 root     root             0 Oct  4  2015 <VM name>.vmsd

-rw-r--r--    1 root     root        179092 Oct  9  2015 vmware-1.log

-rw-r--r--    1 root     root        177929 Oct 14  2015 vmware-2.log

-rw-r--r--    1 root     root     409599604 Nov 24 07:26 vmware-3.log

-rw-------    1 root     root           524 Nov 24 09:55 <VM name>.vmdk

-rw-------    1 root     root     53687091200 Feb 17 12:52 <VM name>-flat.vmdk

-rw-------    1 root     root          8684 Mar 28 04:42 <VM name>.nvram

-rw-r--r--    1 root     root        270087 Mar 28 04:42 vmware.log

[root@esxi6prim:/vmfs/volumes/557dc2e2-2c11eba8-1494-00259066ac58/<VM name>]

[root@esxi6prim:/vmfs/volumes/557dc2e2-2c11eba8-1494-00259066ac58/<VM name>] hexdump -C '<VM name>-flat.vmdk' | less

[root@esxi6prim:/vmfs/volumes/557dc2e2-2c11eba8-1494-00259066ac58/<VM name>] hexdump -C '<VM name>-flat.vmdk'

hexdump: <VM name>-flat.vmdk: Invalid argument

[root@esxi6prim:/vmfs/volumes/557dc2e2-2c11eba8-1494-00259066ac58/<VM name>] which hexdump

/bin/hexdump

[root@esxi6prim:/vmfs/volumes/557dc2e2-2c11eba8-1494-00259066ac58/<VM name>] ls -ltr '<VM name>-flat.vmdk'

-rw-------    1 root     root     53687091200 Feb 17 12:52 <VM name>-flat.vmdk

[root@esxi6prim:/vmfs/volumes/557dc2e2-2c11eba8-1494-00259066ac58/<VM name>] cp '<VM name>-flat.vmdk' ../.

cp: can't open '<VM name>-flat.vmdk': Invalid argument

I'm somewhat lost atm.  I have taken a backup by copying out the directory and noticed now the backup itself is missing this <VM name>-flat.vmdk file in the backed up directory, despite it is showing a sizable 53687091200 bytes in the original directory.  I suspect during the copy it was having the same issue with "Invalid argument".

A strace shows a copy operation failed during the system call 'open' with a returned error EINVAL which is Invalid Argument.

[root@esxi6prim:/vmfs/volumes/557dc2e2-2c11eba8-1494-00259066ac58/<VM name>] strace cp '<VM name>-flat.vmdk' ../.

execve("/bin/cp", ["cp", "<VM name>-fl"..., "../."], [/* 18 vars */]) = 0

[ Process PID=202994 runs in 32 bit mode. ]

brk(0)                                  = 0x1f000000

uname({sys="VMkernel", node="esxi6prim", ...}) = 0

access("/etc/ld.so.preload", R_OK)      = -1 ENOENT (No such file or directory)

open("/etc/ld.so.cache", O_RDONLY)      = -1 ENOENT (No such file or directory)

open("/lib/tls/libm.so.6", O_RDONLY)    = -1 ENOENT (No such file or directory)

stat64("/lib/tls", 0xffb11728)          = -1 ENOENT (No such file or directory)

open("/lib/libm.so.6", O_RDONLY)        = 3

read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0\3\0\1\0\0\0\2004\0\0004\0\0\0"..., 512) = 512

fstat64(3, {st_mode=S_IFREG|0555, st_size=169969, ...}) = 0

mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xffffffffffb12000

mmap2(NULL, 147584, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x259c000

mmap2(0x25bf000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x22) = 0x25bf000

close(3)                                = 0

open("/lib/libpam.so.0", O_RDONLY)      = 3

read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0\3\0\1\0\0\0p\27\0\0004\0\0\0"..., 512) = 512

fstat64(3, {st_mode=S_IFREG|0555, st_size=43568, ...}) = 0

mmap2(NULL, 46380, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x25c1000

mmap2(0x25cc000, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0xa) = 0x25cc000

close(3)                                = 0

open("/lib/libpam_misc.so.0", O_RDONLY) = 3

read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0\3\0\1\0\0\0\300\n\0\0004\0\0\0"..., 512) = 512

fstat64(3, {st_mode=S_IFREG|0555, st_size=8596, ...}) = 0

mmap2(NULL, 11376, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x25cd000

mmap2(0x25cf000, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x1) = 0x25cf000

close(3)                                = 0

open("/lib/libc.so.6", O_RDONLY)        = 3

read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0\3\0\1\0\0\0\300\214\1\0004\0\0\0"..., 512) = 512

fstat64(3, {st_mode=S_IFREG|0555, st_size=1638750, ...}) = 0

mmap2(NULL, 1452696, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x25d0000

mmap2(0x272d000, 12288, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x15c) = 0x272d000

mmap2(0x2730000, 10904, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x2730000

close(3)                                = 0

open("/lib/libdl.so.2", O_RDONLY)       = 3

read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0\3\0\1\0\0\0@\n\0\0004\0\0\0"..., 512) = 512

fstat64(3, {st_mode=S_IFREG|0555, st_size=13168, ...}) = 0

mmap2(NULL, 12408, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x2733000

mmap2(0x2735000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x1) = 0x2735000

close(3)                                = 0

mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xffffffffffb13000

mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xffffffffffb14000

set_thread_area(0xffb11b94)             = 0

mprotect(0x2735000, 4096, PROT_READ)    = 0

mprotect(0x272d000, 8192, PROT_READ)    = 0

mprotect(0x25bf000, 4096, PROT_READ)    = 0

mprotect(0x259a000, 4096, PROT_READ)    = 0

getuid32()                              = 0

stat64("<VM name>-flat.vmdk", {st_mode=S_IFREG|0600, st_size=53687091200, ...}) = 0

stat64("../.", {st_mode=S_IFDIR|S_ISVTX|0755, st_size=4620, ...}) = 0

brk(0)                                  = 0x1f000000

brk(0x1f021000)                         = 0x1f021000

stat64("<VM name>-flat.vmdk", {st_mode=S_IFREG|0600, st_size=53687091200, ...}) = 0

lstat64(".././<VM name>-flat.vmdk", 0xffb11a90) = -1 ENOENT (No such file or directory)

open("<VM name>-flat.vmdk", O_RDONLY|O_LARGEFILE) = -1 EINVAL (Invalid argument)

write(2, "cp: can't open '<VM name>"..., 75cp: can't open '<VM name>-flat.vmdk': Invalid argument

) = 75

exit_group(1)                           = ?

[root@esxi6prim:/vmfs/volumes/557dc2e2-2c11eba8-1494-00259066ac58/<VM name>]

Would it be possible to do a screen share over the weekend?

Reply
0 Kudos
grob115
Enthusiast
Enthusiast

I have other hosts but none of them are ESXi and the process isn't running anymore as I have rebooted the machine.

Reply
0 Kudos
grob115
Enthusiast
Enthusiast

The problem is I am unable to start this VM.  And in fact the original VM shown on the ESXi thick client isn't there anymore.

I was searching online and came across a suggestion to remove the VM, in order to try adding it back from the datastore with "Add to Inventory".

The problem was either the .vmx file wasn't there or it was there but it didn't work.

The VM isn't spreaded on different datastores.

I can't do the edit VM operation and check "vm-options->general options" anymore as the VM was removed from inventory.

Reply
0 Kudos
vFouad
Leadership
Leadership

If you are sure the VM is not registered;

Rebuild the vmx file you should have enough data in the vmware.log files in that folder to save yourself:

VMware Knowledge Base

Try register the VM manually via the CLI: VMware Knowledge Base

vim-cmd solo/registervm /vmfs/volumes/datastore_name/VM_directory/VM_name.vmx

Then try power if on from either the gui or cli:

VMware Knowledge Base

vim-cmd vmsvc/getallvms |grep <vm name>

vim-cmd vmsvc/power.on <vmid>

Reply
0 Kudos
grob115
Enthusiast
Enthusiast

Thanks.  I just did the first step but a few observations and am stuck.

  • The VMware article isn't viewable even are logging in for some reasons.  I searched on Google and loaded the cached version but it has a pop up blocking the view.  I then searched around a bit more and found the following which seems to have the same instructions.

http://www.aionsolution.com/it-support/vmware/recover-vmx-from-log-file/

  • I was successfully able to create the new .sh file but there appears to be a duplicate of the content so I removed one of the copies.
  • I do not understand why this needs to be a shell script because it has the same type of content as other working VMs' .vmx files with the exception of having the following extra line at the top.

#!/usr/bin/vmware

  • When I ran the script I was given the following error.  Assuming the VM is named as "Corp PROD - Atlassian PRIM" the script is named as "Corp PROD - Atlassian PRIM.sh"

[root@esxi6prim]./Corp PROD - Atlassian PRIM.sh

-sh: ./Corp: not found

[root@esxi6prim]./Corp\ PROD\ -\ Atlassian\ PRIM.sh

vmware: Invalid arguments

vmware      -v, --version         Get version information for vmkernel

            -l, --level           Get release level information

            -h, --help            Print this menu

Questions:

  1. Why is it a script and wouldn't the content be usable as is if I rename it as a "Corp PROD - Atlassian PRIM.vmx"?
  2. Does VMware generally speaking have problems with VM names with spaces?
  3. Why am I unable to do anything with the *flat.vmdk file despite being able to see it being there?  I suppose this is the actual content of the VM and is the most important file, and if it's unusable we are screwed, right?

[root@esxi6prim]ls -ltr 'Corp PROD - Atlassian PRIM-flat.vmdk'

-rw-------    1 root     root     53687091200 Feb 17 12:52 Corp PROD - Atlassian PRIM-flat.vmdk

[root@esxi6prim]

[root@esxi6prim]cp -p 'Corp PROD - Atlassian PRIM-flat.vmdk' ../.

cp: can't open 'Corp PROD - Atlassian PRIM-flat.vmdk': Invalid argument

[root@esxi6prim]

[root@esxi6prim]pwd

/vmfs/volumes/datastore1/Corp PROD - Atlassian PRIM

[root@esxi6prim]

[root@esxi6prim]cp -p 'Corp PROD - Atlassian PRIM-flat.vmdk' /vmfs/volumes/datastore1/.

cp: can't open 'Corp PROD - Atlassian PRIM-flat.vmdk': Invalid argument

[root@esxi6prim]

If I tried to create the vmx file something strange happens as if the vmx file is still there and can't be overwritten despite it isn't shown by 'ls'.

[root@esxi6prim]cp 'Corp PROD - Atlassian PRIM.sh' 'Corp PROD - Atlassian PRIM.vmx'

cp: can't create 'Corp PROD - Atlassian PRIM.vmx': File exists

[root@esxi6prim]ls -lt

ls: ./Corp PROD - Atlassian PRIM.vmx: No such file or directory

rtotal 52833288

-rwxr-xr-x    1 root     root          6781 Apr  4 04:34 Corp PROD - Atlassian PRIM.sh

-rw-r--r--    1 root     root        270087 Mar 28 04:42 vmware.log

-rw-------    1 root     root          8684 Mar 28 04:42 Corp PROD - Atlassian PRIM.nvram

-rw-------    1 root     root     53687091200 Feb 17 12:52 Corp PROD - Atlassian PRIM-flat.vmdk

-rw-------    1 root     root           524 Nov 24 09:55 Corp PROD - Atlassian PRIM.vmdk

-rw-r--r--    1 root     root     409599604 Nov 24 07:26 vmware-3.log

-rw-r--r--    1 root     root        177929 Oct 14  2015 vmware-2.log

-rw-r--r--    1 root     root        179092 Oct  9  2015 vmware-1.log

-rw-r--r--    1 root     root             0 Oct  4  2015 Corp PROD - Atlassian PRIM.vmsd

'ls' uses the syscall 'lstat64' and is able to read the stats from the file.  Anything like 'cp' or 'less' require the syscall 'open' and that's when it'd return an error.  The actual error shown when I 'strace' it is the following:

open("Corp PROD - Atlassian PRIM-flat.vmdk", O_RDONLY|O_LARGEFILE) = -1 EINVAL (Invalid argument)

Looks like there are corruption on the VMFS volume.  Yikes!

[root@esxi6prim:/var/log] grep -i corrupt /scratch/log/vmkernel.log

2020-04-04T04:28:08.827Z cpu6:346397)WARNING: DLX: 921: Volume 557dc2e2-2c11eba8-1494-00259066ac58 ("datastore1") might be damaged on the disk. Corrupt lock detected at offset 82253824

2020-04-04T04:28:49.702Z cpu3:346432)WARNING: DLX: 921: Volume 557dc2e2-2c11eba8-1494-00259066ac58 ("datastore1") might be damaged on the disk. Corrupt lock detected at offset 82253824

2020-04-04T04:29:47.035Z cpu9:346463)WARNING: DLX: 921: Volume 557dc2e2-2c11eba8-1494-00259066ac58 ("datastore1") might be damaged on the disk. Corrupt lock detected at offset 82253824

2020-04-04T05:02:45.627Z cpu11:347827)WARNING: DLX: 921: Volume 557dc2e2-2c11eba8-1494-00259066ac58 ("datastore1") might be damaged on the disk. Corrupt lock detected at offset 82253824

2020-04-04T05:03:21.536Z cpu2:347830)WARNING: DLX: 921: Volume 557dc2e2-2c11eba8-1494-00259066ac58 ("datastore1") might be damaged on the disk. Corrupt lock detected at offset 82253824

2020-04-04T05:05:29.728Z cpu9:347938)WARNING: DLX: 921: Volume 557dc2e2-2c11eba8-1494-00259066ac58 ("datastore1") might be damaged on the disk. Corrupt lock detected at offset 82253824

2020-04-04T05:05:45.786Z cpu12:347966)WARNING: DLX: 921: Volume 557dc2e2-2c11eba8-1494-00259066ac58 ("datastore1") might be damaged on the disk. Corrupt lock detected at offset 82253824

2020-04-04T05:16:08.766Z cpu5:348326)WARNING: DLX: 921: Volume 557dc2e2-2c11eba8-1494-00259066ac58 ("datastore1") might be damaged on the disk. Corrupt lock detected at offset 82253824

[root@esxi6prim:/var/log] less vmkwarning.log

2020-03-23T14:38:38.731Z cpu11:5371123)WARNING: [lockAddr 82253824] Invalid lock addr 4728680755771736379

2020-03-23T14:38:38.731Z cpu11:5371123)WARNING: [lockAddr 4728680755771736379] Invalid state: Owner 000a6b57-00000000-0000-000000000000 mode 0 numHolders 0 gblNumHolders 1501603542

2020-03-23T14:38:38.731Z cpu11:5371123)WARNING: DLX: 921: Volume 557dc2e2-2c11eba8-1494-00259066ac58 ("datastore1") might be damaged on the disk. Corrupt lock detected at offset 82253824

2020-03-23T14:38:38.731Z cpu11:5371123)WARNING: [type 81a4 offset 4728680755771736379 v 16, hb offset 4728680757079534805

2020-03-23T14:38:38.737Z cpu13:5371124)WARNING: Invalid lock address 7308814929491424288

2020-03-23T14:38:38.737Z cpu13:5371124)WARNING: [lockAddr 7308814929491424288] Invalid lock type 0x6e692065

2020-03-23T14:38:38.737Z cpu13:5371124)WARNING: [lockAddr 81836032] Invalid lock addr 7308814929491424288

2020-03-23T14:38:38.737Z cpu13:5371124)WARNING: [lockAddr 7308814929491424288] Invalid lock numHolders 1629513843

2020-03-23T14:38:38.737Z cpu13:5371124)WARNING: [lockAddr 7308814929491424288] Invalid lock mode 2036425843

2020-03-23T14:38:38.737Z cpu13:5371124)WARNING: DLX: 921: Volume 557dc2e2-2c11eba8-1494-00259066ac58 ("datastore1") might be damaged on the disk. Corrupt lock detected at offset 81836032

2020-03-23T14:38:38.737Z cpu13:5371124)WARNING: [type 6e692065 offset 7308814929491424288 v 7310842101425793127, hb offset 7526207048542349161

2020-03-23T14:38:39.131Z cpu19:5756607)WARNING: Invalid lock address 4728680755771736379

2020-03-23T14:38:39.131Z cpu19:5756607)WARNING: [lockAddr 4728680755771736379] Invalid lock type 0x81a4

2020-03-23T14:38:39.131Z cpu19:5756607)WARNING: [lockAddr 82253824] Invalid lock addr 4728680755771736379

2020-03-23T14:38:39.131Z cpu19:5756607)WARNING: [lockAddr 4728680755771736379] Invalid state: Owner 000a6b57-00000000-0000-000000000000 mode 0 numHolders 0 gblNumHolders 1501603542

2020-03-23T14:38:39.131Z cpu19:5756607)WARNING: DLX: 921: Volume 557dc2e2-2c11eba8-1494-00259066ac58 ("datastore1") might be damaged on the disk. Corrupt lock detected at offset 82253824

2020-03-23T14:38:39.131Z cpu19:5756607)WARNING: [type 81a4 offset 4728680755771736379 v 16, hb offset 4728680757079534805

2020-03-23T14:38:39.137Z cpu10:5756609)WARNING: Invalid lock address 7308814929491424288

2020-03-23T14:38:39.137Z cpu10:5756609)WARNING: [lockAddr 7308814929491424288] Invalid lock type 0x6e692065

2020-03-23T14:38:39.137Z cpu10:5756609)WARNING: [lockAddr 81836032] Invalid lock addr 7308814929491424288

2020-03-23T14:38:39.137Z cpu10:5756609)WARNING: [lockAddr 7308814929491424288] Invalid lock numHolders 1629513843

2020-03-23T14:38:39.137Z cpu10:5756609)WARNING: [lockAddr 7308814929491424288] Invalid lock mode 2036425843

2020-03-23T14:38:39.137Z cpu10:5756609)WARNING: DLX: 921: Volume 557dc2e2-2c11eba8-1494-00259066ac58 ("datastore1") might be damaged on the disk. Corrupt lock detected at offset 81836032

2020-03-23T14:38:39.137Z cpu10:5756609)WARNING: [type 6e692065 offset 7308814929491424288 v 7310842101425793127, hb offset 7526207048542349161

2020-03-23T14:38:39.531Z cpu5:5756602)WARNING: Invalid lock address 4728680755771736379

2020-03-23T14:38:39.531Z cpu5:5756602)WARNING: [lockAddr 4728680755771736379] Invalid lock type 0x81a4

2020-03-23T14:38:39.531Z cpu5:5756602)WARNING: [lockAddr 82253824] Invalid lock addr 4728680755771736379

2020-03-23T14:38:39.531Z cpu5:5756602)WARNING: [lockAddr 4728680755771736379] Invalid state: Owner 000a6b57-00000000-0000-000000000000 mode 0 numHolders 0 gblNumHolders 1501603542

2020-03-23T14:38:39.531Z cpu5:5756602)WARNING: DLX: 921: Volume 557dc2e2-2c11eba8-1494-00259066ac58 ("datastore1") might be damaged on the disk. Corrupt lock detected at offset 82253824

2020-03-23T14:38:39.531Z cpu5:5756602)WARNING: [type 81a4 offset 4728680755771736379 v 16, hb offset 4728680757079534805

2020-03-23T14:38:39.537Z cpu23:4729105)WARNING: Invalid lock address 7308814929491424288

2020-03-23T14:38:39.537Z cpu23:4729105)WARNING: [lockAddr 7308814929491424288] Invalid lock type 0x6e692065

2020-03-23T14:38:39.537Z cpu23:4729105)WARNING: [lockAddr 81836032] Invalid lock addr 7308814929491424288

2020-03-23T14:38:39.537Z cpu23:4729105)WARNING: [lockAddr 7308814929491424288] Invalid lock numHolders 1629513843

2020-03-23T14:38:39.537Z cpu23:4729105)WARNING: [lockAddr 7308814929491424288] Invalid lock mode 2036425843

Reply
0 Kudos
a_p_
Leadership
Leadership

Depending on the ESXi version, Checking Metadata Consistency with VOMA may help.

André

Reply
0 Kudos
continuum
Immortal
Immortal

Ok ...

make sure lsof does not list "Corp PROD - Atlassian PRIM-flat.vmdk|
Try vmkfstools -p 0 "Corp PROD - Atlassian PRIM-flat.vmdk" > /tmp/flat.map

If you get /tmp/flat,map - send the file to me.

If you dont get /tmp/flat.map

we need to do more.
Download http://sanbarrow.com/livecds/moa64-nogui/MOA64-nogui-incl-src-111014-efi.iso

upload it to an unaffected datastore.

Create Ubuntu64 VM with 5gb ram and 2 vcpus.

Make sure it gets a nic that allows to talk to your ESXi.

I expect that you have putty, winscp and teamviewer/anydesk on your admin host.

We will very likely need to clone via Linux .....


________________________________________________
Do you need support with a VMFS recovery problem ? - send a message via skype "sanbarrow"
I do not support Workstation 16 at this time ...

Reply
0 Kudos