Hello,
The server that host my vCenterServer VM had a disk failure. I just replace the disk and the RAID is not working but the vCenterServer is not working correctly. I power the VM OFF and now it won't boot back up again. When I try I got this error:
Failed - File system specific implementation of OpenFile[file] failed
Errors
File system specific implementation of OpenFile[file] failed
22 (Invalid argument)
Cannot open the disk '/vmfs/volumes/626ff6f1-37eb8410-4c30-b8ca3a6c2bd4/vCenterServer/vCenterServer_9.vmdk' or one of the snapshot disks it depends on.
Module 'Disk' power on failed.
Failed to start the virtual machine.If I SSH into the esxi server I got this:
[root@esxi01:~] vmkfstools -D /vmfs/volumes/HDD-DATA/vCenterServer/vCenterServer_9-flat.vmdk
Lock [type 731bb3b offset 16445328077180802224 v 9094622793657235544, hb offset 14160003458838717268
gen 10961307685017198381, mode 3995997458, owner 96813484-854dce41-a1ea-1ade6f2cba41 mtime 7231553314744408356
num 2848464591 gblnum 439939166 gblgen 11403974778648394952 gblbrk 559427326]
Corrupt lock detected, invalid number of lock holders 2848464591 should be less than 8.Addr <4, 17, 132>, gen 295, links 1, type reg, flags 0x9, uid 0, gid 0, mode 600
len 10737418240, nb 282 tbz 0, cow 0, newSinceEpoch 282, zla 3, bs 1048576
affinityFD <4,17,132>, parentFD <4,4,0>, tbzGranularityShift 20, numLFB 0
lastSFBClusterNum 75, numPreAllocBlocks 0, numPointerBlocks 2
[root@esxi01:~]And If I check the logs I see this:
2022-12-13T17:27:53.609Z cpu30:2100615 opID=c545f2f)Vol3: 4458: Failed to get object 28 type 3 uuid 626ff6f1-37eb8410-4c30-b8ca3a6c2bd4 FD 1400c84 gen d :Not found
2022-12-13T17:28:00.052Z cpu30:2134311)WARNING: NFS41: NFS41_SchedGetWorldStats:4262: World 3541748 not found
2022-12-13T17:28:01.483Z cpu54:2097526)WARNING: Invalid lock address 15011946880546165246
2022-12-13T17:28:01.483Z cpu54:2097526)WARNING: [lockAddr 15011946880546165246] Invalid lock type 0x3499a8ef
2022-12-13T17:28:01.483Z cpu54:2097526)WARNING: [lockAddr 269099008] Invalid lock addr 15011946880546165246
2022-12-13T17:28:01.483Z cpu54:2097526)WARNING: [lockAddr 15011946880546165246] Invalid lock numHolders 3580462633
2022-12-13T17:28:01.483Z cpu54:2097526)WARNING: [lockAddr 15011946880546165246] Invalid lock mode 1981033904
2022-12-13T17:28:01.483Z cpu54:2097526)WARNING: DLX: 1017: Volume 626ff6f1-37eb8410-4c30-b8ca3a6c2bd4 ("HDD-DATA") might be damaged on the disk. Corrupt lock detected at offset 269099008
2022-12-13T17:28:01.483Z cpu54:2097526)WARNING: [type 3499a8ef offset 15011946880546165246 v 5057281179781040768, hb offset 9671198442070634497
gen 7993202248977956459, mode 1981033904, owner 081130cc-64b398df-9364-ce9bbe194340 mtime 14970047313$
2022-12-13T17:28:01.483Z cpu54:2097526)WARNING: FS3: 608: VMFS volume HDD-DATA/626ff6f1-37eb8410-4c30-b8ca3a6c2bd4 on naa.61866da0ba22740029fd762d9476b26b:1 has been detected corrupted
2022-12-13T17:28:01.483Z cpu54:2097526)FS3: 610: While filing a PR, please report the names of all hosts that attach to this LUN, tests that were running on them,
2022-12-13T17:28:01.483Z cpu54:2097526)FS3: 634: and upload the dump by `voma -m vmfs -f dump -d /vmfs/devices/disks/naa.61866da0ba22740029fd762d9476b26b:1 -D X`
2022-12-13T17:28:01.483Z cpu54:2097526)FS3: 641: where X is the dump file name on a DIFFERENT volume
2022-12-13T17:28:01.483Z cpu54:2097526)FS3: 513: FS3DiskLock 882485487 15011946880546165246 9671198442070634497 7993202248977956459 5057281179781040768
2022-12-13T17:28:01.483Z cpu54:2097526)FS3: 515: 1981033904 081130cc-64b398df-9364-ce9bbe194340
2022-12-13T17:28:01.483Z cpu54:2097526)FS3: 520: 14970047313624674252 3580462633 1194859609 16000782118231934224 2182814576 1354256354
2022-12-13T17:28:01.483Z cpu54:2097526)FS3: 525: 4413065475134147462 5021557645988526359 18f07847-f258bb45-d093-ed42b6744efd
2022-12-13T17:28:01.483Z cpu54:2097526)FS3: 525: 4196573432284800136 1606543553481268322 68058b30-d801780a-cf7a-eeb8246f6c97
2022-12-13T17:28:01.483Z cpu54:2097526)FS3: 525: 12331148617143790938 16672102748532811161 b56ba96a-9cf3e164-3163-49cad725360b
2022-12-13T17:28:01.483Z cpu54:2097526)FS3: 525: 17941761427990807431 7767324149175313492 d47eed8b-7300f667-1eae-582e39373673
2022-12-13T17:28:01.483Z cpu54:2097526)FS3: 525: 13782362994581569345 5205836407541140779 7fcfda1a-14655686-25d9-b30b713e8d6a
2022-12-13T17:28:01.483Z cpu54:2097526)FS3: 525: 10114778518955122475 10141761631908326629 7a3b0b91-f5c476f8-2dcf-2d40c8f0c0a9
2022-12-13T17:28:01.483Z cpu54:2097526)FS3: 525: 11170729748227790854 2322537190434862569 5aeb35bc-4514abba-1205-d542622af783
2022-12-13T17:28:01.483Z cpu54:2097526)FS3: 525: 4870495643267792795 5707360151738217920 09f30e61-014ab175-7708-142d9a3bd38c
2022-12-13T17:28:01.483Z cpu54:2097526)FS3: 529: 11445177412033979209 2457481507260016176 77c55ef5-ea4a1689-392b-4de140c23575
2022-12-13T17:28:01.483Z cpu54:2097526)FS3: 533: 16809576325943744038 10 120 201 251 191 229 141 95
2022-12-13T17:28:01.483Z cpu54:2097526)FS3: 535: 9028032991467853447 3728570615808203376
2022-12-13T17:28:20.051Z cpu45:3534396)WARNING: NFS41: NFS41_SchedGetWorldStats:4262: World 3541748 not found
2022-12-13T17:28:40.050Z cpu32:2100616)WARNING: NFS41: NFS41_SchedGetWorldStats:4262: World 3541748 not found
2022-12-13T17:28:42.814Z cpu35:2097526)WARNING: Invalid lock address 15011946880546165246
2022-12-13T17:28:42.814Z cpu35:2097526)WARNING: [lockAddr 15011946880546165246] Invalid lock type 0x3499a8ef
2022-12-13T17:28:42.814Z cpu35:2097526)WARNING: [lockAddr 269099008] Invalid lock addr 15011946880546165246
2022-12-13T17:28:42.814Z cpu35:2097526)WARNING: [lockAddr 15011946880546165246] Invalid lock numHolders 3580462633
2022-12-13T17:28:42.814Z cpu35:2097526)WARNING: [lockAddr 15011946880546165246] Invalid lock mode 1981033904
2022-12-13T17:28:42.814Z cpu35:2097526)WARNING: DLX: 1017: Volume 626ff6f1-37eb8410-4c30-b8ca3a6c2bd4 ("HDD-DATA") might be damaged on the disk. Corrupt lock detected at offset 269099008
2022-12-13T17:28:42.814Z cpu35:2097526)WARNING: [type 3499a8ef offset 15011946880546165246 v 5057281179781040768, hb offset 9671198442070634497
gen 7993202248977956459, mode 1981033904, owner 081130cc-64b398df-9364-ce9bbe194340 mtime 14970047313$
2022-12-13T17:28:42.814Z cpu35:2097526)WARNING: FS3: 608: VMFS volume HDD-DATA/626ff6f1-37eb8410-4c30-b8ca3a6c2bd4 on naa.61866da0ba22740029fd762d9476b26b:1 has been detected corrupted
2022-12-13T17:28:42.814Z cpu35:2097526)FS3: 610: While filing a PR, please report the names of all hosts that attach to this LUN, tests that were running on them,
2022-12-13T17:28:42.814Z cpu35:2097526)FS3: 634: and upload the dump by `voma -m vmfs -f dump -d /vmfs/devices/disks/naa.61866da0ba22740029fd762d9476b26b:1 -D X`
2022-12-13T17:28:42.814Z cpu35:2097526)FS3: 641: where X is the dump file name on a DIFFERENT volume
2022-12-13T17:28:42.846Z cpu35:2097526)FS3: 513: FS3DiskLock 882485487 15011946880546165246 9671198442070634497 7993202248977956459 5057281179781040768
2022-12-13T17:28:42.846Z cpu35:2097526)FS3: 515: 1981033904 081130cc-64b398df-9364-ce9bbe194340
2022-12-13T17:28:42.846Z cpu35:2097526)FS3: 520: 14970047313624674252 3580462633 1194859609 16000782118231934224 2182814576 1354256354
2022-12-13T17:28:42.846Z cpu35:2097526)FS3: 525: 4413065475134147462 5021557645988526359 18f07847-f258bb45-d093-ed42b6744efd
2022-12-13T17:28:42.846Z cpu35:2097526)FS3: 525: 4196573432284800136 1606543553481268322 68058b30-d801780a-cf7a-eeb8246f6c97
2022-12-13T17:28:42.846Z cpu35:2097526)FS3: 525: 12331148617143790938 16672102748532811161 b56ba96a-9cf3e164-3163-49cad725360b
2022-12-13T17:28:42.846Z cpu35:2097526)FS3: 525: 17941761427990807431 7767324149175313492 d47eed8b-7300f667-1eae-582e39373673
2022-12-13T17:28:42.846Z cpu35:2097526)FS3: 525: 13782362994581569345 5205836407541140779 7fcfda1a-14655686-25d9-b30b713e8d6a
2022-12-13T17:28:42.846Z cpu35:2097526)FS3: 525: 10114778518955122475 10141761631908326629 7a3b0b91-f5c476f8-2dcf-2d40c8f0c0a9
2022-12-13T17:28:42.846Z cpu35:2097526)FS3: 525: 11170729748227790854 2322537190434862569 5aeb35bc-4514abba-1205-d542622af783
2022-12-13T17:28:42.846Z cpu35:2097526)FS3: 525: 4870495643267792795 5707360151738217920 09f30e61-014ab175-7708-142d9a3bd38c
2022-12-13T17:28:42.846Z cpu35:2097526)FS3: 529: 11445177412033979209 2457481507260016176 77c55ef5-ea4a1689-392b-4de140c23575
2022-12-13T17:28:42.846Z cpu35:2097526)FS3: 533: 16809576325943744038 10 120 201 251 191 229 141 95
2022-12-13T17:28:42.846Z cpu35:2097526)FS3: 535: 9028032991467853447 3728570615808203376
2022-12-13T17:29:00.051Z cpu46:2100623)WARNING: NFS41: NFS41_SchedGetWorldStats:4262: World 3541748 not foundThe problem is that I'm not sure how to fix this. If the disk is corrupt how can I fix it?
Thanks.
