Corrupted datastore?

[Notso]

6 Drives, set up as Raid 1+0. #5 died and all the others showed up as foreign, replaced the drive and hit F on the server to import the foreign and it loaded to ESXI finally.

Rebooted and checked volume and it shows all 6 drives in the raid now.

In ESXI none of the VMs are showing and it shows the Datastores 0 but I can see the VMFS at 6TB size.

This is from the storage option in the navigator:

(Storage is blank but under the header storage it shows this Drive)

/vmfs/devices/disks/naa.6a4badb02ea41e00ff000042043581ad

1: VMFS 5.46 TB (5721599 blocks)

I try to select the datastore but it does not show in the list, I try to expand storage and it only shows the thumbdrive ESXi is loaded from.

Tried the partedutil but it doesn't exist(unknown command) and the VOMA keeps giving me some message failed to check for heartbeating hosts.

Tried the esxcli storage etc... it pauses for a second then just sends back to the command line.

Any ideas? Here I thought a raid 10 would be enough to keep from losing everything!

[Notso]

Didn't want to edit original since this is a long log...

There may be a way to save the data? I am not sure how to go about using Linux to copy off the VMs to reset the Datastore....

(If that came out correctly) but the Voma shows:

voma -m vmfs -f check -d  /dev/disks/naa.6a4badb02ea41e00ff000042043581ad:1

Checking if device is actively used by other hosts

Running VMFS Checker version 1.2 in check mode

Initializing LVM metadata, Basic Checks will be done

ON-DISK ERROR: Invalid device Size 5999532356096, should be 5999532359168

Phase 1: Checking VMFS header and resource files

   Detected VMFS file system (labeled:'Stuff') with UUID:57f2fd3e-073f7330-c93c-0026b98a0395, Version 5:61

Phase 2: Checking VMFS heartbeat region

Phase 3: Checking all file descriptors.

   Found stale lock [type 10c00001 offset 288458752 v 59, hb offset 3719168

         gen 9, mode 2, owner 00000000-00000000-0000-000000000000 mtime 327960

         num 1 gblnum 0 gblgen 0 gblbrk 0]

   Found stale lock [type 10c00001 offset 288460800 v 1931, hb offset 3719168

         gen 73, mode 1, owner 59a86d66-edde46ca-a14a-0026b98a0395 mtime 3959

         num 0 gblnum 0 gblgen 0 gblbrk 0]

   Found stale lock [type 10c00001 offset 288462848 v 1928, hb offset 3719168

         gen 73, mode 1, owner 59a86d66-edde46ca-a14a-0026b98a0395 mtime 3962

         num 0 gblnum 0 gblgen 0 gblbrk 0]

   Found stale lock [type 10c00001 offset 288466944 v 2016, hb offset 3719168

         gen 73, mode 1, owner 59a86d66-edde46ca-a14a-0026b98a0395 mtime 4011

         num 0 gblnum 0 gblgen 0 gblbrk 0]

   Found stale lock [type 10c00001 offset 288477184 v 1926, hb offset 3719168

         gen 71, mode 1, owner 596bcee0-2a26bbc5-322c-0026b98a0395 mtime 327531

         num 0 gblnum 0 gblgen 0 gblbrk 0]

   Found stale lock [type 10c00001 offset 288485376 v 1931, hb offset 3719168

         gen 73, mode 1, owner 59a86d66-edde46ca-a14a-0026b98a0395 mtime 4019

         num 0 gblnum 0 gblgen 0 gblbrk 0]

   Found stale lock [type 10c00001 offset 288528384 v 797, hb offset 3719168

         gen 73, mode 1, owner 59a86d66-edde46ca-a14a-0026b98a0395 mtime 3964

         num 0 gblnum 0 gblgen 0 gblbrk 0]

   Found stale lock [type 10c00001 offset 288530432 v 159, hb offset 3719168

         gen 73, mode 1, owner 59a86d66-edde46ca-a14a-0026b98a0395 mtime 4036

         num 0 gblnum 0 gblgen 0 gblbrk 0]

   Found stale lock [type 10c00001 offset 288544768 v 2032, hb offset 3719168

         gen 73, mode 2, owner 00000000-00000000-0000-000000000000 mtime 102019

         num 1 gblnum 0 gblgen 0 gblbrk 0]

   Found stale lock [type 10c00001 offset 288546816 v 176, hb offset 3719168

         gen 73, mode 1, owner 59a86d66-edde46ca-a14a-0026b98a0395 mtime 4050

         num 0 gblnum 0 gblgen 0 gblbrk 0]

   Found stale lock [type 10c00001 offset 288552960 v 2034, hb offset 3719168

         gen 73, mode 1, owner 59a86d66-edde46ca-a14a-0026b98a0395 mtime 101868

         num 0 gblnum 0 gblgen 0 gblbrk 0]

   Found stale lock [type 10c00001 offset 288555008 v 2035, hb offset 3719168

         gen 73, mode 1, owner 59a86d66-edde46ca-a14a-0026b98a0395 mtime 101880

         num 0 gblnum 0 gblgen 0 gblbrk 0]

   Found stale lock [type 10c00001 offset 288557056 v 2036, hb offset 3719168

         gen 73, mode 1, owner 59a86d66-edde46ca-a14a-0026b98a0395 mtime 101891

         num 0 gblnum 0 gblgen 0 gblbrk 0]

   Found stale lock [type 10c00001 offset 288563200 v 187, hb offset 3719168

         gen 73, mode 1, owner 59a86d66-edde46ca-a14a-0026b98a0395 mtime 4030

         num 0 gblnum 0 gblgen 0 gblbrk 0]

   Found stale lock [type 10c00001 offset 288569344 v 2037, hb offset 3719168

         gen 73, mode 1, owner 59a86d66-edde46ca-a14a-0026b98a0395 mtime 101915

         num 0 gblnum 0 gblgen 0 gblbrk 0]

   Found stale lock [type 10c00001 offset 288573440 v 2038, hb offset 3719168

         gen 73, mode 1, owner 59a86d66-edde46ca-a14a-0026b98a0395 mtime 101931

         num 0 gblnum 0 gblgen 0 gblbrk 0]

   Found stale lock [type 10c00001 offset 288579584 v 1837, hb offset 3719168

         gen 73, mode 1, owner 59a86d66-edde46ca-a14a-0026b98a0395 mtime 101882

         num 0 gblnum 0 gblgen 0 gblbrk 0]

   Found stale lock [type 10c00001 offset 288581632 v 242, hb offset 3719168

         gen 73, mode 1, owner 59a86d66-edde46ca-a14a-0026b98a0395 mtime 102006

         num 0 gblnum 0 gblgen 0 gblbrk 0]

   Found stale lock [type 10c00001 offset 288595968 v 1656, hb offset 3719168

         gen 63, mode 1, owner 59249d0f-fc24c9e2-1258-0026b98a0395 mtime 3296

         num 0 gblnum 0 gblgen 0 gblbrk 0]

   Found stale lock [type 10c00001 offset 288598016 v 279, hb offset 3719168

         gen 73, mode 1, owner 59a86d66-edde46ca-a14a-0026b98a0395 mtime 314243

         num 0 gblnum 0 gblgen 0 gblbrk 0]

   Found stale lock [type 10c00001 offset 288604160 v 1906, hb offset 3719168

         gen 71, mode 1, owner 596bcee0-2a26bbc5-322c-0026b98a0395 mtime 325878

         num 0 gblnum 0 gblgen 0 gblbrk 0]

   Found stale lock [type 10c00001 offset 288681984 v 221, hb offset 3719168

         gen 73, mode 1, owner 59a86d66-edde46ca-a14a-0026b98a0395 mtime 45

         num 0 gblnum 0 gblgen 0 gblbrk 0]

   Found stale lock [type 10c00001 offset 288684032 v 408, hb offset 3719168

         gen 73, mode 1, owner 59a86d66-edde46ca-a14a-0026b98a0395 mtime 389343

         num 0 gblnum 0 gblgen 0 gblbrk 0]

   Found stale lock [type 10c00001 offset 288686080 v 223, hb offset 3719168

         gen 73, mode 1, owner 59a86d66-edde46ca-a14a-0026b98a0395 mtime 51

         num 0 gblnum 0 gblgen 0 gblbrk 0]

   Found stale lock [type 10c00001 offset 288688128 v 224, hb offset 3719168

         gen 73, mode 1, owner 59a86d66-edde46ca-a14a-0026b98a0395 mtime 54

         num 0 gblnum 0 gblgen 0 gblbrk 0]

   Found stale lock [type 10c00001 offset 288690176 v 225, hb offset 3719168

         gen 73, mode 1, owner 59a86d66-edde46ca-a14a-0026b98a0395 mtime 57

         num 0 gblnum 0 gblgen 0 gblbrk 0]

   Found stale lock [type 10c00001 offset 288692224 v 226, hb offset 3719168

         gen 73, mode 1, owner 59a86d66-edde46ca-a14a-0026b98a0395 mtime 60

         num 0 gblnum 0 gblgen 0 gblbrk 0]

   Found stale lock [type 10c00001 offset 288694272 v 227, hb offset 3719168

         gen 73, mode 1, owner 59a86d66-edde46ca-a14a-0026b98a0395 mtime 63

         num 0 gblnum 0 gblgen 0 gblbrk 0]

   Found stale lock [type 10c00001 offset 288696320 v 850, hb offset 3719168

         gen 73, mode 1, owner 59a86d66-edde46ca-a14a-0026b98a0395 mtime 66

         num 0 gblnum 0 gblgen 0 gblbrk 0]

   Found stale lock [type 10c00001 offset 288698368 v 229, hb offset 3719168

         gen 73, mode 1, owner 59a86d66-edde46ca-a14a-0026b98a0395 mtime 69

         num 0 gblnum 0 gblgen 0 gblbrk 0]

   Found stale lock [type 10c00001 offset 288700416 v 230, hb offset 3719168

         gen 73, mode 1, owner 59a86d66-edde46ca-a14a-0026b98a0395 mtime 72

         num 0 gblnum 0 gblgen 0 gblbrk 0]

   Found stale lock [type 10c00001 offset 288702464 v 231, hb offset 3719168

         gen 73, mode 1, owner 59a86d66-edde46ca-a14a-0026b98a0395 mtime 75

         num 0 gblnum 0 gblgen 0 gblbrk 0]

   Found stale lock [type 10c00001 offset 288704512 v 264, hb offset 3719168

         gen 73, mode 1, owner 59a86d66-edde46ca-a14a-0026b98a0395 mtime 78

         num 0 gblnum 0 gblgen 0 gblbrk 0]

   Found stale lock [type 10c00001 offset 288706560 v 233, hb offset 3719168

         gen 73, mode 1, owner 59a86d66-edde46ca-a14a-0026b98a0395 mtime 81

         num 0 gblnum 0 gblgen 0 gblbrk 0]

   Found stale lock [type 10c00001 offset 288708608 v 234, hb offset 3719168

         gen 73, mode 1, owner 59a86d66-edde46ca-a14a-0026b98a0395 mtime 84

         num 0 gblnum 0 gblgen 0 gblbrk 0]

   Found stale lock [type 10c00001 offset 288710656 v 1171, hb offset 3719168

         gen 73, mode 1, owner 59a86d66-edde46ca-a14a-0026b98a0395 mtime 403628

         num 0 gblnum 0 gblgen 0 gblbrk 0]

   Found stale lock [type 10c00001 offset 288712704 v 236, hb offset 3719168

         gen 73, mode 1, owner 59a86d66-edde46ca-a14a-0026b98a0395 mtime 90

         num 0 gblnum 0 gblgen 0 gblbrk 0]

   Found stale lock [type 10c00001 offset 288714752 v 237, hb offset 3719168

         gen 73, mode 1, owner 59a86d66-edde46ca-a14a-0026b98a0395 mtime 93

         num 0 gblnum 0 gblgen 0 gblbrk 0]

   Found stale lock [type 10c00001 offset 288716800 v 238, hb offset 3719168

         gen 73, mode 1, owner 59a86d66-edde46ca-a14a-0026b98a0395 mtime 96

         num 0 gblnum 0 gblgen 0 gblbrk 0]

   Found stale lock [type 10c00001 offset 288718848 v 239, hb offset 3719168

         gen 73, mode 1, owner 59a86d66-edde46ca-a14a-0026b98a0395 mtime 99

         num 0 gblnum 0 gblgen 0 gblbrk 0]

   Found stale lock [type 10c00001 offset 288720896 v 240, hb offset 3719168

         gen 73, mode 1, owner 59a86d66-edde46ca-a14a-0026b98a0395 mtime 102

         num 0 gblnum 0 gblgen 0 gblbrk 0]

   Found stale lock [type 10c00001 offset 288722944 v 447, hb offset 3719168

         gen 73, mode 1, owner 59a86d66-edde46ca-a14a-0026b98a0395 mtime 371671

         num 0 gblnum 0 gblgen 0 gblbrk 0]

   Found stale lock [type 10c00001 offset 288724992 v 242, hb offset 3719168

         gen 73, mode 1, owner 59a86d66-edde46ca-a14a-0026b98a0395 mtime 108

         num 0 gblnum 0 gblgen 0 gblbrk 0]

   Found stale lock [type 10c00001 offset 288727040 v 243, hb offset 3719168

         gen 73, mode 1, owner 59a86d66-edde46ca-a14a-0026b98a0395 mtime 111

         num 0 gblnum 0 gblgen 0 gblbrk 0]

   Found stale lock [type 10c00001 offset 288729088 v 436, hb offset 3719168

         gen 73, mode 1, owner 59a86d66-edde46ca-a14a-0026b98a0395 mtime 410877

         num 0 gblnum 0 gblgen 0 gblbrk 0]

   Found stale lock [type 10c00001 offset 288731136 v 247, hb offset 3719168

         gen 73, mode 1, owner 59a86d66-edde46ca-a14a-0026b98a0395 mtime 117

         num 0 gblnum 0 gblgen 0 gblbrk 0]

   Found stale lock [type 10c00001 offset 288733184 v 246, hb offset 3719168

         gen 73, mode 1, owner 59a86d66-edde46ca-a14a-0026b98a0395 mtime 120

         num 0 gblnum 0 gblgen 0 gblbrk 0]

   Found stale lock [type 10c00001 offset 288735232 v 247, hb offset 3719168

         gen 73, mode 1, owner 59a86d66-edde46ca-a14a-0026b98a0395 mtime 123

         num 0 gblnum 0 gblgen 0 gblbrk 0]

   Found stale lock [type 10c00001 offset 288737280 v 248, hb offset 3719168

         gen 73, mode 1, owner 59a86d66-edde46ca-a14a-0026b98a0395 mtime 126

         num 0 gblnum 0 gblgen 0 gblbrk 0]

   Found stale lock [type 10c00001 offset 288739328 v 249, hb offset 3719168

         gen 73, mode 1, owner 59a86d66-edde46ca-a14a-0026b98a0395 mtime 129

         num 0 gblnum 0 gblgen 0 gblbrk 0]

   Found stale lock [type 10c00001 offset 288741376 v 250, hb offset 3719168

         gen 73, mode 1, owner 59a86d66-edde46ca-a14a-0026b98a0395 mtime 132

         num 0 gblnum 0 gblgen 0 gblbrk 0]

   Found stale lock [type 10c00001 offset 288743424 v 251, hb offset 3719168

         gen 73, mode 1, owner 59a86d66-edde46ca-a14a-0026b98a0395 mtime 135

         num 0 gblnum 0 gblgen 0 gblbrk 0]

   Found stale lock [type 10c00001 offset 288745472 v 252, hb offset 3719168

         gen 73, mode 1, owner 59a86d66-edde46ca-a14a-0026b98a0395 mtime 138

         num 0 gblnum 0 gblgen 0 gblbrk 0]

   Found stale lock [type 10c00001 offset 288747520 v 253, hb offset 3719168

         gen 73, mode 1, owner 59a86d66-edde46ca-a14a-0026b98a0395 mtime 141

         num 0 gblnum 0 gblgen 0 gblbrk 0]

   Found stale lock [type 10c00001 offset 288749568 v 254, hb offset 3719168

         gen 73, mode 1, owner 59a86d66-edde46ca-a14a-0026b98a0395 mtime 144

         num 0 gblnum 0 gblgen 0 gblbrk 0]

   Found stale lock [type 10c00001 offset 288751616 v 255, hb offset 3719168

         gen 73, mode 1, owner 59a86d66-edde46ca-a14a-0026b98a0395 mtime 147

         num 0 gblnum 0 gblgen 0 gblbrk 0]

   Found stale lock [type 10c00001 offset 288753664 v 302, hb offset 3719168

         gen 73, mode 1, owner 59a86d66-edde46ca-a14a-0026b98a0395 mtime 399859

         num 0 gblnum 0 gblgen 0 gblbrk 0]

   Found stale lock [type 10c00001 offset 288755712 v 257, hb offset 3719168

         gen 73, mode 1, owner 59a86d66-edde46ca-a14a-0026b98a0395 mtime 153

         num 0 gblnum 0 gblgen 0 gblbrk 0]

   Found stale lock [type 10c00001 offset 288757760 v 258, hb offset 3719168

         gen 73, mode 1, owner 59a86d66-edde46ca-a14a-0026b98a0395 mtime 156

         num 0 gblnum 0 gblgen 0 gblbrk 0]

   Found stale lock [type 10c00001 offset 288780288 v 2005, hb offset 3719680

         gen 9, mode 1, owner 59a63df3-68fd01f8-d04a-0026b98a0395 mtime 694

         num 0 gblnum 0 gblgen 0 gblbrk 0]

   Found stale lock [type 10c00001 offset 288804864 v 1908, hb offset 3719168

         gen 71, mode 1, owner 596bcee0-2a26bbc5-322c-0026b98a0395 mtime 325752

         num 0 gblnum 0 gblgen 0 gblbrk 0]

   Found stale lock [type 10c00001 offset 288806912 v 1909, hb offset 3719168

         gen 71, mode 1, owner 596bcee0-2a26bbc5-322c-0026b98a0395 mtime 325764

         num 0 gblnum 0 gblgen 0 gblbrk 0]

   Found stale lock [type 10c00001 offset 288811008 v 1910, hb offset 3719168

         gen 71, mode 1, owner 596bcee0-2a26bbc5-322c-0026b98a0395 mtime 325775

         num 0 gblnum 0 gblgen 0 gblbrk 0]

   Found stale lock [type 10c00001 offset 288813056 v 1911, hb offset 3719168

         gen 71, mode 1, owner 596bcee0-2a26bbc5-322c-0026b98a0395 mtime 325784

         num 0 gblnum 0 gblgen 0 gblbrk 0]

   Found stale lock [type 10c00001 offset 288815104 v 1912, hb offset 3719168

         gen 71, mode 1, owner 596bcee0-2a26bbc5-322c-0026b98a0395 mtime 325803

         num 0 gblnum 0 gblgen 0 gblbrk 0]

   Found stale lock [type 10c00001 offset 288825344 v 1866, hb offset 3719168

         gen 71, mode 1, owner 596bcee0-2a26bbc5-322c-0026b98a0395 mtime 26066

         num 0 gblnum 0 gblgen 0 gblbrk 0]

   Found stale lock [type 10c00001 offset 288831488 v 1927, hb offset 3719168

         gen 71, mode 1, owner 596bcee0-2a26bbc5-322c-0026b98a0395 mtime 587060

         num 0 gblnum 0 gblgen 0 gblbrk 0]

   Found stale lock [type 10c00001 offset 288862208 v 2015, hb offset 3719168

         gen 73, mode 1, owner 59a86d66-edde46ca-a14a-0026b98a0395 mtime 3986

         num 0 gblnum 0 gblgen 0 gblbrk 0]

Phase 4: Checking pathname and connectivity.

Phase 5: Checking resource reference counts.

Total Errors Found:           1

EDIT:

So I probably made matters worse but I used the Voma fix and now it removed all the locks..

voma -m vmfs -f check -d  /dev/disks/naa.6a4badb02ea41e00ff000042043581ad:1

Checking if device is actively used by other hosts

Running VMFS Checker version 1.2 in check mode

Initializing LVM metadata, Basic Checks will be done

ON-DISK ERROR: Invalid device Size 5999532356096, should be 5999532359168

Phase 1: Checking VMFS header and resource files

   Detected VMFS file system (labeled:'Stuff') with UUID:57f2fd3e-073f7330-c93c-0026b98a0395, Version 5:61

Phase 2: Checking VMFS heartbeat region

Phase 3: Checking all file descriptors.

Phase 4: Checking pathname and connectivity.

Phase 5: Checking resource reference counts.

ON-DISK ERROR: FB inconsistency found: (1464,0) allocated in bitmap, but never used

ON-DISK ERROR: FB inconsistency found: (1480,37) allocated in bitmap, but never used

ON-DISK ERROR: FB inconsistency found: (7799,180) allocated in bitmap, but never used

Total Errors Found:           4