VMware Cloud Community
spiffeluringen
Contributor
Contributor

ESXi 6.7 Datastore missing after RAID rebuild.

I have/had? a ESXi host with one VM but two datastores connected.
One disk in one of the RAID (10) setup crashed and the raid adapter rebuilt the RAID and said it was OK.

However one of the datastores did not turn up after reboot (DATASTORE1).
The other (DATA2) one seems ok, I can browse the datastore and see the vmdk file.

I can see a storage device in the web client with correct size but its not a datastore.
Same thing if I look in the RAID config. virtual drive with correct size present.

Anyone that has any more insight in this matter?

 

[root@VMESXi6:~] esxcli storage vmfs extent list
Volume Name VMFS UUID Extent Number Device Name Partition
----------- ----------------------------------- ------------- ------------------------------------ ---------
DATA2 5b645277-8d5e92a9-8bf8-40f2e9a61a78 0 naa.600605b00d1e455022827e6e1572e990 1
DATASTORE1 551b50cd-4562d988-2331-40f2e9a61a78 0 naa.600605b009e5ca201caeb6cc0e5e2939 1

[root@VMESXi6:~] ls -alh /vmfs/devices/disks
total 11713787793
drwxr-xr-x 2 root root 512 Feb 27 21:44 .
drwxr-xr-x 16 root root 512 Feb 27 21:44 ..
-rw------- 1 root root 1.9G Feb 27 21:44 mpx.vmhba32:C0:T0:L0
-rw------- 1 root root 4.0M Feb 27 21:44 mpx.vmhba32:C0:T0:L0:1
-rw------- 1 root root 250.0M Feb 27 21:44 mpx.vmhba32:C0:T0:L0:5
-rw------- 1 root root 250.0M Feb 27 21:44 mpx.vmhba32:C0:T0:L0:6
-rw------- 1 root root 110.0M Feb 27 21:44 mpx.vmhba32:C0:T0:L0:7
-rw------- 1 root root 286.0M Feb 27 21:44 mpx.vmhba32:C0:T0:L0:8
-rw------- 1 root root 2.2T Feb 27 21:44 naa.600605b009e5ca201caeb6cc0e5e2939
-rw------- 1 root root 2.2T Feb 27 21:44 naa.600605b009e5ca201caeb6cc0e5e2939:1
-rw------- 1 root root 3.3T Feb 27 21:44 naa.600605b00d1e455022827e6e1572e990
-rw------- 1 root root 3.3T Feb 27 21:44 naa.600605b00d1e455022827e6e1572e990:1
lrwxrwxrwx 1 root root 20 Feb 27 21:44 vml.0000000000766d68626133323a303a30 -> mpx.vmhba32:C0:T0:L0
lrwxrwxrwx 1 root root 22 Feb 27 21:44 vml.0000000000766d68626133323a303a30:1 -> mpx.vmhba32:C0:T0:L0:1
lrwxrwxrwx 1 root root 22 Feb 27 21:44 vml.0000000000766d68626133323a303a30:5 -> mpx.vmhba32:C0:T0:L0:5
lrwxrwxrwx 1 root root 22 Feb 27 21:44 vml.0000000000766d68626133323a303a30:6 -> mpx.vmhba32:C0:T0:L0:6
lrwxrwxrwx 1 root root 22 Feb 27 21:44 vml.0000000000766d68626133323a303a30:7 -> mpx.vmhba32:C0:T0:L0:7
lrwxrwxrwx 1 root root 22 Feb 27 21:44 vml.0000000000766d68626133323a303a30:8 -> mpx.vmhba32:C0:T0:L0:8
lrwxrwxrwx 1 root root 36 Feb 27 21:44 vml.0200000000600605b009e5ca201caeb6cc0e5e2939536572766552 -> naa.600605b009e5ca201caeb6cc0e5e2939
lrwxrwxrwx 1 root root 38 Feb 27 21:44 vml.0200000000600605b009e5ca201caeb6cc0e5e2939536572766552:1 -> naa.600605b009e5ca201caeb6cc0e5e2939:1
lrwxrwxrwx 1 root root 36 Feb 27 21:44 vml.0200000000600605b00d1e455022827e6e1572e990536572766552 -> naa.600605b00d1e455022827e6e1572e990
lrwxrwxrwx 1 root root 38 Feb 27 21:44 vml.0200000000600605b00d1e455022827e6e1572e990536572766552:1 -> naa.600605b00d1e455022827e6e1572e990:1

Reply
0 Kudos
7 Replies
a_p_
Leadership
Leadership

Please run esxcli storage vmfs snapshot list to see whether the datastore shows up. If it does, follow the "Command line" section at https://kb.vmware.com/s/article/1011387 for how to mount it again.
If not, run vmkfstools -V (note the upper-case -V), and then check the vmkernel.log for related entries.

André

Reply
0 Kudos
spiffeluringen
Contributor
Contributor

I ran esxcli storage vmfs snapshot list but nothing was listed

 

A couple of entrys in the vmkernel.log that probably is related is in the picture

 

Reply
0 Kudos
a_p_
Leadership
Leadership

I'm afraid that in this case only @continuum might be able to help.

André

Reply
0 Kudos
e_espinel
Virtuoso
Virtuoso

Hello.
Your case is very strange, RAID protection systems are very secure.
In my experience there are problems when the disk controller and the disks were not updated the firmware at the beginning of the server installation or were not updated periodically.

It is recommended to update all hardware firmware at least once a year if there are updates.

If the problem was caused by old firmware levels, it may recur.

 

Enrique Espinel
Senior Technical Support on IBM, Lenovo, Veeam Backup and VMware vSphere.
VSP-SV, VTSP-SV, VTSP-HCI, VTSP
Please mark my comment as Correct Answer or assign Kudos if my answer was helpful to you, Thank you.
Пожалуйста, отметьте мой комментарий как Правильный ответ или поставьте Кудо, если мой ответ был вам полезен, Спасибо.
Reply
0 Kudos
spiffeluringen
Contributor
Contributor

Yes I agree, I have never experienced a RAID to mess anything up, die yes, but not corrupt data.

I did a sectorcopy of the raid volume to a normal HDD and get this with only that HDD in the machine.

#
[root@VMESXi6:~] esxcli storage filesystem list
Mount Point Volume Name UUID Mounted Type Size Free
------------------------------------------------- ----------- ----------------------------------- ------- ------ ------------- -------------
/vmfs/volumes/551b50cd-4562d988-2331-40f2e9a61a78 DATASTORE1 551b50cd-4562d988-2331-40f2e9a61a78 true VMFS-5 2397933928448 1877172289536
/vmfs/volumes/7b6be5e5-70263cea-fe0e-fb11006cb6fe 7b6be5e5-70263cea-fe0e-fb11006cb6fe true vfat 261853184 93880320
/vmfs/volumes/5acf16e0-a706c55d-e17d-a153153dfbb5 5acf16e0-a706c55d-e17d-a153153dfbb5 true vfat 261853184 113680384
/vmfs/volumes/551b45e8-eebaf810-c3c1-40f2e9a61a78 551b45e8-eebaf810-c3c1-40f2e9a61a78 true vfat 299712512 80486400

#
[root@VMESXi6:~] esxcli storage vmfs extent list
Volume Name VMFS UUID Extent Number Device Name Partition
----------- ----------------------------------- ------------- ------------------------------------------------------------------------- ---------
DATASTORE1 551b50cd-4562d988-2331-40f2e9a61a78 0 t10.ATA_____ST8000DM0042D2CX188__________________________________ZCT3DDHT 1

Reply
0 Kudos
spiffeluringen
Contributor
Contributor

The LUN is still visible but I cant seem to do anything with it.

Any help would be appreciated, I´m at the edge of my (in)competece.

RAID says all OK, but obviously not.

Some hints to what to check perhaps.

 

Reply
0 Kudos
pozitive
Contributor
Contributor

if you can see your datastores with this comm

esxcli storage vmfs snapshot list

Then mount your datastore(s) with following command.

 esxcli storage vmfs snapshot mount -u UUID

 

Reply
0 Kudos