Datastore no longer available

[ylafont22]

Good afternoon. Apologies if this I am posting in an incorrect forum (was not really sure of the best place, please move is it incorrect).   I am in middle of a pickle which I need to troubleshoot as I am not an ESXI expert.   Today someone noticed  an issue where a datastore is report 0 size, this has been up for months.   The host now  reports machine are invalid.    

I figured I reboot, but do not think it would solve the issue,   can anyone provide any steps to take and resolve this issue? Find what the problem is?   

The volumen is list with the command esxcli storage filesystem list

and this is how there appear on the host

Just  update - i did rescan of the datastore, and new "grey out" storage has appeared with a warning on kernel.log

21-05-27T00:23:35.908Z cpu21:2097708)WARNING: NMP: vmk_NmpSatpIssueTUR:1074: Device naa.6141877033256b002644af641f601b56 path vmhba2:C2:T0:L0 has been unmapped from the array

is my datastore gone?????   can i create a new on with UID and find my files?

thank you in advance - assistance is vey much appreciated. 

[e_espinel]

Hello.
Have you already checked the Hardware logs or do you have access to the server manager service (type ILO, IMM, Clarity)?

Please run the following commands to see the list of devices and partitions on the ESXi host, paste the result in the post

# esxcli storage core adapter list

# df -h

[ylafont22]

e_espinel, thank you for  the assistance.

Server is still up and accesible via the gui and  cli, cant get to anything else.  Info below. 

[e_espinel]

Hello.
Sorry I did not see your update of your first post, it is preferable to make a reply to add information.
Apparently the ESXi host lost a disk or access to it, do you know how many disk and/or volumes it has (internal, external) and what is its configuration (RAID).

Please check if you have the following directory on the ESXi host
/opt/lsi/storcli

If you have it you could enter the directory and run
./storcli show all

Please paste in the post your result

Please indicate what version of vmware you have installed so we can look for the respective storcli to install it.

[ylafont22]

No worries, i appreciate the assistance. thank you again.

There should  3 Disk ( raid 5)  one volume. 

External Raid.

/opt only has the vmware directory (no lsi)

esxi 6.7.0 (Build 8169922)

Please let me know what else can be provided.

[e_espinel]

Hello.
Download the MegaRAID Storcli Version 007.1211.0000.0000 from the attached link (Management Software and Tools)

https://www.broadcom.com/support/download-search?pa=&pf=RAID+Controller+Cards&pg=Storage+Adapters%2C...

You will have to unpack the zip file and in the path 007.1211.0000.0000_Unified_StorCLI>Unified_storcli_all_os>MMwareOP> you will find the vmware-storcli.vib file.

then copy vmware-storcli.vib to ESxi host in some directory (/tmp)

Install the StorCli (vmware-storcli.vib) with the following command:
# esxcli software vib install -v /tmp/vmware-storcli.vib --no-sig-check

You should get something like this (the file is different)

the StorCLI does not require a reboot.

After installing Storcli apply as indicated in the above post

[ylafont22]

Installed successfully, 

Hope this does not matter, notice the number of controllers is "0",  I believe the install control if a Perc H730

esxcli storage core adapter list
HBA Name Driver Link State UID Capabilities Description
-------- -------- ---------- -------------------- ------------ ----------------------------------------------------------
vmhba0 vmw_ahci link-n/a sata.vmhba0 (0000:00:11.4) Intel Corporation Wellsburg AHCI Controller
vmhba1 vmw_ahci link-n/a sata.vmhba1 (0000:00:1f.2) Intel Corporation Wellsburg RAID Controller
vmhba2 lsi_mr3 link-n/a sas.5141877033256b00 (0000:82:00.0) Avago (LSI) PERC H730 Adapter
vmhba32 vmkusb link-n/a usb.vmhba32 () USB

[e_espinel]

Hello.
For Avago (LSI) PERC H730 adapter, you need the vmware-perccli-007.0127.0000.0000.vib you can get it from the following link

https://www.dell.com/support/home/en-us/drivers/driversdetails?driverid=1prwx

You must run it on a PC and in the Dell folder look for the indicated vib.

Then copy it to the ESXi host and install it.
# esxcli software vib install -v /tmp/vmware-perccli-xxx.xxxx.xxxx.xxxx.xxxx.xxxx.vib --no-sig-check

Browse to the /opt/lsi/perccli directory and enter it to execute the following commands

./perccli show

If the controller has ID 0 (as expected) run the command
./perccli /c0 show

[ylafont22]

Can we confirm the file download  "perccli_7.1-007.0127_vmw_ZPE.exe"   from https://www.dell.com/support/home/en-us/drivers/driversdetails?driverid=1prwx

Get a "Bad VIB archive header" error,  

found these in the esxiupdate.log

2021-05-28T01:59:24Z esxupdate: 6696441: downloader: INFO: Downloading ./vmware-perccli-007.0127.000 to /tmp/vib_2zpwny7z
2021-05-28T01:59:24Z esxupdate: 6696441: root: ERROR: Traceback (most recent call last):
2021-05-28T01:59:24Z esxupdate: 6696441: root: ERROR: File "/build/mts/release/bora-8169922/bora/build/esx/release/vmvisor/sys-boot/lib64/python3.5/site-packages/vmware/esximage/Downloader.py", line 244, in _getfromurl
2021-05-28T01:59:24Z esxupdate: 6696441: root: ERROR: File "/build/mts/release/bora-8169922/bora/build/esx/release/vmvisor/sys-boot/lib64/python3.5/site-packages/vmware/esximage/Downloader.py", line 188, in _retry
2021-05-28T01:59:24Z esxupdate: 6696441: root: ERROR: File "/build/mts/release/bora-8169922/bora/build/esx/release/vmvisor/sys-boot/lib64/python3.5/site-packages/vmware/esximage/Downloader.py", line 208, in _download_to_file
2021-05-28T01:59:24Z esxupdate: 6696441: root: ERROR: File "/build/mts/release/bora-8169922/bora/build/esx/release/vmvisor/sys-boot/lib64/python3.5/site-packages/vmware/esximage/Downloader.py", line 180, in _urlopen
2021-05-28T01:59:24Z esxupdate: 6696441: root: ERROR: File "/build/mts/release/bora-8169922/bora/build/esx/release/vmvisor/sys-boot/lib64/python3.5/urllib/request.py", line 451, in open
2021-05-28T01:59:24Z esxupdate: 6696441: root: ERROR: File "/build/mts/release/bora-8169922/bora/build/esx/release/vmvisor/sys-boot/lib64/python3.5/urllib/request.py", line 269, in __init__
2021-05-28T01:59:24Z esxupdate: 6696441: root: ERROR: File "/build/mts/release/bora-8169922/bora/build/esx/release/vmvisor/sys-boot/lib64/python3.5/urllib/request.py", line 295, in full_url
2021-05-28T01:59:24Z esxupdate: 6696441: root: ERROR: File "/build/mts/release/bora-8169922/bora/build/esx/release/vmvisor/sys-boot/lib64/python3.5/urllib/request.py", line 324, in _parse
2021-05-28T01:59:24Z esxupdate: 6696441: root: ERROR: ValueError: unknown url type: './vmware-perccli-007.0127.000'
2021-05-28T01:59:24Z esxupdate: 6696441: root: ERROR:
2021-05-28T01:59:24Z esxupdate: 6696441: root: ERROR: During handling of the above exception, another exception occurred:
2021-05-28T01:59:24Z esxupdate: 6696441: root: ERROR:
2021-05-28T01:59:24Z esxupdate: 6696441: root: ERROR: Traceback (most recent call last):
2021-05-28T01:59:24Z esxupdate: 6696441: root: ERROR: File "/build/mts/release/bora-8169922/bora/build/esx/release/vmvisor/sys-boot/lib64/python3.5/site-packages/vmware/esximage/Transaction.py", line 275, in _getVibsFromUrls
2021-05-28T01:59:24Z esxupdate: 6696441: root: ERROR: File "/build/mts/release/bora-8169922/bora/build/esx/release/vmvisor/sys-boot/lib64/python3.5/site-packages/vmware/esximage/Downloader.py", line 335, in Get
2021-05-28T01:59:24Z esxupdate: 6696441: root: ERROR: File "/build/mts/release/bora-8169922/bora/build/esx/release/vmvisor/sys-boot/lib64/python3.5/site-packages/vmware/esximage/Downloader.py", line 246, in _getfromurl
2021-05-28T01:59:24Z esxupdate: 6696441: root: ERROR: vmware.esximage.Downloader.DownloaderError: ('./vmware-perccli-007.0127.000', '/tmp/vib_2zpwny7z', "unknown url type: './vmware-perccli-007.0127.000'")
2021-05-28T01:59:24Z esxupdate: 6696441: root: ERROR:
2021-05-28T01:59:24Z esxupdate: 6696441: root: ERROR: During handling of the above exception, another exception occurred:
2021-05-28T01:59:24Z esxupdate: 6696441: root: ERROR:
2021-05-28T01:59:24Z esxupdate: 6696441: root: ERROR: Traceback (most recent call last):
2021-05-28T01:59:24Z esxupdate: 6696441: root: ERROR: File "/usr/lib/vmware/esxcli-software", line 475, in <module>
2021-05-28T01:59:24Z esxupdate: 6696441: root: ERROR: main()
2021-05-28T01:59:24Z esxupdate: 6696441: root: ERROR: File "/usr/lib/vmware/esxcli-software", line 466, in main
2021-05-28T01:59:24Z esxupdate: 6696441: root: ERROR: ret = CMDTABLE[command](options)
2021-05-28T01:59:24Z esxupdate: 6696441: root: ERROR: File "/usr/lib/vmware/esxcli-software", line 361, in VibInstallCmd
2021-05-28T01:59:24Z esxupdate: 6696441: root: ERROR: nosigcheck=opts.nosigcheck)
2021-05-28T01:59:24Z esxupdate: 6696441: root: ERROR: File "/build/mts/release/bora-8169922/bora/build/esx/release/vmvisor/sys-boot/lib64/python3.5/site-packages/vmware/esximage/Transaction.py", line 235, in InstallVibsFromSources
2021-05-28T01:59:24Z esxupdate: 6696441: root: ERROR: File "/build/mts/release/bora-8169922/bora/build/esx/release/vmvisor/sys-boot/lib64/python3.5/site-packages/vmware/esximage/Transaction.py", line 165, in GetVibsFromSources

did i miss a step?

[e_espinel]

Hello.
Please try one of the following variants in the command

#esxcli software vib install -v=/tmp/vmware--perccli-007.0127.0000.0000.vib --no-sig-check
or
esxcli software vib install -n=/tmp/vmware--perccli-007.0127.0000.0000.0000.vib --no-sig-check

[ylafont22]

No go. 

[e_espinel]

Hello

# esxcli software vib install -v "/tmp/vmware-perccli-007.0127.0000.0000.vib"

or

# esxcli software vib install -v file:/tmp/vmware-perccli-007.0127.0000.0000.vib

or

# esxcli software vib install -v file:/tmp/vmware-perccli-007.0127.0000.0000.vib --no-sig-check

Run the following command to check for Dell software installed on the ESXi Host

# esxcli software vib list

[ylafont22]

no go, on these as well

according to documentation the command is correct, all that is required is -v along with --no-sig-check

https://www.dell.com/support/kbdoc/en-us/000177280/how-to-use-the-poweredge-raid-controller-perc-com...

[e_espinel]

Hello.
an important detail, which I overlooked
the install command must be executed from the root (/) not inside /tmp.

[ylafont22]

i believe i tried that previously,  but just got the same result "Bad VIB archive header"

[e_espinel]

Hello.
Please delete the file vmware-perccli-007.0127.0000.0000.vib from the tmp directory.
load the file vmware-perccli-007.0127.0027.0000.0000.vib using the Browser Storage of the web client
retry the installation.

[e_espinel]

Hello.
we are going to use another version of Perccli and package
https://www.dell.com/support/home/en-us/drivers/driversdetails?driverid=5v7xx
find the file vmware-perccli-007.1420.vib and load it on ESXi
run the installation
esxcli software vib install -v /path/xxxxxx.vib (full path must be used for installation)

[ylafont22]

Ureka! that one did the trick.

=~=~=~=~=~=~=~=~=~=~=~= PuTTY log 2021.05.28 14:44:46 =~=~=~=~=~=~=~=~=~=~=~=
clasclarear
[root@localhost:/opt/lsi/perccli] [root@localhost:/opt/lsi/perccli] clear [root@localhost:/opt/lsi/perccli] ./perccli /c0 show
Generating detailed summary of the adapter, it may take a while to complete.

CLI Version = 007.1420.0000.0000 Dec 10, 2020
Operating system = VMkernel 6.7.0
Controller = 0
Status = Success
Description = None

Product Name = PERC H730 Adapter
Serial Number = 59500AH
SAS Address = 5141877033256b00
PCI Address = 00:82:00:00
System Time = 05/28/2021 18:45:04
Mfg. Date = 09/08/15
Controller Time = 05/28/2021 18:40:15
FW Package Build = 25.3.0.0016
BIOS Version = 6.23.03.0_4.16.07.00_0x060C0200
FW Version = 4.250.00-4402
Driver Name = lsi_mr3
Driver Version = 7.702.13.00
Current Personality = RAID-Mode
Vendor Id = 0x1000
Device Id = 0x5D
SubVendor Id = 0x1028
SubDevice Id = 0x1F43
Host Interface = PCI-E
Device Interface = SAS-12G
Bus Number = 130
Device Number = 0
Function Number = 0
Domain ID = 0
Security Protocol = None
Drive Groups = 1

TOPOLOGY :
========

-----------------------------------------------------------------------------
DG Arr Row EID:Slot DID Type State BT Size PDC PI SED DS3 FSpace TR
-----------------------------------------------------------------------------
0 - - - - RAID5 OfLn N 18.190 TB dflt N N dflt N N
0 0 - - - RAID5 Dgrd N 18.190 TB dflt N N dflt N N
0 0 0 :0 0 DRIVE Failed N 9.094 TB dflt N N dflt - N
0 0 1 :1 1 DRIVE Onln N 9.094 TB dflt N N dflt - N
0 0 2 :2 2 DRIVE Failed N 9.094 TB dflt N N dflt - N
-----------------------------------------------------------------------------

DG=Disk Group Index|Arr=Array Index|Row=Row Index|EID=Enclosure Device ID
DID=Device ID|Type=Drive Type|Onln=Online|Rbld=Rebuild|Optl=Optimal|Dgrd=Degraded
Pdgd=Partially degraded|Offln=Offline|BT=Background Task Active
PDC=PD Cache|PI=Protection Info|SED=Self Encrypting Drive|Frgn=Foreign
DS3=Dimmer Switch 3|dflt=Default|Msng=Missing|FSpace=Free Space Present
TR=Transport Ready

Virtual Drives = 1

VD LIST :
=======

--------------------------------------------------------------
DG/VD TYPE State Access Consist Cache Cac sCC Size Name
--------------------------------------------------------------
0/0 RAID5 OfLn RW No NRWTD - OFF 18.190 TB
--------------------------------------------------------------

VD=Virtual Drive| DG=Drive Group|Rec=Recovery
Cac=CacheCade|OfLn=OffLine|Pdgd=Partially Degraded|Dgrd=Degraded
Optl=Optimal|dflt=Default|RO=Read Only|RW=Read Write|HD=Hidden|TRANS=TransportReady|B=Blocked|
Consist=Consistent|R=Read Ahead Always|NR=No Read Ahead|WB=WriteBack|
AWB=Always WriteBack|WT=WriteThrough|C=Cached IO|D=Direct IO|sCC=Scheduled
Check Consistency

Physical Drives = 3

PD LIST :
=======

---------------------------------------------------------------------------------
EID:Slt DID State DG Size Intf Med SED PI SeSz Model Sp Type
---------------------------------------------------------------------------------
:0 0 Failed 0 9.094 TB SATA HDD N N 512B ST10000DM0004-2GR11L U -
:1 1 Onln 0 9.094 TB SATA HDD N N 512B ST10000DM0004-2GR11L U -
:2 2 Failed 0 9.094 TB SATA HDD N N 512B ST10000DM0004-1ZC101 U -
---------------------------------------------------------------------------------

EID=Enclosure Device ID|Slt=Slot No.|DID=Device ID|DG=DriveGroup
DHS=Dedicated Hot Spare|UGood=Unconfigured Good|GHS=Global Hotspare
UBad=Unconfigured Bad|Sntze=Sanitize|Onln=Online|Offln=Offline|Intf=Interface
Med=Media Type|SED=Self Encryptive Drive|PI=Protection Info
SeSz=Sector Size|Sp=Spun|U=Up|D=Down|T=Transition|F=Foreign
UGUnsp=UGood Unsupported|UGShld=UGood shielded|HSPShld=Hotspare shielded
CFShld=Configured shielded|Cpybck=CopyBack|CBShld=Copyback Shielded
UBUnsp=UBad Unsupported|Rbld=Rebuild

BBU_Info :
========

----------------------------------------------
Model State RetentionTime Temp Mode MfgDate
----------------------------------------------
BBU Optimal 0 hour(s) 58C - 0/00/00
----------------------------------------------

[e_espinel]

Hello.
In the PERC H730 Adapter we have a RAID 5 with 3 Disks of which 2 are failed.
In general in a RAID 5 with 2 failed disks the data is lost.

You can try to recover the RAID, for which you need 2 good disks to change them in their order of failure.

The disks would be ST10000DM0004 10TB 7200 RPM 256MB Cache SATA 6.0Gb/s 3.5". but you need to physically confirm if you have a p/n for your order.

You need to get the event log to determine the order of failure, usually the first one to fail is the damaged one and the second one to fail was traced by the first one, but it can be good.

The ESXi host is booting from another controller or device.

[e_espinel]

Hello

To get the event log of the controller

perccli /cx show events file=<absolute path>

This command prints the system log to a text file and saves the file in the specified location