ESXi 6.5 SCSI Disk Degraded after Update

[eetee812]

Hi, 

My SCSI fell to "normal, degraded" after the update of Host A, 

This is my spec, 

- 2 Dell PowerEdge R630 - (Host A, B)

- 2 Host sharing the SCSI disk as datastore. 

It's just Host A having the degraded issue. Everything is normal with Host B. 

I tried to google online, and it seems like the SCSI disk is set as remote disk after the update. 

Solved: Disks degraded in ESXi - VMware Technology Network VMTN

However, when I try running the solution on VCenter and CLI. It returns an error. In the CLI it shows the following error, 

2021-07-24T09:47:29.883Z cpu20:66134)WARNING: NMP: nmp_SelectPathAndIssueCommand:4616: PSP selected path "vmhba3:C0:T1:L1" in a bad state (standby) on device "naa.".
2021-07-24T09:47:29.883Z cpu20:66134)WARNING: NMP: nmpCompleteRetryForPath:357: Retry cmd 0x12 (0x439501399640) to dev "naa." failed on path "vmhba3:C0:T1:L1" H:0x1 D:0x0 P:0x0 Invalid sense data: 0x0 0x0 0x0.
2021-07-24T09:47:29.883Z cpu20:66134)WARNING: NMP: nmpCompleteRetryForPath:387: Logical device "naa.": awaiting fast path state update before retrying failed command again...
2021-07-24T09:47:30.883Z cpu16:66560)WARNING: NMP: nmp_RegisterDeviceEvents:937: Could not register events 0x180 on device "naa.", Status: Not found. Event Registration will be retried on next path eval.
2021-07-24T09:47:30.883Z cpu10:66555)WARNING: NMP: nmpDeviceAttemptFailover:640: Retry world failover device "naa." - issuing command 0x439501399640
2021-07-24T09:47:30.883Z cpu10:66555)WARNING: NMP: nmp_SelectPathAndIssueCommand:4616: PSP selected path "vmhba3:C0:T1:L1" in a bad state (standby) on device "naa.".
2021-07-24T09:47:30.883Z cpu10:66555)WARNING: NMP: nmpCompleteRetryForPath:357: Retry cmd 0x12 (0x439501399640) to dev "naa." failed on path "vmhba3:C0:T1:L1" H:0x1 D:0x0 P:0x0 Invalid sense data: 0x0 0x0 0x0.
2021-07-24T09:47:30.883Z cpu10:66555)WARNING: NMP: nmpCompleteRetryForPath:387: Logical device "naa.": awaiting fast path state update before retrying failed command again...

Anyone has any idea on how to resolve this issue?

[e_espinel]

Hello.
Did you install ESXi using the custom image from the server manufacturer or did you use the standard VMware image?
Do you have installed any disk management tool provided by the disk controller manufacturer?

check if the opt/lsi/perccli directory exists.

with the following command check which controller you have installed
# esxcli storage core adapter list

When you said that the two Host sharing the SCSI disk as datastore. does it mean that you have an external disk enclosure or an external storge?

[eetee812]

Thanks e_espinel. I just started managing the ESXi. Please forgive me if I ask stupid questions. 

Did you install ESXi using the custom image from the server manufacturer or did you use the standard VMware image?

It was not installed by me but i think it should be a custom image. 

I used cli to update the ESXi. When i tried to install the latest version, the host return with an signer error. I googled it a bit and decided to install update-from-esxi6.5-6.5_update02 first. Then updated to ESXi650-202107001.zip. When I reboot the system the SCSI disk was gone. 

I remembered that the iSCSI Software Adapter was gone in Host A after the update. I need to add them back in VCenter. 

Do you have installed any disk management tool provided by the disk controller manufacturer?

I am not sure. Is there any way to check?

check if the opt/lsi/perccli directory exists.

No but I cannot find it on Host B neither. I found the following on opt/

- emulex
- vmware

When you said that the two Host sharing the SCSI disk as datastore. does it mean that you have an external disk enclosure or an external storage?

It is an external disk enclosure. Power Edge R630 controlled by iDRAC. 

[eetee812]

For esxcli storage core adapter list it lists the following, 

vmhba0 lsi_mr3 link-n/a sas.id(0000:03:00.0) Avago (LSI) PERC H730 Mini
vmhba1 vmw_ahci link-n/a sata.vmhba1 (0000:00:11.4) Intel Corporation Wellsburg AHCI Controller
vmhba2 vmw_ahci link-n/a sata.vmhba2 (0000:00:1f.2) Intel Corporation Wellsburg AHCI Controller
vmhba3 lsi_msgpt3 link-n/a sas.id (0000:04:00.0) Avago (LSI Logic) Dell 12Gbps SAS HBA external
vmhba64 iscsi_vmk online iscsi.vmhba64 Second Level Lun ID iSCSI Software Adapter

[eetee812]

I guess it's the mulitpathing issue. On Host A which I installed the patch, 

On Host A, 

On Host B, 

It seems I miss an active I/O. Any advise on how to fix it?

[eetee812]

I ran through the 

Troubleshooting ESXi connectivity to iSCSI arrays using software initiators (1003952) (vmware.com)

esxcfg-mpath -b -d 

naa.id : COMPELNT Serial Attached SCSI Disk (naa.6000d31003f626000000000000000006)
vmhba3:C0:T1:L1 LUN:1 state:standby sas Adapter: 5d09466025089100 Target: 5000d31003f6261

[e_espinel]

Hello

Run the following command to verify the version and build of the installed VMware vSphere
# vmware -vl

To check if it is installed with a custom image from a manufacturer e.g. Lenovo, HP, Dell etc. run the following command and look at the listing header
# esxcli software profile get

You sent the vmhba3 path captures i.e. you are using the Dell 12Gbps SAS HBA adapter for the connection of the ESXi host to the external disk enclosure (this is a SAS connection). This means that between the ESXi host and the external stora the cables are SAS.
I also see the vmhba64 adapter which is software for ISCSI connections, this requires connections between the ESXi host and an external storage with Ethernet cables.
In summary you have both SAS (SCSI) and ISCSI (Ethernet) connections defined and working on your servers?

Let's install Dell's PERCCLI
1. Download the VMware PERCCLI utility from Dell support site. Select your system, then Drivers & downloads filtering by category SAS RAID or by using the keyword PERCCLI
https://www.dell.com/support/home/en-us/drivers/driversdetails?driverid=1prwx

2. Run the perccli_7.1-007.0127_vmw_ZPE.EXE in PC
It will create a directory C:\Dell\Drivers\1PRWX\VMMWare
Where you can obtain the file vmware-perccli-007.0127.0000.0000.vib

3. Copy the file vmware-perccli-007.0127.0027.0000.0000.vib to ESXi in the /tmp directory.

4. Install the VIB package using the command:
# esxcli software vib install -v /tmp/vmware-perccli- 07.0127.0000.0000.0000.vib --no-sig-check

go to the following directory cd/opt/lsi/perccli
execute the command
# ./perccli show
verify which controllers are listed.

to verify the vmhb3 Firmware and driver run the following command
# esxcli storage san sas list

Attach in this post the result of the indicated commands

[eetee812 · Accepted solution]

Thanks for the reply. 

I have resolved the issue by following this post, 

Storage Center systems with Front End SAS connectivity show lun capacity 0MB (67032) (vmware.com)

It's working fine after adding the value "0" to ssue_scsi_cmd_to_bringup_drive. The value should be overridden after the update. 

[e_espinel]

Hello.

This is the link from Dell, I did not send it until I was sure your connection was SAS.

https://www.dell.com/support/kbdoc/en-us/000118040/preparing-vmware-esxi-hosts-to-attach-to-scv20x0-...

Verify that the vcenter server version and the ESXi hosts are compatible and if possible try to have them updated.

https://partnerweb.vmware.com/comp_guide2/sim/interop_matrix.php