HP Serial Attached SCSI Disk shows as "degraded" even though there's no hardware failure and file system health is good?

[bbiandov]

Hi everyone,

I noticed something recently on an HP server running the proper OEM specialized ISO image (HPE-ESXi-6.0.0-Update3-iso-600.9.7.0.17 (Hewlett Packard Enterprise) ESXi 6.0.0 Update 3 (Build 5050593)

HP Serial Attached SCSI Disk shows as "degraded" even though there's no hardware failure and file system health is good too? I've confirmed that the bare metal has no errors and that the disk array is NOT running in recovery mode due to disk failure or any other cause. This is iLO bare metal means of answering the question. Then within ESXi the hardware health shows green at the level of host management and monitoring.

The issue illustrated here only shows up inside the storage section, then datastore1 monitoring section of the UI. Can anyone assist with an explanation of this "degraded" status which I am seeing? Thank you

[Dave_the_Wave]

First make sure all the firmware in your ProLiant is updated, you'll have to do it from a SPP (Service Pack for ProLiant) .iso as the Windows updaters only work when they are its bare metal OS, not a Guest VM.

A recent SPP will update your mainboard, Smart Array controller, and firmware for each hot plug drive.

Then browser into your iLO (Integrated Lights Out) and read all logs and statuses, and I mean everything, drill down into the power supplies, the smart array, and each drive.

Your ProLiant will not disable a bad drive, most of the time it will still work.

Make sure the BBWC (Battery Backed Write Cache) battery is good too, those all fail when they are about 5 years old. A bad battery even with good drives will pop up alerts both the iLO's IML (Integrated Management Log) and ESXi.

[bbiandov]

Dave_the_Wave​ Thank you for your quick reply. All of those points were covered already , yes an SPP from a USB stick was applied and none of the pitfalls exist to explain the degraded status. There is nothing in IML historically speaking related to any bare metal failure. Neither drive nor any SmartArray conditions exist to justify ESXi's status which I now suspect is erroneous and defective so to speak. Can anyone provide in-depth means of determine WHERE ESXi is getting the info which triggers the display of that degraded status?

Thank you

[Dave_the_Wave]

It'll be getting it straight from the ProLiant's "Sea of sensors".

Do you have any non-HPE harddrives in the bays? ProLiant's freak out when that happens too.

You might also want to upgrade your host with the lastest iso:

VMware-ESXi-6.0.0-Update3-9313334-HPE-preGen9-600.9.8.5.4-Sep2018.iso

The one you mentioned now was from Feb2017.

Just put your host into maintenance mode, reboot with the USB stick, and choose upgrade (default).

[bbiandov]

So just because I'm stubborn I pulled a totally different ProLiant out of storage, single 300G disk, installed the same exact version of ESXi and guess what? Degraded status again LOL - You are running HPE Customized Image ESXi 6.0.0 Update 3 version 600.9.7.0.17 released on February 2017 and based on ESXi 6.0.0 Update 3 Vmkernel Release Build 5050593.

[Dave_the_Wave]

Why are you using the Feb2017 one when there is a Sep2018 released?

Maybe they fixed it ? (shrug)

[bbiandov]

Fair is fair. Upgraded -- You are running HPE Customized Image ESXi 6.0.0 Update 3 version 600.9.8.5 released on September 2018 and based on ESXi 6.0.0 Update 3 Vmkernel Release Build 9313334.

Now more detail is displayed and it looks like it doesn't like the the firmware level of HP Smart Array P410i Controller : Embedded : HPSA1 assuming that it is too old? However that is the latest firmware for this specific HP Smart Array P410i Controller released by HPE as of today. What gives VMWare?

[bbiandov]

Anyone has any update on this issue? ddavenport you got anything man? I'm running the latest possible 6.0 U3 from VMware-ESXi-6.0.0-Update3-9313334-HPE-preGen9-600.9.8.5.4-Sep2018.iso  and it still does not like the controller firmware level even though this is the latest available firmware from HPE

~B