Very High Disk Latency ESXI 5.5

[lasanthaj]

   Hi,

we have upgraded one of our ESXI Host to ESXI 5.1 to ESXI 5.5. This host has been working for  few weeks  without any issue. but suddenly we faced huge disk latency on Local attached LUN.ESXI run on HP DL 380p G8 Server. this LUN on direct attached local disk array (HP Smart Array P420i). we tried to troubleshoot using esxtop. please refer below screenshots related to faulty LUN

    ESXTOP HBA mode output : vmhba1 is backed with HP Smart Array P420i . very high DAVG and KAVG

  

    ESXTOP LUN  mode output : naa.600508b1001 is LUN caused high latency.

  

    there is no any alert on server Array side. how to troubleshoot this issue ??

    please advice.

    Thanks & Regards,

    Lasantha.

[a_p_]

Did you already check the hardware, especially the controller cache/battery? If there's an issue with the battery, the controller will switch from write-back to write-trough mode.

As a side not, HP released a couple of urgent advisories for Gen8 servers. One which requires a Microcode upgrade for v2 CPUs, and another one which requires iLO to be upgraded to v1.51 to avoid possible PSOD's. Both fixes are included in the June SPP.

André

[MKguy]

Seeing a big latency with few IOPS happening as in your screenshots is not that uncommon, though your case seems a bit extreme. Can you run IOmeter or something in a VM?

Also check your hpsa driver version:

esxcli software vib list | grep hpsa

The current one is scsi-hpsa 5.5.0.60-1OEM.550.0.0.1331820

Update the driver accordingly:

http://vibsdepot.hp.com/hpq/jun2014/esxi-550-devicedrivers/hpsa-5.5.0-1874913.zip

And make sure the firmware of the p420i is up to date as well, the current version is 5.42. You can use this bundle from the ESXi shell to update:

http://www.hp.com/swpublishing/MTX-5424e8ae66a84ed08ee64d98c7

[lasanthaj]

Dear MKGUY thanks for the info. ESXI hpsa version is  5.5.0.58-1OEM.550.0.0.1331820. but p420i firmware is 4.68.we have run with this firmware without any issue. i analysed IOPS usage with Dell Foglight (vKernel) there no huge IOPS usage on VMs. Data store latency shows as follows.

there are 8 power on VMs on this data store.

Thanks & Regards,

Lasantha.

[lasanthaj]

Dear a.p thanks for info. i informed to HP support team to check hardware. meanwhile i observed below error on iLO system logs.

   it seems something wrong with Cache module.

Thanks & Regards,

Lasantha.