ESXi 6 on HP Proliant Event warning / Information

[WallyL]

Good day experts,

We are running vmware on HP C7000 chassis with BL460 Gen 9 servers.

Each blade has Smart Array P244br with a 1 gig cache.

On our LOCAL disk inside vmware, we are getting this error: and this informational event:

Device naa.600508b1001ce43aaeef419836cfa629 performance has improved. I/O latency reduced from 71148

microseconds to 13944 microseconds.

info

1/16/2017 6:11:39 PM

vmhost10.scdgroup.ca

Device naa.600508b1001ce43aaeef419836cfa629 performance has deteriorated. I/O latency increased from average

value of 1616 microseconds to 71148 microseconds.

warning

1/16/2017 6:07:53 PM

vmhost10.scdgroup.ca

Device naa.600508b1001ce43aaeef419836cfa629 performance has deteriorated. I/O latency increased from average

value of 1577 microseconds to 34107 microseconds.

warning

1/16/2017 6:07:50 PM

vmhost10.scdgroup.ca

Now it just seems to be an warning and an informational. I have googled my butt off, and cant find a resolution to it. I do not know why this is happening either.

THIS IS LOCAL STORAGE NOT A SAN/LAN/ATTACHED ENVIRONMENT.

We have the exact same error on 6 hosts, in the exact same configuration.

We have tried changing the local raid controller cache to 30/70 read/write, turning on drive cache etc but to no avail. I'm quite honestly stuck.

I do not think that this is impacting performance in any way, the times are also all over the map, its never the same. I cant correlate this with a backup job. (the host that these came from in this post, does not have any guests that get backed up)

We are running according to the HP recipe guide from October 2016. With the appropriate firmware for that recipe guide.

Any thoughts would be great.

Regards,
Wally

[homerzzz]

I have the same hardware as you and I only see these messages logged if Storage I/O Control is enabled on the local datastore. I disable this on the local datastore since I do not run any VMs locally.

[WallyL]

This makes sense, let me try and walk through it to be sure.

LOCATION: VMclient\home\inventory\datastores and clusters\

Click on the datastore in question, click on configuration, click on properties, then its the storage I/O Control checkbox?

Ours is unchecked on all local datastores.

If I try and check this, I get "license is not available to perform the operation"

Do you have SSDs in your servers? ours are just spinning disks.

[homerzzz]

No SSDs. I see this same behavior on Gen8 blades also.

[WallyL]

We also do not run any VMs locally, and I still get the error.

[ArjunDooti]

Those warning messages are expected when ever there is huge IO on the datastore.

Is it possible to send the out put for below commands.

esxcfg-scsidevs -m

Only for reference.

"<storage_device> performance has deteriorated" message in ESXi (2007236) | VMware KB

Thanks & Regards

Arjun Dooti

[homerzzz]

Just curious, is this setting enabled for your local disks? If this is disabled, I never see these messages.

Also, there has been an update to the driver since the 10/2016 recipe.

HPE Support document - HPE Support Center

[WallyL]

Homerzzz,

Yes that is enabled on my servers local storage.

Yes there is a new version of the driver, I just finished the research on the next stream for us. This is a highly productive system so changes are infrequent.

This is the next stream plan:

December SPP - with Vmware build 600.9.5.0.48 - 2494585 - 3par 3.2.2.MU3 (current 3.2.1.MU3 p50)

[WallyL]

naa.60002ac0000000000000001200012593:1	/vmfs/devices/disks/naa.60002ac0000000000000001200012593:1 55b02a71-73a6327c-e7e7-8cdcd41ba710  0  3par_xxxx
naa.60002ac0000000000000002200012593:1	/vmfs/devices/disks/naa.60002ac0000000000000002200012593:1 55c1531d-28abd067-57c7-8cdcd41ba710  0  3par_xxxx
naa.60002ac0000000000000002700012593:1	/vmfs/devices/disks/naa.60002ac0000000000000002700012593:1 55e5bfc0-0a381d99-da05-8cdcd41ba710  0  3par_xxxx
naa.60002ac0000000000000002400012593:1	/vmfs/devices/disks/naa.60002ac0000000000000002400012593:1 55c38849-00ff16e1-7cb3-8cdcd41ba780  0  3par_xxxx
naa.60002ac0000000000000001e00012593:1	/vmfs/devices/disks/naa.60002ac0000000000000001e00012593:1 55c0d1e3-0520058c-9bfe-8cdcd41ba710  0  3par_xxxx
naa.600508b1001ce43aaeef419836cfa629:3	/vmfs/devices/disks/naa.600508b1001ce43aaeef419836cfa629:3 55a68d2e-5f42ae0b-a16c-8cdcd41bae50  0  VMhost10_Local
naa.60002ac0000000000000002500012593:1	/vmfs/devices/disks/naa.60002ac0000000000000002500012593:1 55d49690-bc60f57d-c119-8cdcd41ba710  0 xxxx

So we have zero issues or warnings or any other problems on anything 3par. Its only on the local storage.
In this case, naa.600508b1001ce43aaeef419836cfa629:3                       /vmfs/devices/disks/naa.600508b1001ce43aaeef419836cfa629:3 55a68d2e-5f42ae0b-a16c-8cdcd41bae50  0  VMhost10_Local

[WallyL]

Ok, I have disabled "Storage DRS I/O Metrics" On VMhost10 Local datastore. Lets see what happens.

[WallyL]

Turning this off did not make any changes.

Attached are screen shots of the errors and of the disabled I/O settings.

[WallyL]

What about this setting?

I only checked the bottom one.

[WallyL]

I uploaded the output. and there should be nothing on this datastore. its just local esxi installation nothing more.

[WallyL]

I decided to try and check the highlighted check box, and I will reboot the host too, just to see what happens.

I did not reboot the host between changes last week.

[WallyL]

No Errors as of 2:53 MST January 25th 2017

[WallyL]

No further Errors on VMhost10, I have made this change to all other hosts and will now monitor for changes.