Dear All,
Our esxi 6 hosts encountering high latency issue for long, and recently, cause more problem, such as vm guests on specific datastore being not responding.
we got 3 hosts with iSCSI connections to 3 datastores volume created on EqualLogic array using fiber,
We found the following message below, randomly appear in log
1) HBX: 2802: 'SANVOLUMEname': HB at offset 3887104 - Waiting for timed out HB:
2) esx.problem.vmfs.heartbeat.timedout
3)
[vob.scsi.device.io.latency.high] Device naa.datastorenumbers performance has deteriorated. I/O latency increased from average value of 37331 microseconds to 776880 microseconds.
[scsiCorrelator] 421249202885us: [esx.problem.scsi.device.io.latency.high] Device naa.datastorenumbers performance has deteriorated. I/O latency increased from average value of 37331 microseconds to 776880 microseconds.
With a search, I find the following KB for those message, which indicate hosts lost access to volume and high latency cause by overload.
https://kb.vmware.com/s/article/2007236
https://kb.vmware.com/s/article/2136081
three hosts server setting as a HA/FT cluster, and share same set of datastores on same storage, but the message logged on servers not consistent
for example,
host01 always logged with high latency on datastore03, but it is not logged by host02/03
some heartbeat timedout messages are not show with high latency, but when there are high latency, heartbeat timedout messages also logged
so can i think like that:
host01 logged with high latency on datastore03, but it is not logged by host02/03, as the VM Guest on datastore03 is manage under host01.
overload is not on SAN Storage / switches, but on the intermediate connection from server that logged the latency to switch.
if it is SAN Storage / switches overloaded, all host should be affected.
heartbeat timedout messages can be cause by high latency during overload.
Please advice, thanks.
