High Delayed IO Count

[seamusobr1]

We are running a stretched cluster 6.7 u3 with 10 hosts but every now and then we get impacted by storage latency issues. All devices cache and storage are ok and are not reporting any issues

However we get with storage latency issues with the VMs and when looking at the vsan storage metrics at the host level we get very high delayed IO Average Latency

Assuming the network is ok does anyone know what could cause this measurement to go so high

I have also checked cache evictions and this checks out ok as well

Thanks in advance

[paudieo]

https://kb.vmware.com/s/article/2144493?lang=en_US the KB doesnt give much detail as this can be a generic symptom of something else

This could potentially be a symtom of bursty IO,   where most of the time there was no delay, but due to the bursty nature of  IO workload pattern triggered some stress on the internal queues.
you may need to work back to what VMs were doing at that time to see if it was driven by temp spike on workload
But then again you are using the stretch cluster so the network can never be discounted 

I would advise to attempt to characterise any sudden change in workload from the VMs
you may need to get help from support at this to attempt to narrow down where this spike is coming from.