VM Disk Latency in Seconds - Why the large delta.

[PurpJake]

Hello, I have been scratching my head with the results of a report from Vrealize Operations Manager. It is listed under Performance, Summary| MAX Vm Disk Latency (ms)

There is a VM Disk Latency (95th percentile) column. Out of our 30-40 datastores, there are two used for VDI's that are in seconds. I have been unable to find a reason for this. I have looked at the flash storage array, the hosts, used esxtop, used iometer... I do see a reference in the vRealize Operations Manager manual that Datastore | Disk Command Latency shows the adjusted read and write latency on the datastore level. I did see that there was quite a bit more read IO's than write IO's.

At this point I'm wondering if it has to do with firmware/drivers/incorrect readings? Esx is at 6.5.0, on a Cisco UCS.

I appreciate any thoughts/ideas.

Max VM Disk Latency	VM Disk Latency (95th percentile)	VMs
68,832.89	10.27 ms	6
65,418.18	17.57 ms	20
58,868.16	129.02 ms	25
24,742.77	5.73 ms	26
24,719	8.47 Second(s)	42
21,058.99	1.59 Second(s)	24
3,297.15	329.07 ms	14
1,419.53	826.2 μs	24
916.58	133.33 μs	20
674.07	4.57 ms	45
197.6	321.57 μs	26
86.33	130.02 μs	28
82.91	16.02 μs	7
69.39	70.28 μs	19
68.69	913.98 μs	12
55.07	14.48 ms	19
53.67	20.56 ms	18
51.07	1 ms	18

[PurpJake]

The datastores listed are across four different datastore clusters across multiple arrays. The two datastores with VM Disk Latency in seconds, are the only ones on 6.5.0 and using a PureStorage all flash array. The others are 6.0.0 running on on flash arrays. Connectivity is via fiber channel. I attempted to look for correlations between the high latency and other metrics (read, write, io, etc...) and do not see one (for example backups).