Write latency of 110ms for an all flash seems high to me, but then again what is the VM doing in those 25mins? Thats important to know.
There are buffers all along the path from the application to the physical disk platter. Also a place where I would look for beter performance.
Which version are you using?
As KabirAli82 noted, you should be starting here with determining what is adding the additional load at this time of day (e.g. scheduled backups, large database job, large file-server dump etc.)
While you mentioned that this is an 'All Flash' cluster this doesn't really say much about the performance potential of this cluster, e.g. I have seen Hybrid clusters that outperform All-Flash clusters by an order of magnitude.
If the cluster can't handle 200MB/s without hitting contention then this doesn't sound like too performant (or potentially well-configured) of a cluster - if everything is on the vSAN HCL and you have a support contract then I would advise opening a Support Request with vSAN GSS to determine whether this is being pushed to its limits or misconfigured/other issues.
That being said, what type of workloads are we talking about here? I ask as if you have 200MB/s and ~275 IOPS then these are relatively massive IOs.
thank you both, Version is 6.7U2
Will check what is the VM scheduled to do at that time but in any case it looks like I will need to open a ticket with vSAN GSS
Let me know if there are any further things I need to ask/check out about the VM workload taking place that can help/explain this.
Hopefully the above steps will help determine if this write latency is to be expected or not.