VMware Cloud Community
andvm
Hot Shot
Hot Shot

VM write latency (vSAN)

Hi,

From VM vSAN Performance graphs I routinely (daily) see at a specific time each day for around 25mins the following:

Write Latency go up from around 1ms to 110ms

Write IOPS go up from around 12 to 275

Write Throughput go up from around 1MB/s to 200MB/s

Outstanding IOPS from around 10 to to 600

I also see spikes of Local Client Cache Hit IOPS go up from 0 to around 18 at the same

Is this expected performance for an all flash vSAN with regards to the latency experienced by the VM when taking into consideration the Write throughput increase?

Read throughput also increases around the same period but Read latency remains almost the same (only goes up by 1ms).

Thanks

Reply
0 Kudos
3 Replies
KabirAli82
Expert
Expert

Hi there,

Write latency of 110ms for an all flash seems high to me, but then again what is the VM doing in those 25mins? Thats important to know.

There are buffers all along the path from the application to the physical disk platter. Also a place where I would look for beter performance.

Which version are you using?


Was I helpful? Give a kudo for appreciation!
Braindumping @ http://kablog.nl/
Tweeting @ https://twitter.com/_Kabir_Ali_
TheBobkin
Champion
Champion

Hello andvm

As  KabirAli82 noted, you should be starting here with determining what is adding the additional load at this time of day (e.g. scheduled backups, large database job, large file-server dump etc.)

While you mentioned that this is an 'All Flash' cluster this doesn't really say much about the performance potential of this cluster, e.g. I have seen Hybrid clusters that outperform All-Flash clusters by an order of magnitude.

If the cluster can't handle 200MB/s without hitting contention then this doesn't sound like too performant (or potentially well-configured) of a cluster - if everything is on the vSAN HCL and you have a support contract then I would advise opening a Support Request with vSAN GSS to determine whether this is being pushed to its limits or misconfigured/other issues.

That being said, what type of workloads are we talking about here? I ask as if you have 200MB/s and ~275 IOPS then these are relatively massive IOs.

Bob

andvm
Hot Shot
Hot Shot

thank you both, Version is 6.7U2

Will check what is the VM scheduled to do at that time but in any case it looks like I will need to open a ticket with vSAN GSS

Let me know if there are any further things I need to ask/check out about the VM workload taking place that can help/explain this.

Hopefully the above steps will help determine if this write latency is to be expected or not.

Reply
0 Kudos