3 Replies Latest reply on Dec 21, 2019 1:41 AM by andvm

    VM write latency (vSAN)

    andvm Enthusiast

      Hi,

       

      From VM vSAN Performance graphs I routinely (daily) see at a specific time each day for around 25mins the following:

       

      Write Latency go up from around 1ms to 110ms

      Write IOPS go up from around 12 to 275

      Write Throughput go up from around 1MB/s to 200MB/s

      Outstanding IOPS from around 10 to to 600

       

      I also see spikes of Local Client Cache Hit IOPS go up from 0 to around 18 at the same

       

      Is this expected performance for an all flash vSAN with regards to the latency experienced by the VM when taking into consideration the Write throughput increase?

      Read throughput also increases around the same period but Read latency remains almost the same (only goes up by 1ms).

       

       

      Thanks

        • 1. Re: VM write latency (vSAN)
          KabirAli82 Enthusiast

          Hi there,

           

          Write latency of 110ms for an all flash seems high to me, but then again what is the VM doing in those 25mins? Thats important to know.

           

          There are buffers all along the path from the application to the physical disk platter. Also a place where I would look for beter performance.

           

          Which version are you using?

          1 person found this helpful
          • 2. Re: VM write latency (vSAN)
            TheBobkin Virtuoso
            VMware EmployeesvExpert

            Hello andvm

             

            As  KabirAli82 noted, you should be starting here with determining what is adding the additional load at this time of day (e.g. scheduled backups, large database job, large file-server dump etc.)

             

            While you mentioned that this is an 'All Flash' cluster this doesn't really say much about the performance potential of this cluster, e.g. I have seen Hybrid clusters that outperform All-Flash clusters by an order of magnitude.

            If the cluster can't handle 200MB/s without hitting contention then this doesn't sound like too performant (or potentially well-configured) of a cluster - if everything is on the vSAN HCL and you have a support contract then I would advise opening a Support Request with vSAN GSS to determine whether this is being pushed to its limits or misconfigured/other issues.

            That being said, what type of workloads are we talking about here? I ask as if you have 200MB/s and ~275 IOPS then these are relatively massive IOs.

             

            Bob

            1 person found this helpful
            • 3. Re: VM write latency (vSAN)
              andvm Enthusiast

              thank you both, Version is 6.7U2

               

              Will check what is the VM scheduled to do at that time but in any case it looks like I will need to open a ticket with vSAN GSS

               

              Let me know if there are any further things I need to ask/check out about the VM workload taking place that can help/explain this.

               

              Hopefully the above steps will help determine if this write latency is to be expected or not.