About checksum or write latencies

[ManuelDB]

Hello,

I'm in a situation where a customer has an "old" 4 nodes VSAN 6.7u3 cluster (E5-26xx V3 processors, full flash but SATA SSDs, 2DG per Host). All is correctly configured, but write latencies are not the best.

Their new ERP supplier asked them to doing some test to validate the infrastructure, and one of them is this:

diskspd.exe -w100 -t8 -o8 -d900 -r -b64k -h -L -c80G

I know that this is single disk (and not whole cluster like HCI Bench) so not the best test for VSAN, but the requirement is that launching thist test on the DB disk of the ERP, the system could achieve more than 100MB/S (and this is ok) but also LESS than 5ms latency.

This last requirement is our issue. We have managed to achieve around 7ms latency increasing the stripe to 3 (all cluster is using a simple RAID 1 policy, no RAID 5 involved) and formatting disk with 64k blocks, but can't get better performance on 64K (with 4K we get 1-2ms performance).
Using RAID 0 and putting the VM over the host with the disk object the perfomance is around 5ms with the same test.

Network is 10G, Active/Passive with MTU 1500

What we have not tried is to disable checksum.

I think that this cluster can't manage 64K very well.

I think that disabling checksum could help me to achieve a better results, maybe to get more in the 5-6ms range, because the main problem is on writes. So my question is: what's worst case scenario without checksum?
I know that there are other systems that doens't use checksum at all but are production certified, so is this really needed?

Other 2 types of improvement could be:

- MTU 9000 on VSAN vmkernel

- LAG on VSAN ports

Could any of these 2 improve latencies, or will they improve mainly CPU utilization but nothing regarding write latencies because these are limited by slow cache disks?

Thanks

Manuel

PS: I have just tried on another cluster, 2 nodes but with very fast SAS disks. Also in this cluster, the result with diskspd are 5.5ms with RAID1 and 4K blocks, and 3.8ms with RAID1 3 stripes and 64k blocks. So better but considering 3.8ms on 2 Node SAS cluster, maybe around 7ms on 4 nodes SATA cluster could be right...

[depping]

are you using dedupe/compression at this point?

and yes, latency will probably go down when you disable checksumming, but I would HIGHLY recommend against disabling it as it could have a serious impact on recoverability of the data if it is corrupted in any shape or form.

[ManuelDB]

no dedupe or compression.

So if checksum is mandadory, my last resource is the network tweaking with LAG + MTU9000.

From your experience, will this help in latency or only in IOPS and mainly in reads (where the bandwidth requirements are much higher)?

[paudieo]

So going to MTU 9000K for medium to large block workloads can give you up to 10 - 15% reduction on observed latency and a similar increase in IOPS , thus more throughput

NOTE:- This is assuming the network itself is not already  bottlenecked or indeed if you are not disk performance bound already

One other thing you need to be cognisant of is the Power Management policy used on the hosts in the vSAN cluster
Make sure you are using the most performant policy for vSAN , e..g. static high performance 

I am curious why your partner is sticking with 6.7 U3, end of general support for 6.7 is Oct 2022

Going to 7.0 will help in general went it comes to performance of vSAN,
have a read off this post 

https://blogs.vmware.com/virtualblocks/2020/09/30/performance-improvements-in-vsan-7-u1/ 

Namely we now have multiple receive threads in VMkernel NIC (vmknic) and we have added further optimizations on some of the internal layers of vSAN

Hope this helps