VSAN 7.0 U2 poor performance

[ManivelR]

Hi Team,

We have a query here and like to get your valuable feedback as soon as possible.

We are running a 3-node VSAN cluster with 7.0 U2.

Specs:

Dell PE with 2 sockets(each server) and each socket has 20 cores. Intel(R) Xeon(R) Gold 6148 CPU @ 2.40GHz

Memory is 512 GB per node.

VSAN has 2 disk group per node. Each disk group has 1 * 1 TB enterprise NVME & 2 *4 TB enterprise SSD disks

VSAN has 2* 40 G Mellanox NIC cards.

Dell HBA 330 mini cards

While assessing the performance(ex 4k block size and other sizes), using the HCI benchmark with vdbench, we see very low throughput. It’s less than 800 MB/s as per the below benchmark report.

We use 12 SSD capacity disks from all the 6 disk groups and each SSD throughput is more than 1000 MB/s as per the Samsung message.

Questions:

1)How to address this issue?

2) Can we get the ideal value from your environment to compare it?

3) How to check the disk performance from the guest OS? Can we use ATTO,Crystal disk etc..?

Test Case Name:vdb-8vmdk-100ws-4k-50rdpct-75randompct-4threads-1666762968

Thanks,

Raj

[IRIX201110141]

I dont call 200k IOPS bad.  Wanna see huge MB/s? Increase the blocksize and try again.

[ManivelR]

Ok. 

My concern is about throughout. 

Vsan uses default 4k throughout. 

I see, the write performance is bad from both hci benchmark results plus guest os write performance. 

[Tibmeister]

So you are pushing 6Gbit/s overall, not too bad, and with the 200K IOPS, that's not bad either.  Couple of things to consider.  Is the HBA firmware up to date?  Is the NIC firmware up to date?  Are you running Jumbo Frames?  What does the network switch utilization look like?  Are you running at 40Gb on the network or 10Gb?

I have found that the network can be the biggest single bottleneck for a vSAN infrastructure.

[ManivelR]

Thanks for your valuable inputs.

yes.I got your point.

1) HBA is not up to date and going to upgrade and test it.

2) mellanox NIC also needs to be upgraded.

3) We are using 2 * 40 G Mellanox NIC cards with LB(port id).MTU is default 1500.

Regards,

Raj

[Tibmeister]

We are using 2 * 40 G Mellanox NIC cards with LB(port id).MTU is default 1500.

Using 40G cards doesn't explain the connectivity.  Are these connected to 10G ports or true 40G ports?  Also, I would enable Jumbo Frames for the storage traffic, but first, upgrade the firmware on the NIC and the HBA.  Remember, in vSAN, the NIC is also an HBA from the perspective of storage traffic.

[ManivelR]

Yes,We are using 40G switches and 40G NICs connected with 40G ports only.
We have not set the MTU as 9000 yet.

I will upgrade the HBA and NIC and confirm back.

Thanks for your valuable inputs.

Regards,

Raj

[ManuelDB]

also try with a simple Active/Passive network configuration for VSAN interfaces and be sure that all active interfaces end on the same physical switch, in order to be sure that extraswitch connections are not killing your performance