VMTN

Nexus 55xx used a VoQ buffer system. Basically the (limited) buffer isn't split across all the ports in a shared pool. There's also very few queues in general per port.

It's not a great switch for high performance operations, and I believe CIsco abandoned that VoQ system in the 56xx.

If you want a deep buffer Cisco switch, get a C36180YC-R 8GB of buffer allocation for the ports.

The 55xx is a puny 470 for the default traffic class (it's 640KB but stuff by default is reserved for FCoE and other stuff).

The 3548 is interesting. IT can achieve some crazy low port to port lat, but that's really only a big deal with RDMA traffic. Might help in the future but not now.

We recently had a similar issue on a 4-node vSAN cluster and it turned out to be the x710 NICs. Switched all 4 nodes to Broadcom and latency is gone.

Hi John,

Thanks for your switch recommendation. We will care them in next investment.

By the way second recommendation of Cisco related to my case is to increase connected ports to backbone switches. Because my servers connected to Nexus, Nexus connected to Catalyst6500 with 2 ports in port-channel. these 2 ports are congested ports, and so it is applying back pressure to end devices.

We will increase the count of ports or replace the switches.

Tolga,

What model NICs in your servers?

Hi,

My NIC model installed on servers is

Model : Synergy 3820C 10/20Gb CNA

Firmware : 7.13.109.0 BC: 7.15.24

Driver : qfle3 - 1.0.77.2

8 nodes are running in streched VSAN cluster.

Nowadays I am planing to update them to 6.7 U2 and the latest HPE Synergy SPP bundle.

Hi Tolga,

Be aware with the latest SPP-2019.03 for HPE Synergy.

It is not completely certified for vSAN right now, only NVMe configurations.

If you are using SAS disks... don´t upgrade yet.

The vSAN certification for the new Smart Array Firmware 1.98 and the new driver is not completed right now.

BR/JO

Hi Johannes,

Thanks for your update.

I am in contact with VMware now.

My HPE Smart Array Card is not certified on Vmware VSAN Compatibility Guide for new firmware.

I am following up, after it is approval by VMware, I will apply immediately the latest HPE Synergy bundle to my all Synergy environment.

Model : P204i-c

Approved firmware is 1.65 for VSAN.