VMware Cloud Community
Vladimir_Lysenk
Contributor
Contributor

PSOD problem with ESXi 6.5 when I use RDMA\iSER connection

Good Afternoon,

I have faced a PSOD problem with ESXi 6.5.

Each port of Mellanox Connect x4 passes 2 virtual functions to Windows 2016 VM with StarWind HA target. Synchronization channel works through the first port of the virtual function. iSER connection to StarWind HA IMG Datastore works through virtual functions of the second port. Targets are being connected to ESXi using VMware iSCSI over RDMA (iSER) Adapter. After I have connected StarWind HA IMG Datastore to the ESXi cluster, which consist of 2 nodes, I am creating a Test VM on this Datastore. I see PSOD on both nodes in different moments of time.

IMG_20161125_121121.jpg IMG_20161125_122522.jpg

I have not found any Mellanox Connect x4 drivers for ESXi 6.5 on Mellanox website.

http://www.mellanox.com/page/products_dyn?product_family=29

11.png

Environment:

2 identical servers: Dell r720 2x Intel Xeon E5-2660, 128GB RAM, 1 HDD WD1002FAEX (RAID0) 2x Intel SSD DC S3610 480Gb (RAID0), NIC - Mellanox connect x4 100Gb 2 ports;

OS: ESXi 6.5.0 SMP Release build-4564106 Oct 26 2016;

Additional ESXi modules: iser - 7.7.7.7-1OEM.650.0.0.4240417, nmlx5-core - 1.2.3.4-1OEM.650.0.0.4240417, nmlx5-rdma - 1.2.3.4-1OEM.650.0.0.4240417, nmst - 4.4.0.44-1OEM.600.0.0.2768847, mft - 4.4.0.44-0;

Configuration StarWind Virtual Machines: Windows Server 2016 Datacenter Evavuation (4 cpu, 10GB RAM Hard disk 40gb LZ, 3 nic - Mellanox connect x4 (SR-IOV) virtual function);

Normal.png

Have you faced such problems? Does anyone has any idea what to do with this?


Reply
0 Kudos
3 Replies
time81
Contributor
Contributor

Hey,

mellanox 6.5 drivers will be released on 31.12.2016, their support told me today Smiley Happy

Im using the out-of-the-box nmlx5-core drivers from the 6.5 ISO to get my Connect-X4 running. Works fine so far.

Using HP Hardware here with PCI-E 8x but not 16x. Not sure if this is a problem :smileygrin:

But my 2 test-hosts work fine so far with 100G and the Mellanox 100G SN2700 Switch

Reply
0 Kudos
Cryptz
Enthusiast
Enthusiast

How are you configuring rdma/iser on esxi 6.5? I also have cx4 adapters but only saw the standard iscsi options.

Reply
0 Kudos
Jae-Hoon_Choi
Enthusiast
Enthusiast

Hi!

Here is my experience.

Which ESXi driver to use for SRP/iSER over IB (... | Mellanox Interconnect Community

ESXi 6.5 native driver for EDMA (NOT inbox driver) cause problem using SRP initiator.

My friend also say to me using Inbox driver that cause packet drop on PFC configured Arista switch.

I think that VMware must release ESXi 6.5 update with stable inbox driver + native driver for RDMA.

Reply
0 Kudos