VMware Cloud Community
MakerOfGames
Enthusiast
Enthusiast
Jump to solution

Issues Installing Mellanox ConnectX-3 40/56 GbE InfiniBand QSFP NIC under ESXi 5.5 (u3)

I have a Dual-Port ConnectX-3 Pro EN 40/56 GbE QSFP+ NIC (MCX314A-BCCT)

I have been struggling for days now to get this card working.

To explain the topology, I have two other Hosts, with an Infiniband Switch between them.

Both of the other Hosts are plugged into the InfiniBand Switch via Passive QSFP Copper Cables.

The other two Hosts and the Mellanox Cards in them are working fine.

Example:

NIC-1.PNG

I am now trying to add a third host, with this NIC in it.

The NIC is VMware-certified, and I am using the VMware-provided Driver specifically made for it.

(VMware Compatibility Guide - I/O Device Search​)

But the NIC Ports will not connect and link when plugged in to the Switch.

I tried replacing the Card. I tried replacing the Cables with known, tested cables.

I have tried changing the port_type_array to Ethernet, and to Infiniband.

When the card's two ports are directly plugged into one another (while the NIC in Ethernet Mode), Link is active.

But the card has to be in InfiniBand mode because it is an InfiniBand Switch on the other end, and the other two Hosts are running in InfiniBand mode.

I notice that the new NIC is using a different Driver:

NIC-2.PNG

The existing, working NICs are using the "ib_ipoib" driver, and the new Host's NIC is using the "mlx4_en" driver.

I don't know of any esxcli or esxcfg command to specify or change the Driver that's being used for the card, and switch it to ib_ipoib.

I've tried installing older drivers, going all the way back to the ESXi 5.0 Drivers, but nothing I did -- for days -- got those ports to energize.

The InfiniBand card's ports remain in "Down" state.

NIC-3.PNG

I am at my wits end at this point, and so I am reaching out to the community here.

Does anyone have any suggestions on how I can get this card working?

Reply
0 Kudos
1 Solution

Accepted Solutions
MakerOfGames
Enthusiast
Enthusiast
Jump to solution

Mellanox cards are dual mode -> they are both infiniband and ethernet capable

Yes, that's what I thought. It's also alluded to in the card's documentation.

However, I am beginning to suspect that the "EN" model cards only support Ethernet.

If the card only supports Ethernet mode, it won't work with the InfiniBand Switch.

The Manual for the MCX314A-BCCT only provides GBE throughput speeds, not IB speeds, as shown below:

MCX314A-BCCT.PNG

I think the references to InfiniBand in the manual are just lingering hold-overs from whoever wrote the manual at Mellanox using an IB product's manual as a starting point.

When I tried to use mlxconfig to change the Port Settings, it gave an error, saying the Firmware does not support that command.

If this is the case, instead of an MCX314A-BCCT, I need a MCX354A-QCBT.

That this card's manual clearly references InfiniBand and Ethernet in the Data Transmission Rates.

MCX354A-QCBT.PNG

View solution in original post

Reply
0 Kudos
9 Replies
MakerOfGames
Enthusiast
Enthusiast
Jump to solution

Just in case anyone asks, as part of my troubleshooting before I reached out here, the NIC was updated to the very latest Firmware version.

NIC-4.PNG

Reply
0 Kudos
MakerOfGames
Enthusiast
Enthusiast
Jump to solution

Also, the InfiniBand switch is a 36-Port 40 GbE Mellanox Voltaire Grid Director 4036 (VLT-30015-IBM).

gd-4036-1_2.gif

Reply
0 Kudos
peetz
Leadership
Leadership
Jump to solution

Greetings,

can you try to use the latest Mellanox driver 2.4.0: VMware InfiniBand Driver: Firmware - Driver Compatibility Matrix ?

And please note: vSphere 5.5 is out of support since Sep 19 2018 ... It's really time to upgrade now.

- Andreas

Twitter: @VFrontDe, @ESXiPatches | https://esxi-patches.v-front.de | https://vibsdepot.v-front.de
Reply
0 Kudos
MakerOfGames
Enthusiast
Enthusiast
Jump to solution

Can you try to use the latest Mellanox driver 2.4.0

Hi! Thanks for responding on a Saturday.

As can be seen in my second post above, I am already currently using the latest Driver, version 2.4.0.

Also the card has been upgraded to the latest Firmware version.

Reply
0 Kudos
MakerOfGames
Enthusiast
Enthusiast
Jump to solution

Shouldn't it be using the "mlx4_ib" Driver, and not the "mlx4_en" Driver?

Is there a way to change which Driver it is using?

Reply
0 Kudos
ZibiM
Enthusiast
Enthusiast
Jump to solution

Mellanox cards are dual mode -> they are both infiniband and ethernet capable.

You can switch between the modes using the driver provided utility.

Please check whether your card is not working in the ethernet mode and if this is the case - try to switch it to infiniband.

Reply
0 Kudos
MakerOfGames
Enthusiast
Enthusiast
Jump to solution

Mellanox cards are dual mode -> they are both infiniband and ethernet capable

Yes, that's what I thought. It's also alluded to in the card's documentation.

However, I am beginning to suspect that the "EN" model cards only support Ethernet.

If the card only supports Ethernet mode, it won't work with the InfiniBand Switch.

The Manual for the MCX314A-BCCT only provides GBE throughput speeds, not IB speeds, as shown below:

MCX314A-BCCT.PNG

I think the references to InfiniBand in the manual are just lingering hold-overs from whoever wrote the manual at Mellanox using an IB product's manual as a starting point.

When I tried to use mlxconfig to change the Port Settings, it gave an error, saying the Firmware does not support that command.

If this is the case, instead of an MCX314A-BCCT, I need a MCX354A-QCBT.

That this card's manual clearly references InfiniBand and Ethernet in the Data Transmission Rates.

MCX354A-QCBT.PNG

Reply
0 Kudos
ZibiM
Enthusiast
Enthusiast
Jump to solution

Reply
0 Kudos
MakerOfGames
Enthusiast
Enthusiast
Jump to solution

That was it. Problem solved.

Reply
0 Kudos