VMware Cloud Community
DCasota
Expert
Expert

VMware ESXi 6.7 Driver for Mellanox Technologies MT27500/MT27520 Family

For Mellanox ethernet controller ConnectX-3/ConnectX-3, VMware Knowledge Base describes to use driver "nmlx4_en 3.16.11.10 (6.5 driver) or new releases (6.7 driver)".

The nmlx4_en 3.16.11.10 release is for 6.5 only and can be found at https://my.vmware.com/group/vmware/details?downloadGroup=DT-ESXI65-MELLANOX-NMLX4-EN-3161110&product...

According https://www.vmware.com/resources/compatibility/search.php?deviceCategory=io&details=1&partner=55&rel... the MT27500 Family is builtin supported with ESXi 6.7U3 (with a newer release nmlx4_en version 3.17.13.1-1vmw).

As there is no standalone ESXi 6.7 driver, which release for "or new releases (6.7 driver)" is officially supported for ESXi 6.7 non-U3 releases?

Reply
0 Kudos
7 Replies
berndweyand
Expert
Expert

the driver-version 3.17.13.1 is integrated in the 6.7u2 and 6.7u3 ISO-Image

Reply
0 Kudos
DCasota
Expert
Expert

Here's the lspci output:

9365:00:02.0 Ethernet controller: Mellanox Technologies MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function]

b260:00:02.0 Ethernet controller: Mellanox Technologies MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function]

I used the MLNX-NATIVE-ESX-ConnectX-3_3.16.11.10-10EM-650.0.0.4598673-offline_bundle-12539849.zip but it fails on 6.5 and 6.7. The builtin driver in 6.7U3 isn't recognized, too. It's a study work / homelab setup. ESXi installation stops with no network adapters were detected.

Reply
0 Kudos
peetz
Leadership
Leadership

Can you post the exact PCI IDs please?

Run

lspci -p | grep 9365:00:02.0

lspci -p | grep b260:00:02.0

to find out.

Twitter: @VFrontDe, @ESXiPatches | https://esxi-patches.v-front.de | https://vibsdepot.v-front.de
Reply
0 Kudos
DCasota
Expert
Expert

Hi,

Still no clue. Here's the output:

lspci -nn -v | grep Mellanox

82d1:00:02.0 Ethernet controller [0200]: Mellanox Technologies MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function] [15b3:1004]

        Subsystem: Mellanox Technologies Device [15b3:61b0]

9832:00:02.0 Ethernet controller [0200]: Mellanox Technologies MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function] [15b3:1004]

        Subsystem: Mellanox Technologies Device [15b3:61b0]

For ESXi65 the offline bundle Download VMware vSphere  contains in \metadata\vibs the nmlx4core file with PCIID 15b31004.

Bundling the esxi ISO on a windows box worked!

# Download and store MLNX-NATIVE-ESX-ConnectX-3_3.16.11.10-10EM-650.0.0.4598673-offline_bundle-12539849.zip in j:\driver-offline-bundle

cd j:

j:\ESXi-Customizer-PS-v2.6.0.ps1 -ozip -v65

j:\ESXi-Customizer-PS-v2.6.0.ps1 -izip j:\ESXi-6.5.0-20191203001-standard.zip -v65 -pkgDir j:\driver-offline-bundle

So far the setup throws no network adapters detected in all tests ESXi65u3, ESXi67u3 or with vanilla ESXi67u3 ISO.

Found kb VMware Knowledge Base. Strange...

Reply
0 Kudos
DCasota
Expert
Expert

The same exposure of Mellanox hardware issue occurs on Photon OS as well. Having installed the Mellanox Firmware Tools (MFT), mst status throws 'no MST devices were found'.

mst start

Starting MST (Mellanox Software Tools) driver set

Loading MST PCI module - Success

Loading MST PCI configuration module - Success

Create devices

Unloading MST PCI module (unused) - Success

Unloading MST PCI configuration module (unused) - Success

mst status -v

MST modules:

------------

    MST PCI module is not loaded

    MST PCI configuration module is not loaded

    No MST devices were found or MST modules are not loaded.

    You may need to run 'mst start' to load MST modules.

/dev/mst is empty, nevertheless lsipci -nnn presents

aca4:00:02.0 Ethernet controller [0200]: Mellanox Technologies MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function] [15b3:1004]

b75a:00:02.0 Ethernet controller [0200]: Mellanox Technologies MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function] [15b3:1004]

The content in /sys/bus/pci/devices/b75a\:00\:02.0/ is

ari_enabled               current_link_speed  driver           local_cpus      modalias   pools   resource      subsystem_device

broken_parity_status      current_link_width  driver_override  max_link_speed  msi_bus    power   resource2     subsystem_vendor

class                     d3cold_allowed      enable           max_link_width  msi_irqs   remove  resource2_wc  uevent

config                    device              irq              mlx4_port1      net        rescan  revision      vendor

consistent_dma_mask_bits  dma_mask_bits       local_cpulist    mlx4_port1_mtu  numa_node  reset   subsystem

It seems the links are down

cat /sys/bus/pci/devices/b75a\:00\:02.0/current_link_width

0

cat /sys/bus/pci/devices/b75a\:00\:02.0/current_link_speed

Unknown speed

The underlying hardware of the study lab is a MS Azure setup. Accelerated Networking, for the network adapter presentation, and IP forwarding, aka the prerequirement ESXi promiscuous mode for nested virtualization, both setting are enabled.

Reply
0 Kudos
DCasota
Expert
Expert

According to the findings and the vendor link https://www.mellanox.com/page/products_dyn?product_family=29&mtag=vmware_driver (click on View the list of the latest VMware driver version for Mellanox products) gives the answer: the nic type ConnectX-3®/Pro is not supported on any VMware ESXi version for SR-IOV Ethernet.

The Azure Standard_DS3_v2 offering I've been using includes the accelerated networking feature, but exposes ConnectX-3®/Pro only.

Reply
0 Kudos
veldthui
Enthusiast
Enthusiast

the driver-version 3.17.13.1 is integrated in the 6.7u2 and 6.7u3 ISO-Image

This driver has the same issue as the other drivers. I can get it to stop responding under the same conditions everytime. That is run a Veam Zip backup backing up 4 VM's over the 10GBe connection which has the Management and vMotion connection.

Best fix I can see is to change network cards to Intel which is what I am doing.

Reply
0 Kudos