For Mellanox ethernet controller ConnectX-3/ConnectX-3, VMware Knowledge Base describes to use driver "nmlx4_en 3.16.11.10 (6.5 driver) or new releases (6.7 driver)".
The nmlx4_en 3.16.11.10 release is for 6.5 only and can be found at https://my.vmware.com/group/vmware/details?downloadGroup=DT-ESXI65-MELLANOX-NMLX4-EN-3161110&product...
According https://www.vmware.com/resources/compatibility/search.php?deviceCategory=io&details=1&partner=55&rel... the MT27500 Family is builtin supported with ESXi 6.7U3 (with a newer release nmlx4_en version 3.17.13.1-1vmw).
As there is no standalone ESXi 6.7 driver, which release for "or new releases (6.7 driver)" is officially supported for ESXi 6.7 non-U3 releases?
the driver-version 3.17.13.1 is integrated in the 6.7u2 and 6.7u3 ISO-Image
Here's the lspci output:
9365:00:02.0 Ethernet controller: Mellanox Technologies MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function]
b260:00:02.0 Ethernet controller: Mellanox Technologies MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function]
I used the MLNX-NATIVE-ESX-ConnectX-3_3.16.11.10-10EM-650.0.0.4598673-offline_bundle-12539849.zip but it fails on 6.5 and 6.7. The builtin driver in 6.7U3 isn't recognized, too. It's a study work / homelab setup. ESXi installation stops with no network adapters were detected.
Can you post the exact PCI IDs please?
Run
lspci -p | grep 9365:00:02.0
lspci -p | grep b260:00:02.0
to find out.
Hi,
Still no clue. Here's the output:
lspci -nn -v | grep Mellanox
82d1:00:02.0 Ethernet controller [0200]: Mellanox Technologies MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function] [15b3:1004]
Subsystem: Mellanox Technologies Device [15b3:61b0]
9832:00:02.0 Ethernet controller [0200]: Mellanox Technologies MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function] [15b3:1004]
Subsystem: Mellanox Technologies Device [15b3:61b0]
For ESXi65 the offline bundle Download VMware vSphere contains in \metadata\vibs the nmlx4core file with PCIID 15b31004.
Bundling the esxi ISO on a windows box worked!
# Download and store MLNX-NATIVE-ESX-ConnectX-3_3.16.11.10-10EM-650.0.0.4598673-offline_bundle-12539849.zip in j:\driver-offline-bundle
cd j:
j:\ESXi-Customizer-PS-v2.6.0.ps1 -ozip -v65
j:\ESXi-Customizer-PS-v2.6.0.ps1 -izip j:\ESXi-6.5.0-20191203001-standard.zip -v65 -pkgDir j:\driver-offline-bundle
So far the setup throws no network adapters detected in all tests ESXi65u3, ESXi67u3 or with vanilla ESXi67u3 ISO.
Found kb VMware Knowledge Base. Strange...
The same exposure of Mellanox hardware issue occurs on Photon OS as well. Having installed the Mellanox Firmware Tools (MFT), mst status throws 'no MST devices were found'.
mst start
Starting MST (Mellanox Software Tools) driver set
Loading MST PCI module - Success
Loading MST PCI configuration module - Success
Create devices
Unloading MST PCI module (unused) - Success
Unloading MST PCI configuration module (unused) - Success
mst status -v
MST modules:
------------
MST PCI module is not loaded
MST PCI configuration module is not loaded
No MST devices were found or MST modules are not loaded.
You may need to run 'mst start' to load MST modules.
/dev/mst is empty, nevertheless lsipci -nnn presents
aca4:00:02.0 Ethernet controller [0200]: Mellanox Technologies MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function] [15b3:1004]
b75a:00:02.0 Ethernet controller [0200]: Mellanox Technologies MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function] [15b3:1004]
The content in /sys/bus/pci/devices/b75a\:00\:02.0/ is
ari_enabled current_link_speed driver local_cpus modalias pools resource subsystem_device
broken_parity_status current_link_width driver_override max_link_speed msi_bus power resource2 subsystem_vendor
class d3cold_allowed enable max_link_width msi_irqs remove resource2_wc uevent
config device irq mlx4_port1 net rescan revision vendor
consistent_dma_mask_bits dma_mask_bits local_cpulist mlx4_port1_mtu numa_node reset subsystem
It seems the links are down
cat /sys/bus/pci/devices/b75a\:00\:02.0/current_link_width
0
cat /sys/bus/pci/devices/b75a\:00\:02.0/current_link_speed
Unknown speed
The underlying hardware of the study lab is a MS Azure setup. Accelerated Networking, for the network adapter presentation, and IP forwarding, aka the prerequirement ESXi promiscuous mode for nested virtualization, both setting are enabled.
According to the findings and the vendor link https://www.mellanox.com/page/products_dyn?product_family=29&mtag=vmware_driver (click on View the list of the latest VMware driver version for Mellanox products) gives the answer: the nic type ConnectX-3®/Pro is not supported on any VMware ESXi version for SR-IOV Ethernet.
The Azure Standard_DS3_v2 offering I've been using includes the accelerated networking feature, but exposes ConnectX-3®/Pro only.
the driver-version 3.17.13.1 is integrated in the 6.7u2 and 6.7u3 ISO-Image
This driver has the same issue as the other drivers. I can get it to stop responding under the same conditions everytime. That is run a Veam Zip backup backing up 4 VM's over the 10GBe connection which has the Management and vMotion connection.
Best fix I can see is to change network cards to Intel which is what I am doing.