No Active ports in Port Group

[manasfirst]

Hi Experts,

I am very new user of VMWARE and facing weird issue. I having a Mellanox NIC with 2 40G ports. I installed ESX 6.5 on Supermicro server.

esxcli network nic list | grep Mell

vmnic6  0000:d8:00.0  nmlx5_core  Up            Up           40000  Full    ec:0d:9a:a0:1c:ec  1500  Mellanox Technologies MT27800 Family [ConnectX-5]

vmnic7  0000:d8:00.1  nmlx5_core  Up            Up           40000  Full    ec:0d:9a:a0:1c:ed  1500  Mellanox Technologies MT27800 Family [ConnectX-5]

I enabled SRIOV for the NIC.

/opt/mellanox/bin/mlxconfig -d mt4119_pciconf0 q| grep SRIOV_EN

         SRIOV_EN                            True(1)

/opt/mellanox/bin/mlxconfig -d mt4119_pciconf0 q| grep NUM_OF_VFS

         NUM_OF_VFS                          8

I tried to create two VMs and tried to attach two VF per VM. So created 4 port groups - 2 groups have vlan 200 and other 2 has vlan 201

vmdumper -l

wid=69542 pid=-1 cfgFile="/vmfs/volumes/5c86eb1e-ffd424e8-8e10-3cfdfecd31d0/Node1/Node1.vmx" uuid="56 4d 8d 85 6a 82 be b8-83 81 ff f8 16 c8 e3 1a" displayName="Node1" vmxCartelID=69541

wid=78779 pid=-1 cfgFile="/vmfs/volumes/5c86eb1e-ffd424e8-8e10-3cfdfecd31d0/Node0/Node0.vmx" uuid="56 4d 78 13 14 85 13 bf-20 f7 49 c8 b6 22 ea b3" displayName="Node0" vmxCartelID=78778

esxcli network sriovnic vf list -n vmnic6

VF ID  Active  PCI Address     Owner World ID

-----  ------  --------------  --------------

    0    true  00000:216:00.2  78778

    1    true  00000:216:00.3  78778

    2    true  00000:216:00.4  69541

    3    true  00000:216:00.5  69541

esxcli network vswitch standard portgroup list

Name                Virtual Switch  Active Clients  VLAN ID

------------------  --------------  --------------  -------

Downlink Port1      vSwitch0                     1      200 <<<<<<<<<<<<<<<< This is attached to VM1

Downlink Port2      vSwitch0                     1      201 <<<<<<<<<<<<<<<< This is attached to VM1

Downlink Port3      vSwitch0                     0      200 <<<<<<<<<<<<<<<< This is attached to VM2

Downlink Port4      vSwitch0                     0      201 <<<<<<<<<<<<<<<< This is attached to VM2

Management Network  vSwitch0                     1     4095

VM Network          vSwitch0                     2        0

If you look at the previous output (esxcli network vswitch standard portgroup list), two port groups Downlink Port3 & Downlink Port4 does not have any Active ports........Am I missing something here ?

As a result, traffic does not reach to VM2. Any help appreciable.

[daphnissov]

Why are you enabling SRIOV for these NICs in the first place? If you're a new user to VMware, it's highly likely that you don't need to be doing this in the first place. What you're attempting to do is considered an advanced use case.

[manasfirst]

SRIOV - thats the customer need. Its not for my own test.

Even if it is a advance case, I have to achieve the same .. thats the goal.

So if you have some clue, please do share why the port group does not have any Active port.

[sk84]

I can't help you with your problem with SRIOV, because I have always been able to avoid using this feature. But I can tell you from years of experience with dozens of different VMware setups and customer needs that in 99.9% of the cases SRIOV is not necessary or the disadvantages outweigh the advantages of this feature.

The background is that there is exactly 1 reason for SRIOV: applications that require extremely low network latency. But honestly, I don't know of any application that has problems with 1ms network latency. Maybe they exist in the academic field.

Another point is that such performance sensitive applications usually have extreme CPU and memory requirements and virtualization with shared CPU and memory ressources isn't a problem there?

Furthermore, by using SRIOV you lose almost all advantages of virtualization. The list of unsupported features for SRIOV VMs is very long: no vMotion, no DRS, no high availability in case of host failures, no snapshots (so backups are also difficult), no hot-adding of hardware resources, etc.

See: SR-IOV Support

All in all, the question arises whether the customer would not be better off with a dedicated system than trying to cover this special case in a virtual environment. But in my experience there's also a good chance that he won't need this feature so badly if he's aware of all the disadvantages. 😉

I would therefore get in touch with the customer again and tell him whether he really knows what he wants and can live with the disadvantages mentioned.