8 Replies Latest reply on Aug 11, 2016 4:13 AM by EcoBassam

    Problem with the order of interfaces using vmxnet3

    jguilloux Lurker

       

      Hello everybody,

       

       

      On my ESX server 4.0, I have a Linux Debian virtual host with using 4 interfaces.

       

       

      Problem

       

       

       

      Interfaces using a vmxnet3 driver are not ordered correctly (ie = in comparison to what is configured in vmx file) on my Linux system.

       

       

       

       

       

       

      Observations

       

       

       

      Interfaces are named in the vmx file: ethernet0, ethernet1, and ethernet2 ethernet3.

       

      At beginning of my virtual machine, in vmware.log file from my virtual machine (on ESX datastore), I see MAC addresses are correctly associated with Ethernet interfaces, as they have been configured in my vmx file :

       

       

      Jun 15 14:30:40.436: vmx| Ethernet0 MAC Address: 00:50:56:09:30:22

      Jun 15 14:30:40.442: vmx| Ethernet1 MAC Address: 00:50:56:09:30:23

      Jun 15 14:30:40.448: vmx| Ethernet2 MAC Address: 00:50:56:09:30:24

      Jun 15 14:30:40.455: vmx| Ethernet3 MAC Address: 00:50:56:09:30:25

       

       

      Then comes the moment where these interfaces are "passed" to the virtual machine and it's there where the problem occurs :

      Jun 15 14:30:57.596: vcpu-0| Ethernet2 MAC Address: 00:50:56:09:30:24

      Jun 15 14:30:57.674: vcpu-0| Ethernet3 MAC Address: 00:50:56:09:30:25

      Jun 15 14:30:57.751: vcpu-0| Ethernet0 MAC Address: 00:50:56:09:30:22

      Jun 15 14:30:57.829: vcpu-0| Ethernet1 MAC Address: 00:50:56:09:30:23

       

      • MAC addresses match with good ethernet interfaces : OK

      • But the order of presentation is not correct: eth2 will be presented first, then eth3, then eth0 and finally eth1

       

      As a Linux system appoint interfaces in according to their order of presentation to the kernel, here, we will have:

      - eth0 (linux) will be eth2 (vmware) : 00:50:56:09:30:24

      - eth1 (linux) will be eth3 (vmware) : 00:50:56:09:30:25

      - eth2 (linux) will be eth0 (vmware) : 00:50:56:09:30:22

      - eth3 (linux) will be eth1 (vmware) : 00:50:56:09:30:23

       

       

       

      Analyzes

       

      I found why the vmware server presents the interfaces in a specific order: it looks the "pciSlotNumber".

       

      ESX server presents interfaces that have the smallest pciSlotNumber in first. In my vmx file, I have this:

      ethernet0.pciSlotNumber = "224"

      ethernet1.pciSlotNumber = "256"

      ethernet2.pciSlotNumber = "160"

      ethernet3.pciSlotNumber = "192"

      It presents in first ethernet2, then ethernet3, then ethernet0 and finally ethernet1.

       

      If I change (manually, directly in vmx file) in a "correct order", the "pciSlotNumber", then, I have a Linux system identical at my vmx file

       

       

      ethernet0.pciSlotNumber = "160"

      ethernet1.pciSlotNumber = "192"

      ethernet2.pciSlotNumber = "224"

      ethernet3.pciSlotNumber = "256"

      =>

      - eth0 (linux) will be eth0 (vmware) : 00:50:56:09:30:22

      - eth1 (linux) will be eth1 (vmware) : 00:50:56:09:30:23

      - eth2 (linux) will be eth2 (vmware) : 00:50:56:09:30:24

      - eth3 (linux) will be eth3 (vmware) : 00:50:56:09:30:25

       

       

       

       

       

      Details

       

      pciSlotNumber are not created when creating ethernet interfaces, but only when you start the virtual machine.

       

       

      This leads to following observations :

       

      • If you create 4 interfaces, at first starting, the pciSlotNumber will allocate in ascending order: therefore, all interfaces are in good order for this first starting.

      • As long as you do not add or not delete an interface: OK, the order stays correct.

      • If you remove interfaces in order to recreate (typically what one does when one wants to change drivers: e1000 => vmxnet3), then order of allocation of pciSlotNumber will change, because vmware server takes a new number of pciSlotNumber if nothing is available. Or so, vmware may take an old pciSlotNumber became free to assign it to the new interface.

       

      In short, with this mechanism, we have no control over the order of the interfaces on a Linux system !

       

      This behavior is specific to the driver vmxnet3: it does not do that with E1000 or vmxnet2