VMware Cloud Community
jguilloux
Contributor
Contributor

Problem with the order of interfaces using vmxnet3

Hello everybody,

On my ESX server 4.0, I have a Linux Debian virtual host with using 4 interfaces.

Problem

Interfaces using a vmxnet3 driver are not ordered correctly (ie = in comparison to what is configured in vmx file) on my Linux system.

Observations

Interfaces are named in the vmx file: ethernet0, ethernet1, and ethernet2 ethernet3.

At beginning of my virtual machine, in vmware.log file from my virtual machine (on ESX datastore), I see MAC addresses are correctly associated with Ethernet interfaces, as they have been configured in my vmx file :

Jun 15 14:30:40.436: vmx| Ethernet0 MAC Address: 00:50:56:09:30:22

Jun 15 14:30:40.442: vmx| Ethernet1 MAC Address: 00:50:56:09:30:23

Jun 15 14:30:40.448: vmx| Ethernet2 MAC Address: 00:50:56:09:30:24

Jun 15 14:30:40.455: vmx| Ethernet3 MAC Address: 00:50:56:09:30:25

Then comes the moment where these interfaces are "passed" to the virtual machine and it's there where the problem occurs :

Jun 15 14:30:57.596: vcpu-0| Ethernet2 MAC Address: 00:50:56:09:30:24

Jun 15 14:30:57.674: vcpu-0| Ethernet3 MAC Address: 00:50:56:09:30:25

Jun 15 14:30:57.751: vcpu-0| Ethernet0 MAC Address: 00:50:56:09:30:22

Jun 15 14:30:57.829: vcpu-0| Ethernet1 MAC Address: 00:50:56:09:30:23

  • MAC addresses match with good ethernet interfaces : OK

  • But the order of presentation is not correct: eth2 will be presented first, then eth3, then eth0 and finally eth1

As a Linux system appoint interfaces in according to their order of presentation to the kernel, here, we will have:

- eth0 (linux) will be eth2 (vmware) : 00:50:56:09:30:24

- eth1 (linux) will be eth3 (vmware) : 00:50:56:09:30:25

- eth2 (linux) will be eth0 (vmware) : 00:50:56:09:30:22

- eth3 (linux) will be eth1 (vmware) : 00:50:56:09:30:23

Analyzes

I found why the vmware server presents the interfaces in a specific order: it looks the "pciSlotNumber".

ESX server presents interfaces that have the smallest pciSlotNumber in first. In my vmx file, I have this:

ethernet0.pciSlotNumber = "224"

ethernet1.pciSlotNumber = "256"

ethernet2.pciSlotNumber = "160"

ethernet3.pciSlotNumber = "192"

It presents in first ethernet2, then ethernet3, then ethernet0 and finally ethernet1.

If I change (manually, directly in vmx file) in a "correct order", the "pciSlotNumber", then, I have a Linux system identical at my vmx file

ethernet0.pciSlotNumber = "160"

ethernet1.pciSlotNumber = "192"

ethernet2.pciSlotNumber = "224"

ethernet3.pciSlotNumber = "256"

=>

- eth0 (linux) will be eth0 (vmware) : 00:50:56:09:30:22

- eth1 (linux) will be eth1 (vmware) : 00:50:56:09:30:23

- eth2 (linux) will be eth2 (vmware) : 00:50:56:09:30:24

- eth3 (linux) will be eth3 (vmware) : 00:50:56:09:30:25

Details

pciSlotNumber are not created when creating ethernet interfaces, but only when you start the virtual machine.

This leads to following observations :

  • If you create 4 interfaces, at first starting, the pciSlotNumber will allocate in ascending order: therefore, all interfaces are in good order for this first starting.

  • As long as you do not add or not delete an interface: OK, the order stays correct.

  • If you remove interfaces in order to recreate (typically what one does when one wants to change drivers: e1000 => vmxnet3), then order of allocation of pciSlotNumber will change, because vmware server takes a new number of pciSlotNumber if nothing is available. Or so, vmware may take an old pciSlotNumber became free to assign it to the new interface.

In short, with this mechanism, we have no control over the order of the interfaces on a Linux system !

This behavior is specific to the driver vmxnet3: it does not do that with E1000 or vmxnet2

Reply
0 Kudos
8 Replies
jguilloux
Contributor
Contributor

Please,

I need of your help.

Thanks

Reply
0 Kudos
jguilloux
Contributor
Contributor

Nobody has this embarrassing problem ?

It's very strange.

Zido

Reply
0 Kudos
jguilloux
Contributor
Contributor

Hello,

Nobody to answer my problem

...

Reply
0 Kudos
Fredzone
Contributor
Contributor

Hello jguilloux,

I have exactly the same problem with ESXi 4.1u2.

Did you find a solution about it?

Regards,

Fredzone

Reply
0 Kudos
vpendyal
Contributor
Contributor

Surprising there are no follow ups on this thread for this problem. This should be such a common problem that we also see. Was wondering how everyone else is dealing with it.

Reply
0 Kudos
lvaibhavt
Hot Shot
Hot Shot

Hi,

I have not seen this error or dealt with it but just wanted to verify -- have you tried changing the bind order of nic's from OS

http://unix.stackexchange.com/questions/10254/how-to-change-the-order-of-the-network-cards-eth1-eth0...

Reply
0 Kudos
vpendyal
Contributor
Contributor

Am on PTO on 7/5.

Reply
0 Kudos
EcoBassam
Enthusiast
Enthusiast

Hello,

This is an old thread, but I wanted just to mention that we encounter the same symptoms in a vSphere 5.5 environment.

Regards,

Reply
0 Kudos