Dorilevy
Enthusiast
Enthusiast

Rhel 6 nic order with 4 network adapter or more

Hi,

I'm having problem with NIC order on RHEL 6 VM.

When creating 4 NIC's or more, the RHEL will map the last NIC in order to eth0. I.E

"Network Adapater 1" is mapped to eth1

"Network Adapater 2" is mapped to eth3

"Network Adapater 3" is mapped to eth3

"Network Adapater 4" is mapped to eth0

( Note - i tired deleting the  /etc/udev/rules.d/70-persistent-net.rules file & rebooting the VM. This did not solve the problem.)

On VM with only 3 NIC's or less this does not happen.

When i modified the VM and removed the 4th network adapter, deleted the 70-persistent-net.rules) and rebooted, the issue was solved. But when i add it back again, it returned.

I'm using vSphere 5.1, and the VM is installed with vmware-tools 9.0.0

Also - all nic's are vmxnet3 devices.

Does anyone know this issue?

Regards,

Dori

Tags (4)
7 Replies
muddin
Contributor
Contributor

Have you chaeck your NIC MAC address?

Is there any duplications?

0 Kudos
Dorilevy
Enthusiast
Enthusiast

Hi muddin,

Thanks for the reply.

The MAC are managed by vCenter and no there is no duplicate.

I can also say now that it only happens when using vmxnet3 nic's (when using e1000 nics there is no problem).

We found out the rhel6 will map the nics according to bus ID order. When creating a VM with vmxnet3 or adding a vmxnet3 nic to vm the bus ID created is not consistent. While using the e1000 it is.

Regards,

Dori

0 Kudos
Dorilevy
Enthusiast
Enthusiast

Hi All,

After playing with it a bit i think there is a problem with vmxnet3 devices. I found a work around for this, but first let me explain the problem:

Currently the RHEL6 is using the /etc/udev/rules.d/70-persistent-net.rules to determine the naming of eth's. if this file is missing on boot it will be re-created, and the order of the NIC's will be decided by there BUS ID.

When i checked e1000 virtual NIC's i found out the bus ID assigned is sequential - when a new NIC is added, it will get the next availble bus ID. While in vmxnet3 - it is not sequential. Here is what happen when i add a new nic to a 3 nic's system:

3 NICS:
-------------
[root@dscs1 ~]# lspci | grep -i eth
0b:00.0 Ethernet controller: VMware VMXNET3 Ethernet Controller (rev 01)
13:00.0 Ethernet controller: VMware VMXNET3 Ethernet Controller (rev 01)
1b:00.0 Ethernet controller: VMware VMXNET3 Ethernet Controller (rev 01) --> NEW
4 NICS:
-------------
[root@dscs1 ~]# lspci | grep -i eth
04:00.0 Ethernet controller: VMware VMXNET3 Ethernet Controller (rev 01) --> NEW
0b:00.0 Ethernet controller: VMware VMXNET3 Ethernet Controller (rev 01)
13:00.0 Ethernet controller: VMware VMXNET3 Ethernet Controller (rev 01)
1b:00.0 Ethernet controller: VMware VMXNET3 Ethernet Controller (rev 01)

My soultion is based on that the even if the vmxnet3 bus ID is not sequential, it is allways the same - meaning any new/cloned system will allways get the same BUS ID.

So - we created a new udev rules file for RHEL6, and set it to process before the 70-persistent-net.rules. Instead of using the macaddress to identify the NIC, i use the BUS ID.

This is how it looks like:

--------------------------------------------

# cat /etc/udev/rules.d/69-vmxnet3-net.rules

SUBSYSTEM=="net", ACTION=="add", KERNELS=="0000:0b:00.0" DRIVERS=="vmxnet3", ATTR{type}=="1", KERNEL=="eth*", NAME="eth0"

SUBSYSTEM=="net", ACTION=="add", KERNELS=="0000:13:00.0" DRIVERS=="vmxnet3", ATTR{type}=="1", KERNEL=="eth*", NAME="eth1"

SUBSYSTEM=="net", ACTION=="add", KERNELS=="0000:1b:00.0" DRIVERS=="vmxnet3", ATTR{type}=="1", KERNEL=="eth*", NAME="eth2"

SUBSYSTEM=="net", ACTION=="add", KERNELS=="0000:04:00.0" DRIVERS=="vmxnet3", ATTR{type}=="1", KERNEL=="eth*", NAME="eth3"

SUBSYSTEM=="net", ACTION=="add", KERNELS=="0000:0c:00.0" DRIVERS=="vmxnet3", ATTR{type}=="1", KERNEL=="eth*", NAME="eth4"

SUBSYSTEM=="net", ACTION=="add", KERNELS=="0000:14:00.0" DRIVERS=="vmxnet3", ATTR{type}=="1", KERNEL=="eth*", NAME="eth5"

SUBSYSTEM=="net", ACTION=="add", KERNELS=="0000:1c:00.0" DRIVERS=="vmxnet3", ATTR{type}=="1", KERNEL=="eth*", NAME="eth6"

SUBSYSTEM=="net", ACTION=="add", KERNELS=="0000:05:00.0" DRIVERS=="vmxnet3", ATTR{type}=="1", KERNEL=="eth*", NAME="eth7"

---------------------------------------------

As you can see there is no MAC address. Instead it uses the bus ID ("KERNELS") as key. It is also applies to vmxnet3 drivers only.

This solved the issue to me - it also solved the problem with removing the 70-persistent-net.rules after/before any clone.

Some notes to consider, if you want to use it:

1. This file us limited up to 8 nics - if you need more then you should add new lines with the bus ID.

2. Since it only apply to vmxnet3 nics - if you set your system with e1000 nics, this rule will not apply on it, and the regular 70-persistent-net.rules will be created to map it.

3. I assume each system is using one type of nics (either e1000 OR vmxnet3), not both toghter - on our systems we don't use "mix" of e1000 and vmxnet3 - if you do - you should test how it behaves before - on your own risk.

Hope this post will help someone.

Best Regards,

Dori


zeljkon
Contributor
Contributor

Hi Dorilevy,

have you had any progress in dealing with the vmxnet3 NIC ordering issue besides the workaround you mentioned in your last post?

I'm having trouble finding anything about it more recently than your posts.

0 Kudos
dking1525
Contributor
Contributor

We had this exact problem and fixed it as explained in comment #3.

However, the most recent upgrade to RHEL6.6 (kernel-2.6.32-504.3.3.el6) reintroduced this issue and seems to be immune modifying.


Looks like Red Hat is blaming VMware How to fix "Device eth0 has different MAC address than expected, ignoring." error ... RedHat recommends reinstalling VMware tools and if that doesn't help open a case with VMware.


I haven't seen anything specific in VMware tools that defines NIC ordering. 

0 Kudos
SamLiddicott
Contributor
Contributor

Under vmware there are often these files (possibly you need the vmware kernel modules):

/sys/bus/pci/devices/0000:*/label
/sys/bus/pci/devices/0000:*/acpi_index
and this synmlink:
/sys/bus/pci/devices/0000:*/firmware_node

The file label (which might only exist for VMWare Workstation) contains a line of text like: Ethernet0, Ethernet1, etc, and are numbered according to the original number in the OVF file.

The file acpi_index (possibly also for VMWare Workstation) has a number (long int probably), and the numbers when sorted, match the original order of the interfaces in the OVF file.

The filename part of the destination of the symlink firmware_node (on ESXi 6) also collates in the same order as the interfaces in the original OVF file. You can read that with readlink under the shell.

The most useful is probably the label file as it would be simple to extract the numeric part of the name and use it as a device name, but it doesn't always exist.

NOTE: it appears that firmware_node only works as a sort key for up to 8 devices, after which it the sort order doesn't hold true any more

Maybe this will be of use: VMware Knowledge Base

0 Kudos
NoTerminal
Contributor
Contributor

I wrote a small bash script to help solve this problem. You feed the script an index number representing the order in which the devices appear in the vmware console and the script responds with the name of the device that corresponds with that index number.  If I find any problems with it I'll keep a gist updated here: https://gist.github.com/logicalmethods/da1d89f8cfcfd58ad365e6ea98461f16

#! /bin/bash

###
# Written by Alex Speaks
# Takes the position in which a NIC appears in the vmware console and returns the ethernet device that corresponds to that position.
# Position numbers start at 0
# You can think of this as the old-style ethX number

NIC_INDEX=$1

for dir in /sys/bus/pci/devices/*/
do
    if [ -f $dir/label ] && [ $(cat ${dir}/label|tr -d '\n') == "Ethernet${NIC_INDEX}" ]
    then
        ls $dir/net/
    fi
done

0 Kudos