vmnics renumbered after replacing a DIMM

[kurtwest]

I have had this happen two times on two different DELL 6850's. I have vmnics 0 - 5 on my hosts normally. After replacing a bad DIMM and rebooting the host vmnic1 & 2 are renamed to vmnic6 & 7. This causes my vSwitch1 to lose one of it's uplinks which was vmnic2. I renamed the vmnic6 & 7 in the esx.conf back to the original names and reboot the server. The vmnic names are back to normal and the vmnic2 uplink shows back up in vSwitch1. The PCI numbering apears to be unchanged as far as I can tell from the esx.conf. My question is waht causes this? I am not adding hardware or even moving hardware. I simply replaced a DIMM.

[esiebert7625 · Accepted solution]

Good timing, I just wrote a doc on that very subject. On some servers simply removing a card and plugging it back in can trigger this depending on your system BIOS.

How can I recover from vmnic re-numbering on ESX servers? - If you add or replace NIC's in your ESX server you may run into a situation where ESX will re-number your NIC's causing your already configured networking to no longer work. This can also happen if you make any other hardware changes on your ESX server like replacing a fiber channel card. Your servers system BIOS will sometimes change the physical topology of hardware devices, when changes are made it can subsequently change the order in which devices are discovered. When this happens ESX will see the NIC's as new devices and assign them new NIC numbers. For example if you replace a 2-port NIC with a 4-port NIC, your original NIC's were probably numbered vmnic0 and vmnic1. Once ESX sees the new NIC it will be numbered vmnic2, vmnic3, vmnic4 and vmnic5 because vmnic0 and vmnic1 already exist in the ESX configuration (even though they are no longer physically present). Because your existing vswitches and service console are linked to vmnic0 and vmnic1 they will no longer work. There are two ways to recover from this, my preference is method 1 which is generally easier then re-configuring your vswitches. Also method 1 preserves your existing NIC numbering if you don't want your NIC's starting at vmnic2.

Method 1 - Editing the esx.conf file

• Login to the Service Console

• Check your existing NIC numbering by typing ‘esxcfg-nics -l'

• Type ‘cd /etc/vmware' to change to the correct directory

• Type ‘cp esx.conf esx.con.bak' to make a backup of this file as it is a critical configuration file for ESX

• Type ‘nano esx.conf' to open the file for editing

• Type CTRL-W and then enter ‘vmnic2' to search for the new first NIC

• Change ‘vmnic2' to ‘vmnic0'

• Change the subsequent NIC's from ‘vmnic3' to ‘vmnic1', ‘vmnic4' to ‘vmnic2' and ‘vmnic5' to ‘vmnic3'

• Type CTRL-O to save the file

• Type CTRL-X to exit the Nano editor

• Shutdown and restart the ESX server, when the server comes back up the NIC's will be numbered vmnic0 - vmnic3, verify this by typing ‘esxcfg-nics -l'

Method 2 - Modify your vswitch configuration

• Login to the Service Console

• Check your existing NIC numbering by typing ‘esxcfg-nics -l'

• Check your current vswitch configuration by typing ‘esxcfg-vswitch -l' , note which NIC's are assigned to which vswitches (uplink column)

• Remove the old NIC's that have been renamed by typing ‘esxcfg-vswitch -U ', ie. esxcfg-vswitch -U vmnic0 vSwitch1

• Add the new NIC's with the correct names by typing ‘esxcfg-vswitch -L ', ie. esxcfg-vswitch -L vmnic2 vSwitch1

• Repeat this process for any additional NIC's. Once you have the vswitch that contains the Service Console corrected you can log in via the VI Client and correct t he other vswitches that way also.

• Your newly renamed NIC's should now be assigned to the original vswitches and your networking should now work again.

Also see these kb articles for more info on this:

How To Configure Networking from the Service Console Command Line -

VI Client loses connectivity to the ESX Server Host after you add a new network adapter -

-=-=-=-=-=-=-=-=-=-=-==-=-=-=-=-=-=-=-=-=-=-=-

Visit my website:

-=-=-=-=-=-=-=-=-=-=-==-=-=-=-=-=-=-=-=-=-=-=-

[kurtwest]

I didn't actually do the DIMM replacement myself. I will ask the guy that did it what he did exactly. We have a seperate team that handles that. I do know this server has memory boards with DIMMs on them and the memory board plugs into PCI slots. The memory board would had to of been pulled out to replace the DIMM. That reseating of the memory board must have been enough to trigger the renumbering of the vmnics. I am suprised that reseating a memory board would make my vmnics renumber, but that is certianly what happend.

Thank you and good job on the article.

[esiebert7625]

Apparently any type of re-seating of hardware will trigger this in some servers, here's a couple more links that talk about it...

Vmnic renumbering - http://www.vmware.com/community/thread.jspa?messageID=477265 and http://www.vmware.com/community/thread.jspa?messageID=551433

-=-=-=-=-=-=-=-=-=-=-==-=-=-=-=-=-=-=-=-=-=-=-

Visit my website:

-=-=-=-=-=-=-=-=-=-=-==-=-=-=-=-=-=-=-=-=-=-=-

[bazza52]

This issue happens all the time, even when upgrading firmware, as ESX detects a hardware modification.

Using esxcfg-vswitch -U doesn't seem to work as the uplinks aren't actually there. So if you add the new vmnic numbers back into the vSwitch, everything works fine - but the esx.conf file will still be messy with the old (original) vmnic numbering. Seems like either way, to get a clean result, you need to play with esx.conf (but I have heard that VMware do not support any methods which modify esx.conf).

Surely this widespread issue is fixed in vSphere??