VMware Communities: Message List - Renaming VMnics to match PCI Slot Numbers

Re: Renaming VMnics to match PCI Slot Numbers

livinma1 — Mon, 28 Sep 2009 19:22:40 GMT

Hey Guys --

Currently installing vsphere 4.0 and I noticed that my nics are out of order. I want the internal broadcom nic to come up as nic 0 but it is showing up has nic2.

Linux Dynasty I spent all day trying to get your script to run but since I am very new to linux it took me a bit to realize I needed to install Python and then once I downloaded Python I then needed GCC. I have spent the better half of the day trying to locate a binary for GCC but with no luck. I have not yet installed GCC which means I cannot install Python.

How can I get pass this ? What information will you like ?

Re: Renaming VMnics to match PCI Slot Numbers

keinstein — Thu, 27 Aug 2009 15:55:19 GMT

Hello,

I downloaded the script and used it, it works great, however since i'm new to the whole ks and post script arena i would like to know how could i include this script in the %post section of my ks. I need to deploy more than 40 hosts in one night and i don't want to configure them manually. i jsut want to automate as much as possible. Thanks for your help!!

-- Ken

Re: Renaming VMnics to match PCI Slot Numbers

linuxdynasty — Tue, 14 Apr 2009 21:21:45 GMT

that's all I was asking for. I just wanted to know why you have to set modules.conf. and what you are saying does makes sense. I never had to boot into linux mode on any of my esx servers so I never ran into this issue. the reason for the flaming is that you made this personal by implying that I am soliciting my site and throwing your title around as well. all I wanted to know is why and now that you explained it to me. I will research this issue. fyi... I run my site out of my own pocket and the sole purpose of the site is to share my knowledge with every one..

Re: Renaming VMnics to match PCI Slot Numbers

RichD — Tue, 14 Apr 2009 20:37:31 GMT

The bnx aliases point to the LAST NICs even though they are the on-boards.

It's important to note that modules.conf and esx.conf are two different problems, for two different boot states/kernels.

It's also important to note that the kernel in the installer has the same problem with NIC ordering, but that this is unique to that kernel and the driver loading order on the install image.

So there are three places you need to implement a workaround:

By selecting a different adapter during kickstart;

by re-ordering NICs during POST using either of the scripts in this thread, and;

by reordering modules.conf.

We wrote the make-modules_conf script to address the latter, to get the bnx's in their appropriate spot, and this consistent with nic renumeration of esx.conf (i.e. always in PCI address order, consistent with Best Practices).

make-modules_conf achieves this by parsing the PCI IDs, the driver tables provided by vmware, and building modules.conf from scratch.

I'm not sure why I'm getting flamed for pointing this out, but modules.conf needs to be fixed in order for the nic re-ordering to be complete. Of course fixing esx.conf will resolve nic ordering while running vmnix, and that is what matters 99.9% of the time. But if you reboot in linux mode you lose connectivity. make-modules_conf is the only script I am aware of that currently fixes this issue, which is why I wanted to share it with the class.

-Richard Dagenais, VCP

Re: Renaming VMnics to match PCI Slot Numbers

dconvery — Tue, 14 Apr 2009 20:06:57 GMT

I just thought of something when I looked at my post...The bnx aliases point to the LAST NICs even though they are the on-boards. The python script was not run on this server.

Dave Convery
VMware vExpert 2009
http://www.dailyhypervisor.com

Careful. We don't want to learn from this.
Bill Watterson, "Calvin and Hobbes"

Re: Renaming VMnics to match PCI Slot Numbers

dconvery — Tue, 14 Apr 2009 20:04:26 GMT

Looking at a modules.conf file from a recent engagement, it looks like it MAY need to be edited. I am not sure if kudzu discovers the changes are re-orders the modules.conf file. I would have to try it the next time I come into the need to give a definite answer. I CAN say, however, that on the last engagement where I used it with different NICs, the server ran fine. I do not have a modules.conf file available from one where I actually ran the script. Like I said, I would assume that kudzu would handle the changes to modules.conf. Here's what I did last time:

kickstart install
reboot
ran python script on ESX console
reboot
Kudzu detects changes during boot up and reboots again
system comes up fine

Maybe someone can confirm this on their system? The latest engagement I am working is ALL e1000 NICs, so modules.conf does not NEED to be edited

Here's an output from a modules.conf file from an HP:

alias eth0 e1000
alias eth1 e1000
alias eth2 e1000
alias eth3 e1000
alias eth4 e1000
alias eth5 e1000
alias eth6 e1000
alias eth7 e1000
alias eth8 e1000
alias eth9 e1000
alias eth10 e1000
alias eth11 e1000
alias eth12 e1000
alias eth13 e1000
alias eth14 e1000
alias eth15 e1000
alias eth16 bnx2
alias eth17 bnx2

Dave Convery
VMware vExpert 2009
http://www.dailyhypervisor.com

Careful. We don't want to learn from this.
Bill Watterson, "Calvin and Hobbes"

Re: Renaming VMnics to match PCI Slot Numbers

linuxdynasty — Tue, 14 Apr 2009 17:44:24 GMT

WoW, you really went there, as professional as possible you are throwing your title around as well as your years of experience.... All I have been asking since the beginning of this thread and the other threads is WHY do you have to modify modules.conf. And I am not looking for because that is the VMware standard. Technically WHY??? The only reason I am harping on this, is because the Linux Kernel will assign the NIC's in the proper order based on PCI ID. Then after you are done with the ESX install, ESX will reorder them based on what ever criteria they are using. So All my script is doing is reordering back in the correct order based on PCI ID. Which you will never have to do again. So my question stands.. WHY do you HAVE to modify modules.conf??

I do not resent that you have another script out there. I really do not, as I am not in competition with you or anyone else, I just love giving back to the community. That is the sole purpose of my website! I just do not understand why you "HAVE" to modify modules.conf..... Let me ellaborate further.... You are the 1st user who has told me you "HAVE" to modify modules.conf and made it mandatory in all of your postings. I have yet to modify modules.conf with HP boxes and successfully pxe kicked my esx host's and yes my script has been used in 100's of esx deployments. So that is the only reason for me going back and forth with you. This is the first time that you say there is no reason not to modify modules.conf.

And you really do not want to make this posting about Experience and throwing around titles, because I can embarrass Admins who have been doing this type of work 3 to 4 longer then I have.

Re: Renaming VMnics to match PCI Slot Numbers

RichD — Tue, 14 Apr 2009 17:29:38 GMT

It's funny that you say that because the impression I got is that you kinda resented that I posted an alternate solution to the problem.

I think its quite cool that you posted your script online, and that you've been actively maintaining it, and I've given you credit for that since my first post.

There are two reasons why I started from scratch on this problem after trying your much earlier script in the lab.

First, you used python which was not supported by any of our resources in-house. We decided to leverage bash, gawk/awk, sed - tools that were very supportable and consistent with the rest of the deployment script suite we were developing. Your original version was not working correctly in our environment and we couldn't debug it, so we figured the problem out on our own and wrote our own script.

Also, you ignored the modules.conf file which is necessary to complete the nic renumeration properly. Otherwise you are going to have a port assignment mismatch in linux boot mode and lose connectivity. Further, you would leave the system inconsistent with the way it is intended to be deployed by VMware and when you do a support call and run into that problem, support will poo-poo the script.

Since I am a VCP and my primary mandate is to make things as conform to standards of practice as possible, I could not leave out the modules.conf fix from my script suite. If you look at the code I shared here, you'll see that the modules.conf fix is quite a bit more fancy that the esx.conf renumeration script. It had to be, to interpret the driver tables and pick the correct one.

The script was made by myself and another expert; combined we have over 30 years of Unix experience. We felt strongly that this was a required part of the solution and spent the extra time for good reason. The client (Federal Gov't) agreed. We wanted a reliable fix, not a partial hack.

An important note here is that there is no reason why you can't use my modules.conf fix script with your own python script. The two stand alone.

I don't have a website which I am driving traffic to with this thread. I published the scripts on the VMware community to help out my colleagues, not to compete with you. I think there is plenty of room for different solutions to this problem. In fact, I think that having a little "competition" of sorts has driven you to improve your script greatly which can only be a positive thing all around for those who choose to use it.

Cheers!

-Richard Dagenais, VCP

Re: Renaming VMnics to match PCI Slot Numbers

linuxdynasty — Tue, 14 Apr 2009 16:03:07 GMT

Hey RichD I know in the past we had our little tit for tat about the modules.conf on esx3.5, but You are the first person who I have heard mention that they needed to change modules.conf and use my script at the same time.

Dconvery do you have to modify the modules.conf when you run my script http://www.linuxdynasty.org/View-details/Linux-Dynasty-Scripts-and-Programs/1-Vmware-ESX-NIC-Reordering-FIX-Script.html as well??
Till this day I have not have to modify modules.conf with 4+ nics running in my environment.

Re: Renaming VMnics to match PCI Slot Numbers

RichD — Tue, 14 Apr 2009 15:25:49 GMT

You need to fix both esx.conf and modules.conf in order to complete the fix properly.

I used the attached scripts to achieve that.

If you run into problems, respond to thread and I'll post an update.

-
Richard Dagenais, VCP

Re: Renaming VMnics to match PCI Slot Numbers

ksf — Tue, 14 Apr 2009 12:24:27 GMT

Has anyone bothered with updating the modules.conf as well?

If so, has anyone had success with the make-modules_conf script(?), I just got a load of syntax errors when trying this...

Re: Renaming VMnics to match PCI Slot Numbers

dconvery — Sun, 22 Mar 2009 18:34:29 GMT

You're right Tom. It is weird too. I have it with the DL580 servers, but blades are OK. I did a deployment recently with 16 Dell R900 servers and had no problem with it. I include itin all deployments just in case now anyway. I think it has something to do with the way the kernel treats the addressing of internal components vs. slot components.

Dave Convery
VMware vExpert 2009
http://www.dailyhypervisor.com

Careful. We don't want to learn from this.
Bill Watterson, "Calvin and Hobbes"

Re: Renaming VMnics to match PCI Slot Numbers

tom howarth — Sun, 22 Mar 2009 13:57:00 GMT

It is not a just HP servers that suffer from it, Dell , and Fujitsu Seimens server do as well, I have not really found it to be an issue with HP servers, but then I could have been lucky.

If you found this or any other answer useful please consider the use of the Helpful or correct buttons to award points

Tom Howarth VCP / vExpert
VMware Communities User Moderator
Blog: www.planetvm.net
Contributing author for the upcoming book "VMware Virtual Infrastructure Security: Securing ESX and the Virtual Environment”.

Re: Renaming VMnics to match PCI Slot Numbers

dconvery — Sun, 22 Mar 2009 13:44:51 GMT

aenagy wrote:
Will this script work on ESXi via RCLI? (Obviously not for the vswif NICs )

I don't think this will work using the RCLI because of the way the script works. It needs to be executed locally, I believe. Although I haven't tested it, I would assume it would work by entering "unsupported mode" (also sometimes referred to as "Tech support mode") and running it from there.

If you DO run it from the unsupported mode on the ESXi server, it WILL change vswif0 as well.

After running the script, a reboot is required. The vswif0 reference could now potentially cause a disconnect if your vswif0 is not pointing to the first NIC on the PCI bus.

The intent here is for scripted installations where the NIC order is not what is expected and needs to be reset. I have seen this primarily on HP servers. It could be differences in the way isolinux kernel and the hypervisor kernel handles the PCI addresses.

Dave Convery
VMware vExpert 2009
http://www.dailyhypervisor.com

Careful. We don't want to learn from this.
Bill Watterson, "Calvin and Hobbes"

Re: Renaming VMnics to match PCI Slot Numbers

linuxdynasty — Sun, 22 Mar 2009 01:57:35 GMT

I have not tested this script on ESXi, so I really can not say. But if you have issues with the vmnic names being out of order with the PCI ID's then I would assume it should work as well.

Re: Renaming VMnics to match PCI Slot Numbers

RichD — Sun, 22 Mar 2009 01:39:35 GMT

Haven't tested it at all, so can't say.

-Rich

Re: Renaming VMnics to match PCI Slot Numbers

aenagy — Sun, 22 Mar 2009 00:42:41 GMT

Will this script work on ESXi via RCLI? (Obviously not for the vswif NICs )

Re: Renaming VMnics to match PCI Slot Numbers

dconvery — Sun, 15 Mar 2009 17:50:55 GMT

EXACTLY. I used this script after a kickstart and before a configure script. Works like a charm.

Dave Convery
VMware vExpert 2009
http://www.dailyhypervisor.com

Careful. We don't want to learn from this.
Bill Watterson, "Calvin and Hobbes"

Re: Renaming VMnics to match PCI Slot Numbers

linuxdynasty — Sun, 15 Mar 2009 04:01:35 GMT

The reason for the renaming with the script is for automation purposes. Especially in an environment when you have to kickstart many ESX hosts. If you are running ESX only on a few host, then you are write just log in to each one manually.

Re: Renaming VMnics to match PCI Slot Numbers

RParker — Sun, 15 Mar 2009 03:53:41 GMT

You don't have to reinstall ESX you can simply do:

esxcfg-vswitch -U vmnicx vSwitch0
esxcfg-vswitch -L vmnicx vSwitch0

The first one unlinks the vmnic(x) from your service console (vSwitch0 by default) the second links it to the correct vmnic(x). So you really don't need to renumber them (unless for some reason you are a control freak with OCD and it drives you nuts to see the numbers unmatched Otherwise once your system is up, it won't make a difference anyway.

Re: Renaming VMnics to match PCI Slot Numbers

linuxdynasty — Sat, 14 Mar 2009 17:07:09 GMT

Version 2.0 of the script is now available. The article is here linuxdynasty and the download link is there as well. You must become a member to download and membership is free.
Once you run the script, then all you have to do is reboot and vmware will reorder the nics back in the correct order.

1- Added the verbose option to the script.

2- fixed the verious non updates of the esx.conf..
example below.....

./esx_nic_fix.py --verbose
Pre Sort
011:00.0
005:00.0
003:00.0
011:00.1
015:00.0

Post Sort
003:00.0
005:00.0
011:00.0
011:00.1
015:00.0
Original Line: /device/011:00.0/vmkname = "vmnic2"
New Line: /device/011:00.0/vmkname = "vmnic2"

Original Line: /device/005:00.0/vmkname = "vmnic0"
New Line: /device/005:00.0/vmkname = "vmnic1"

Original Line: /device/003:00.0/vmkname = "vmnic1"
New Line: /device/003:00.0/vmkname = "vmnic0"

Original Line: /device/011:00.1/vmkname = "vmnic3"
New Line: /device/011:00.1/vmkname = "vmnic3"

Original Line: /device/015:00.0/vmkname = "vmnic4"
New Line: /device/015:00.0/vmkname = "vmnic4"

Re: Renaming VMnics to match PCI Slot Numbers

linuxdynasty — Mon, 16 Feb 2009 13:11:55 GMT

Hey no Problem, I will see what I can do. Make sure you grab my latest script since its fixed 2 issues in my previous esx_nic_fix.py.

Hey do me a favor, could you post this request on linuxdynasty.org in the Forums section. Also in the post please give me a little more details.. Like for the esx_nics_fix.py script I run a esxcfg-nics -l to get the output. How would I do it for the problem you are having with the qlogic adapters and SAS. And wat does the entry look like in esx.conf..

Re: Renaming VMnics to match PCI Slot Numbers

dconvery — Mon, 16 Feb 2009 12:55:53 GMT

Allen -

Kudos on the script. I used it again last week to get out of a jam. The customer forgot to disable the onboard NICs before an install and the NICs got out of order. Incidentally, it also changed the order of the onboard SAS and the PCI QLogic adapters, not as big of a deal, but it would be nice to have a script to fix this as well... Same type of info in the esx.conf file.

Dave

************************
Accomplishing the impossible means only that the boss will add it to your regular duties.
Doug Larson

Re: Renaming VMnics to match PCI Slot Numbers

linuxdynasty — Sat, 14 Feb 2009 23:02:28 GMT

Hey guys, just to let you know I updated my esx_nics_fix.py script that I have posted on http://linuxdynasty.org. It fixed 2 issues...
One was that I assumed their would not be more then 1 0 in the beginning of the PCI ID in esx.conf and the other because of issue 1 it would not reorder the vmnic name.

Both issues are fixed... Also I have yet a need to modify modules.conf at both my revious employment and at my current one..

the script is here http://www.linuxdynasty.org/script-to-fix-vmware-esx-35-nic-reordering-after-kickstart.html

Re: Renaming VMnics to match PCI Slot Numbers

RichD — Sat, 29 Nov 2008 20:44:51 GMT

I think the audience for this thread is clustered environments, where standardized and automated deployment is considered a requirement.

In order to deliver a consistent host install across varied hardware, you need to be absolutely sure of your NIC assignments. In a big company, many different people touch the various components of the system - one person is racking servers and patching things, another configuring switches, and yet another deploying the OS. Everything is documented, and fully tested beforehand.

I would agree that, if you only have one ESX box, you clearly aren't doing HA. And probably not looking for this thread. You'd most likely just swap the cables around or tinker with the file by hand, as you suggest.

My hope is that the community can benefit from the time I spent to solve this problem.

P.S. I also hope that editing my posts is not spamming everyone in the thread. This post editor is a little persnickety.

Re: Renaming VMnics to match PCI Slot Numbers

RichD — Fri, 28 Nov 2008 08:35:37 GMT

You can implement the nic reordering scripts during the first boot. Because networking is broken at that point, retrieving scripts during POST is a no-go. So I like to inject files during pre-install from the kickstart:

%pre
# declare the deploy path
http_addr=http://<deploy_host>/scripts
# get the pre-install script
echo "* GETTING the pre-install script..."
wget $http_addr/scripts/pre-install.sh
echo "*"
# make a backup, create a new file, inject the http_addr variable from above
echo "* INJECTING the http deploy host URL to pre-install.sh..."
mv pre-install.sh pre-install.orig
echo "*"
echo http_addr=$http_addr > pre-install.sh
echo "*"
# append the rest of the script
cat ./pre-install.orig >> pre-install.sh
echo "*"
# make it executable
echo "* MAKING pre-install.sh executable..."
chmod 755 ./pre-install.sh
echo "*"
# so lets go
echo "* STARTING pre-install.sh..."
./pre-install.sh &
echo "*"

Attached is a vanilla version of my pre-install script.

It will install and cue up a post-install.sh of your design on second boot, after the NICs have been re-ordered.

Updated Nov29

Re: Renaming VMnics to match PCI Slot Numbers

RichD — Fri, 28 Nov 2008 07:39:52 GMT

This script is to make modules.conf correspond to PCI device ID order.

usage: make-modules_conf < /etc/vmware/esx.conf > /etc/modules.conf.renum

It will re-order your modules.conf according to PCI device order and detect modules from /etc/vmware/vmware-devices.map

Use after running the vmnic-renum script during first boot tasks.

Updated with bugfix on Nov29

Re: Renaming VMnics to match PCI Slot Numbers

RichD — Fri, 28 Nov 2008 04:25:22 GMT

I had trouble with that python script. The author responds to posts on the forum about it though, which deserves a nod.

But I got impatient (see attached):

usage: renum-vmnic < esx.conf > esx.conf.renum

The key to vmnic ordering in esx.conf is to remember that the devices are listed in ascending PCI address order.

The first instance of an Ethernet card in the device section of esx.conf (look for vmkname) should be vmnic0, and increment up one from there for each subsequent device with a vmkname.

Then you need to go to the pnic section and make sure that child 0000 is vmnic0 and child 0001 is vmnic1 and so on. Don't worry about the MAC addresses - they are correct. It's all in PCI address ascending order.

If you end up with more pnic childs than vmnics in the device section, these are orphans from ESX trying to remap things. I delete them. You should have one device instance per pnic "child" id, with the vmkname/name being the association between them.

When you experience this NIC ordering insanity as I have with DL380 G5s, you will also note that modules.conf is wrong. modules.conf is going to be used when you boot in linux mode.

For example:

alias eth0 e1000
alias eth1 e1000
alias eth2 e1000
alias eth3 e1000
alias eth4 bnx2
alias eth5 bnx2

In a nutshell, e1000 gets loaded first, even if the Broadcom NICs have lower PCI addresses. Change it to:

alias eth0 bnx2
alias eth1 bnx2
alias eth2 e1000
alias eth3 e1000
alias eth4 e1000
alias eth5 e1000

I'm still working on a script to reorder modules.conf in a portable and programmatic way, based on PCI order, and the device to module mappings.

If you are kickstarting via http or ftp, you may be scratching your head about connectivity. In the above example, where e1000 got loaded first by default, it was the same in the kernel on the ISO. (See for yourself - in esx text mode installation, ALT-F2 once anaconda installer is kicked off to enter the shell, and go ahead and 'more /tmp/modules.conf')

For the DL380 G5's with this issue, I had to ksdevice=eth4 in order to use my first onboard NIC (i.e. the first bnx2 device in modules.conf) for network installation. By the time we get to post-install and my little awk script runs, we are in the kernel installed on the disk, so we've made this right - vmnic0 and eth0 are where they are supposed to be. But you can't do anything about the boot image other than hack the module load order and make a new ISO (I've been tempted to...) We just noted in our build book that for our specific G5's use ksdevice=eth4.

Re: Renaming VMnics to match PCI Slot Numbers

depping — Fri, 28 Nov 2008 08:43:15 GMT

if it's just one machine then manually editting "esx.conf" is the best solution in my opinion.

Duncan
Blogging: http://www.yellow-bricks.com

If you find this information useful, please award points for "correct" or "helpful".

Re: Renaming VMnics to match PCI Slot Numbers

dconvery — Tue, 19 Aug 2008 15:18:19 GMT

Jason -

Check out this python script. I use it all the time for kiskstarts...http://www.linuxdynasty.org/script-to-fix-vmware-esx-35-nic-reordering-after-kickstart.html

Dave

Renaming VMnics to match PCI Slot Numbers

JDLangdon — Tue, 19 Aug 2008 14:06:19 GMT

When installing ESX is choose the wrong Pnic for the service console. The end result is that vmnic0 is in PCI slot 2, vmnic1 is in PCI slot 2, and so on. Is there any way to ny way to rename the vmins so that the vmnic number matches the PCI slot number? I know I can very easily reinstall ESX but I'd like to know if there is an other way.

Jason