Upgrading from 6.0 to 6.5 VIB trouble

[Super6VCA]

Upgrading one of my Dell R610's and having issues and looking for help.  During the upgrade process there were conflicting VIB's that i had to remove.  I did that and the upgrade was successful however, Now my nics are not showing up.  Looks to me like the net-tg3 VIB is installed since it shows on the list.  Is there something else that needs to be done to get these updated?  These are Broadcom BMC57xx.  Any help is appreciated.  Thanks

[ryanrpatel]

Can you run this command and reply back with the output?  esxcli fcoe nic discover -n

[Super6VCA]

No output at all wants a value for -n and everything i tried comes back incorrect

[TheBobkin]

Hello Perry,

"During the upgrade process there were conflicting VIB's that i had to remove. "

What exactly did you remove, why, when (e.g. before update and with a reboot after) and how?

"I did that and the upgrade was successful however, Now my nics are not showing up."

Did you test if these were functional after you removed some vibs but before updating? Don't forget that removing components that are loaded won't really apply until after reboot.

How did you update? If via the CLI did you use 'update' or 'install'? There is a big difference between these options, the former checks if you have a version of vib installed and if the one in the update package is higher it updates it, if it isn't there it won't install it. Conversely the 'install' option just blindly installs everything in the package - even if it is a lower version that currently present or wasn't installed already.

So it's likely going to come down to one of two things as the cause: 1. you don't have the required compatible driver installed or 2. your driver version/family has changed and is now no longer compatible with the firmware installed on the NIC.

First step would be to assess what is currently installed (driver+firmware) and cross-check these with the VMware HCL for these devices (don't just check 'BMC57xx' - get the specific part number and/or vendor IDs for the specific model of this device).

If you don't know what it was using before (that was functional), depending on the order of what you did before you may be able to just Shift+R at boot and roll-back to the last configuration (note that this doesn't remove your 6.5 install, just make no changes and Shift+R to revert back once more):

VMware Knowledge Base

Do you have other hosts that are on the old 6.0 configuration and/or ones that are upgraded to 6.5 and see their NICs? If they are/were the same config then check what driver+firmware they are using.

Bob

[Super6VCA]

Hi Bob,

"During the upgrade process there were conflicting VIB's that i had to remove. "

What exactly did you remove, why, when (e.g. before update and with a reboot after) and how?  Thru the command line. and i rebooted after.  Just the ones that were listed as conflicting.

How did you update?  I updated with a bootable CD. and I did not test during besides to see that the host was still active in vCenter until reboot.  For obvious reasons now.

Do you have other hosts that are on the old 6.0 configuration and/or ones that are upgraded to 6.5 and see their NICs? If they are/were the same config then check what driver+firmware they are using.  I do have one more that is identicle.  I did look and they are broadcom 5719 and 5720 drivers.  After looking it up it looks like the VIB net-tg3   3.135b.v55.1-1OEM.550.0.0.1331820     Broadcom  VMwareCertified   2019-02-22 is installed.  Is this not the case? 

Thanks for the great reply.  Appreciate the response.   

[Super6VCA]

Another thing that is odd to me.  When i go to CLI and look at the NICs, they are showing up and the status, speed, MTU and description show up as well.  However vCenter shows the server as not responding.  Can ping the address just fine, can ping the vmknic addresses just fine.  However when i look at the server console and look at the network cards it shows only two nics out of 12.  Confused!

[TheBobkin]

Hello Perry

The one that is 'identical' - has it been updated to 6.5? Whether it is pre-upgrade or not, compare the driver and firmware in-use on both:

VMware Knowledge Base

Log into the iDRAC and check which firmware each is using - as I said above, potentially they are not being picked up due to driver/firmware mismatch. Should be able to get part number and/or VID/DID/SVID/SSID there also which will allow you to confirm which driver family and version and firmware should be used e.g.:

VMware Compatibility Guide - I/O Device Search

Do note that there are generally different HCL entries and driver/firmware recommendations depending on what the card is being used for (e.g. FC vs Network) but then again I can only see 'Network' usage references for those Broadcom 5719 on the HCL.

If it appears to have correct driver and firmware installed then you could always try rolling it back to previous working configuration to confirm there is no physical issue with the card and/or look in the boot log if they are being detected at all.

Bob

[Super6VCA]

The Identical server has NOT been updated yet.  It is next..lucky me.  Unfortunately not sure i have any of the iDRAC info since i took over for someone else who set it up.  Can i get teh info i need off teh working host??  That was is Identical down to the nic?

[TheBobkin]

Hello Perry,

If you can SSH to the problem host then you *should* be able to get all the information about what NIC driver family and version it is using there.

Alternatively if no VMs are running on the problem host you could test reverting to the last config via Shift+R at boot as I noted above - this doesn't wipe the current configuration, it merely switches the order of /bootbank and /altbootbank. Then if the previous configuration is functional you can assess better what was done incorrectly in the upgrade by comparing what is in use there vs current.

Bob

[Super6VCA]

Hey Bob,

OK here is what i found out from the command line. The driver being on all the nics when i list them are ntg3.  Those show up as VENDOR VMW not broadcom.  the one that i think i need is net-tg3 which is on the system and on the HCL.  Questions...

Is the wrong driver being used?

Is the correct one on the system?

Once again thank you for all the help.  This is a little out of my wheelhouse.

Here is the cli output. 

[root@HOST3:~] esxcli network nic list

Name     PCI Device    Driver  Admin Status  Link Status  Speed  Duplex  MAC Address         MTU  Description

-------  ------------  ------  ------------  -----------  -----  ------  -----------------  ----  -------------------------------------------------------

vmnic0   0000:01:00.0  ntg3    Up            Up            1000  Full    d4:ae:52:a2:f1:16  1500  Broadcom Corporation NetXtreme BCM5720 Gigabit Ethernet

vmnic1   0000:01:00.1  ntg3    Up            Up            1000  Full    d4:ae:52:a2:f1:17  1500  Broadcom Corporation NetXtreme BCM5720 Gigabit Ethernet

vmnic10  0000:42:00.2  ntg3    Up            Down             0  Half    00:0a:f7:7d:8d:42  1500  Broadcom Corporation NetXtreme BCM5719 Gigabit Ethernet

esxcli network nic get -n vmnic0

   Advertised Auto Negotiation: true

   Advertised Link Modes: Auto, 1000BaseT/Full, 100BaseT/Half, 100BaseT/Full, 10BaseT/Half, 10BaseT/Full

   Auto Negotiation: false

   Cable Type: Twisted Pair

   Current Message Level: 7

   Driver Info:

         Bus Info: 0000:01:00:0

         Driver: ntg3

         Firmware Version: bc 1.17 ncsi 1.0.60.0

         Version: 4.1.2.0

   Link Detected: true

   Link Status: Up

esxcli network nic get -n vmnic5

   Advertised Auto Negotiation: true

   Advertised Link Modes: Auto, 1000BaseT/Full, 100BaseT/Half, 100BaseT/Full, 10BaseT/Half, 10BaseT/Full

   Auto Negotiation: false

   Cable Type: Twisted Pair

   Current Message Level: 7

   Driver Info:

         Bus Info: 0000:05:00:1

         Driver: ntg3

         Firmware Version: bc 1.24 ncsi 1.0.60.0

         Version: 4.1.2.0

esxcli software vib list |grep tg

net-tg3                        3.135b.v55.1-1OEM.550.0.0.1331820     Broadcom  VMwareCertified   2019-02-22

ntg3                           4.1.2.0-1vmw.650.1.26.5969303         VMW       VMwareCertified   2019-02-22

[Super6VCA]

Hey Bob,

OK here is what i found out from the command line. The driver being on all the nics when i list them are ntg3.  Those show up as VENDOR VMW not broadcom.  the one that i think i need is net-tg3 which is on the system and on the HCL.  Unable to rollback since i booted from an ISO.  Questions...

Is the wrong driver being used?

Is the correct one on the system?

Once again thank you for all the help.  This is a little out of my wheelhouse.

Here is the cli output. 

[root@HOST3:~] esxcli network nic list

Name     PCI Device    Driver  Admin Status  Link Status  Speed  Duplex  MAC Address         MTU  Description

-------  ------------  ------  ------------  -----------  -----  ------  -----------------  ----  -------------------------------------------------------

vmnic0   0000:01:00.0  ntg3    Up            Up            1000  Full    d4:ae:52:a2:f1:16  1500  Broadcom Corporation NetXtreme BCM5720 Gigabit Ethernet

vmnic1   0000:01:00.1  ntg3    Up            Up            1000  Full    d4:ae:52:a2:f1:17  1500  Broadcom Corporation NetXtreme BCM5720 Gigabit Ethernet

vmnic10  0000:42:00.2  ntg3    Up            Down             0  Half    00:0a:f7:7d:8d:42  1500  Broadcom Corporation NetXtreme BCM5719 Gigabit Ethernet

esxcli network nic get -n vmnic0

   Advertised Auto Negotiation: true

   Advertised Link Modes: Auto, 1000BaseT/Full, 100BaseT/Half, 100BaseT/Full, 10BaseT/Half, 10BaseT/Full

   Auto Negotiation: false

   Cable Type: Twisted Pair

   Current Message Level: 7

   Driver Info:

         Bus Info: 0000:01:00:0

         Driver: ntg3

         Firmware Version: bc 1.17 ncsi 1.0.60.0

         Version: 4.1.2.0

   Link Detected: true

   Link Status: Up

esxcli network nic get -n vmnic5

   Advertised Auto Negotiation: true

   Advertised Link Modes: Auto, 1000BaseT/Full, 100BaseT/Half, 100BaseT/Full, 10BaseT/Half, 10BaseT/Full

   Auto Negotiation: false

   Cable Type: Twisted Pair

   Current Message Level: 7

   Driver Info:

         Bus Info: 0000:05:00:1

         Driver: ntg3

         Firmware Version: bc 1.24 ncsi 1.0.60.0

         Version: 4.1.2.0

esxcli software vib list |grep tg

net-tg3                        3.135b.v55.1-1OEM.550.0.0.1331820     Broadcom  VMwareCertified   2019-02-22

ntg3                           4.1.2.0-1vmw.650.1.26.5969303         VMW       VMwareCertified   2019-02-22

[Super6VCA]

UPDATE...

After the upgrade my host showed that it was not responding (was in maint mode) after the upgrade.  After looking at everything i was a bit puzzled since i was able to login thru the web client directly to the host, it seen all of my volumes and networks.  So, i removed it from the cluster and re-added it and it looks normal.  Everything looks fine and vMotion works fine.  The only thing  that still bothers me is that when i look at my local console and look at "configure management network" and go to network adapters, it only shows me 2 adapters and not 12.  And the adapters it shows me do not have the correct models.  Do i need to update or install new drivers on the console?? 

[TheBobkin · Accepted solution]

Hello Perry,

"Everything looks fine and vMotion works fine.  The only thing  that still bothers me is that when i look at my local console and look at "configure management network" and go to network adapters, it only shows me 2 adapters and not 12."

But do you actually see the 12 links e.g. in networking configuration in GUI or with #esxcli network nic list ?

ntg3  4.1.2.0-1vmw is the 'inbox' driver that comes with non-vendor specific VMware ISOs and thus why it shows as 'VMW', however our HCL listing for this provides 'N/A' as minimum firmware version which basically means we don't advise on this via HCL and vendor provides this information (which may of course change as new releases are out). So potentially the firmware you have on those is not compatible with the inbox (and thus why models showing incorrect, NICs not showing etc.). What I would suggest is considering either installing/updating ESXi with a vendor-specific ISO for these servers and acquiring the matching firmware from their portal. Alternatively (provided you have bootbank AND altbootbank from installing on physical medium) you could try removing the inbox driver, updating the tg3 driver to one on the HCL that has a listed minimum firmware version (e.g. tg3 version 3.137l.v60.1 + firmware 1.45) and updating the firmware to the listed one.

Bob

_______________________________________________________________________________________________________

"Did you find this helpful? Let us know by completing this survey (takes 1 minute!)"

[Super6VCA]

Thanks again for the explanation.  Now it's a bit clearer to me.  Was puzzled why i could still see everything.  I will find a vendor specific ISO and update that way.  Not sure if they will help since it's out of warranty.  Thank you very much for the assistance on this.  Really do appreciate it!