4 Replies Latest reply on Aug 17, 2020 2:50 PM by Bill Oyler

    NSX-T 3.0 to 3.0.1 upgrade fails on ESXi 7.0 host VIB upgrades due to "Failed to load module nsxt-vsip"

    Bill Oyler Enthusiast

      Problem: Cannot upgrade ESXi 7.0 hosts from NSX-T 3.0.0 VIBs to 3.0.1 VIBs.

       

      Scenario: vCenter 7.0, ESXi 7.0, NSX-T 3.0.0, ESXi hosts are running N-VDS exclusively (2 pNIC, upgraded from NSX-T 2.5).

       

      NSX-T Edge upgrade from 3.0.0 to 3.0.1 was successful, but none of the ESXi hosts in the 4-node cluster are able to have their VIBs upgraded from 3.0.0 to 3.0.1.  Error message in NSX-T Manager is:

       

      Install of offline bundle failed on host 09e41e11-6ce5-4fd8-a4ad-3295f927e540 with error : [LiveInstallationError] Error in running ['/etc/init.d/nsx-datapath-dl', 'start', 'upgrade']: Return code: 1 Output: start upgrade begin Exception: Traceback (most recent call last): File "/etc/init.d/nsx-datapath-dl", line 963, in <module> DualLoadUpgrade() File "/etc/init.d/nsx-datapath-dl", line 835, in DualLoadUpgrade LoadKernelModules() File "/etc/init.d/nsx-datapath-dl", line 180, in LoadKernelModules nsxesxutils.loadModule(modName, modParam) File "/usr/lib/vmware/nsx-esx-datapath/lib/python3.5/nsxesxutils.py", line 360, in loadModule (moduleName, out.decode())) Exception: Failed to load module nsxt-vsip-16404614: vmkmod: VMKModLoad: VMKernel_LoadKernelModule(nsxt-vsip-16404614): Failure Cannot load module nsxt-vsip-16404614: Failure vmkmod: VMKModLoad: VMKernel_LoadKernelModule(nsxt-vsip-16404614): Failure Cannot load module nsxt-vsip-16404614: Failure It is not safe to continue. Please reboot the host immediately to discard the unfinished update. Please refer to the log file for more details..

       

      Error in esxupdate.log:

       

      [LiveInstallationError]

      Error in running ['/etc/init.d/nsx-datapath-dl', 'start', 'upgrade']:

      Return code: 1

      Output: start upgrade begin

      Exception:

      Traceback (most recent call last):

         File "/etc/init.d/nsx-datapath-dl", line 963, in <module>

           DualLoadUpgrade()

         File "/etc/init.d/nsx-datapath-dl", line 835, in DualLoadUpgrade

           LoadKernelModules()

         File "/etc/init.d/nsx-datapath-dl", line 180, in LoadKernelModules

           nsxesxutils.loadModule(modName, modParam)

         File "/usr/lib/vmware/nsx-esx-datapath/lib/python3.5/nsxesxutils.py", line 360, in loadModule

           (moduleName, out.decode()))

      Exception: Failed to load module nsxt-vsip-16404614: vmkmod: VMKModLoad: VMKernel_LoadKernelModule(nsxt-vsip-16404614): Failure

      Cannot load module nsxt-vsip-16404614: Failure

      vmkmod: VMKModLoad: VMKernel_LoadKernelModule(nsxt-vsip-16404614): Failure

      Cannot load module nsxt-vsip-16404614: Failure

       

      It is not safe to continue. Please reboot the host immediately to discard the unfinished update.

      Please refer to the log file for more details.

       

      Installing the VIBs manually via Lifecycle Manager (KB 78682) fails with exact same error as above in esxupdate.log.  Installing VIBs via CLI (KB 78679) results in same error.  I verified the boot banks (KB 74864) and they have plenty of free space (95% free space).  ESXi install was fresh install of ESXi 7.0 on 128 GB boot from SAN LUN (SSD).

       

      I never had any problems with NSX-T VIB upgrades from 2.4.x to 2.5.x to 3.0.0, so I'm curious why the 3.0.0 -> 3.0.1 upgrade is so challenging.  Has anyone else run into this?

       

      Thanks,

      Bill

        • 1. Re: NSX-T 3.0 to 3.0.1 upgrade fails on ESXi 7.0 host VIB upgrades due to "Failed to load module nsxt-vsip"
          Lalegre Expert

          Hello Bill!

           

          I can see that you are only facing issues with one of the modules that is the "nsxt-vsip" one. Reading a little bit i saw that this was fixed in older versions just rebooting the ESXi.

           

          You said that if you also try to install the VIBs manually fails but does it give you exactly the same error in the esxupdate.log?

          • 2. Re: NSX-T 3.0 to 3.0.1 upgrade fails on ESXi 7.0 host VIB upgrades due to "Failed to load module nsxt-vsip"
            Bill Oyler Enthusiast

            Yes, I saw that KB article too and tried rebooting each host about 6-7 times (each time between failed upgrade attempts) - no luck.  The error message in esxupdate.log is exactly the same regardless of whether I install the VIBs using NSX Manager, using Lifecycle Manager, or using the offline VIB bundle through esxcli software vib install.  The only thing I have not tried is a complete un-install of NSX VIBs, but I don't want to do this because the ESXi Management interface (vmk0) uses the N-VDS, so if I uninstall NSX VIBs, the ESXi host will lose all network conn ectivity unless I re-configure on VDS or VSS.  I can certainly do that if needed, but that seems a little overkill for a small NSX-T 3.0 to 3.0.1 upgrade... I never had to do that when upgrading from 2.4.x or 2.5.x to 3.0.

            • 3. Re: NSX-T 3.0 to 3.0.1 upgrade fails on ESXi 7.0 host VIB upgrades due to "Failed to load module nsxt-vsip"
              Lalegre Expert

              You are right about all the work that is needed to do but let me give you my opinion on this cases. Sometimes when you tried a lot of different procedures and workarounds sometimes the easier way is to reconfigure everything as the ESXi does not involve so much effort.

               

              Also you can raise a support ticket with VMware GSS but i guess that your option is the quickest and cleaner one. But who knows maybe you raise a support ticket and they alreadi have an internal KB for it.

               

              From my experience on this issues sometimes i prefer to remove the ESXi from the cluster, do all the steps needed and then do a clean install of everything needed (Usually not so much to do)

               

              Sorry that i cannot help you more but i have no clue.

               

              Let us know how it goes!

              • 4. Re: NSX-T 3.0 to 3.0.1 upgrade fails on ESXi 7.0 host VIB upgrades due to "Failed to load module nsxt-vsip"
                Bill Oyler Enthusiast

                FYI, I worked with GSS to resolve this issue by forcibly removing the "nsx-esx-datapath" VIB using the ESXi CLI:

                 

                esxcli software vib remove --vibname=nsx-esx-datapath --force --no-live-install

                 

                I needed to then manually load all of the 3.0.1 VIBs (using Lifecycle Manager in my case), and reboot each host one at a time.  I then reset the Upgrade Coordinator in the NSX-T Manager, and the upgrade was successful.  I had to do this on every host in the cluster.  No idea why that particular VIB was unable to be upgraded in-place...

                 

                Bill

                1 person found this helpful