1 2 Previous Next 21 Replies Latest reply on Sep 19, 2014 5:23 AM by vmware088

    Server Crash/Reboot: vnetflt.sys

    yzennezy Novice

      Hi,

       

      Does anyone know or has anyone heard or seen any issue using the VMXNET3 driver with Windows 2003 R2?

       

      We're running ESXi 5.5 with a mix of Windows and Linux guests. We have migrated everything from VMware Server 2 using the VMware vCenter Converter to bring everything up to VM version 10. VMware Tools have been installed.

       

      Since the migration, the Windows guests have been very unstable:

      * Windows 7 guest has 'locked' or 'frozen' once and we had to reboot the host to get it back up and running

      * Windows 2003 Server R2 guest has also 'frozen' requiring a reboot of the host

      * A different Windows 2003 R2 guest is crashing and rebooting with a TCPIP issue.

       

      All three guests are configured with the VMXNET3 Adapter Type. All three machines have experienced 'lockup'. Last night both the Windows Server 2003 R2 guests have crashed and rebooted on STOP ERROR 7f.

       

      I have analysed the crash memory dump for both Windows Server 2003 R2 guests and they both point the finger directly at vnetflt.sys. The guests are configured with Adapter Type VMXNET3 and the underlying hardware on the ESXi host is an Intel 82599EB 10-Gigabit SFI/SFP+. The guests' interfaces come up using the vmxnet3 Ethernet Adapter driver version 1.5.1.0 (vmxnet3n51x86.sys) and report 10Gbps speed. Also the check-box for power management is on ('Allow the computer to turn off this device to save power' option). Perhaps that should be switched off?

       

      Can anyone say or does anyone know or has heard about problems with the vnetflt.sys. Is it a component of the VMXNET3? Should I be using a different driver?


      Any clues are much appreciated.

       

      Kind regards,
      Tom

        • 1. Re: Server Crash/Reboot: vnetflt.sys
          Josh26 Master

          The obvious question here is going to be, is your server AND network card on the HCL?

          • 2. Re: Server Crash/Reboot: vnetflt.sys
            yzennezy Novice

            Hi Josh,

             

            Thanks for your feedback.

             

            I had a look and found this:

             

            VMware Compatibility Guide: I/O Device Search

             

            It lists the following properties for a compatible device:

            VID:    8086

            DID:    10FB

            SVID:    8086

            SSID:    0000


            I'm not sure how to match these on the host but I've tried the following:

             

            # vmkchdev -l | grep -i 10fb
            0000:81:00.0 8086:10fb 8086:0003 vmkernel vmnic2
            0000:81:00.1 8086:10fb 8086:0003 vmkernel vmnic3

            # esxcfg-nics -l | grep 82599EB
            vmnic2    0000:81:00.00 ixgbe       Up   10000Mbps Full   90:e2:ba:3f:cf:30 1500   Intel Corporation 82599EB 10-Gigabit SFI/SFP+ Network Connection
            vmnic3    0000:81:00.01 ixgbe       Down 0Mbps     Half   90:e2:ba:3f:cf:31 1500   Intel Corporation 82599EB 10-Gigabit SFI/SFP+ Network Connection


            I'm assuming that I have the following:

            VID:    8086
            DID:    10FB
            SVID:    8086
            SSID:    0003

             

            If I've got that right (please confirm), what is the significance of the SSID difference (HCL SSID=0000, my SSID=0003)?

             

            Regards,

            Tom

            • 4. Re: Server Crash/Reboot: vnetflt.sys
              john23 Expert

              If you want to get the information about VID,DID etc from your system, run this command.. lspci -vvvvvv

               

              It will show the information, and then do the matches from compatibility guide.

               

              -A

              • 5. Re: Server Crash/Reboot: vnetflt.sys
                john23 Expert


                esxcfg-info command also provide VID,DID information...

                • 6. Re: Server Crash/Reboot: vnetflt.sys
                  yzennezy Novice

                  Hi,

                   

                  Thanks for all your feedback. Sorry for not responding for so long, I have been unwell.

                   

                  I changed the guest's configured Network Adapter Type to 'Flexible' and Installed the device as 'VMware Accelerated AMD PCNet Adapter', 'Driver Version 2.2.0.0'. The adapter shows up in the guest at 1Gbps speed. Since this change, it has been more stable but I still get the occasional 'lock-up' or crash (stop 7f).

                   

                  Apart from the Sub-Device Id (mine is 0x0003), everything else matches up for the supported cards being supported. Is this purely a guest compatibility issue?

                   

                  Kind regards,

                  Tom

                   

                  esxcfg-info shows the following (it's dual port so there are two entries)

                             \==+PCI Device :

                                 |----Segment.........................................0x0000

                                 |----Bus.............................................0x81

                                 |----Slot............................................0x00

                                 |----Function........................................0x00

                                 |----Runtime Owner...................................vmkernel

                                 |----Has Configured Owner............................false

                                 |----Configured Owner................................

                                 |----Vendor Id.......................................0x8086

                                 |----Device Id.......................................0x10fb

                                 |----Sub-Vendor Id...................................0x8086

                                 |----Sub-Device Id...................................0x0003

                                 |----Vendor Name.....................................Intel Corporation

                                 |----Device Name.....................................82599EB 10-Gigabit SFI/SFP+ Network Connection

                                 |----Device Class....................................512

                                 |----Device Class Name...............................Ethernet controller

                                 |----PIC Line........................................11

                                 |----Old IRQ.........................................11

                                 |----Vector..........................................56

                                 |----PCI Pin.........................................0

                                 |----Spawned Bus.....................................0

                                 |----Flags...........................................513

                                 \==+BAR Info :

                                    \==+BAR0 :

                                       |----Type......................................0x00000003

                                       |----Address...................................0x00000000fb880000

                                       |----Size......................................524288

                                       |----Flags.....................................0x0000000c

                                    \==+BAR1 :

                                       |----Type......................................0x00000004

                                       |----Address...................................0

                                       |----Size......................................0

                                       |----Flags.....................................0

                                    \==+BAR2 :

                                       |----Type......................................0x00000001

                                       |----Address...................................0x000000000000f020

                                       |----Size......................................32

                                       |----Flags.....................................0x00000001

                                    \==+BAR3 :

                                       |----Type......................................0

                                       |----Address...................................0

                                       |----Size......................................0

                                       |----Flags.....................................0

                                    \==+BAR4 :

                                       |----Type......................................0x00000003

                                       |----Address...................................0x00000000fb904000

                                       |----Size......................................16384

                                       |----Flags.....................................0x0000000c

                                    \==+BAR5 :

                                       |----Type......................................0x00000004

                                       |----Address...................................0

                                       |----Size......................................0

                                       |----Flags.....................................0

                                 |----Module Id.......................................4120

                                 |----Chassis.........................................0

                                 |----Physical Slot...................................5

                                 |----VmKernel Device Name............................vmnic2

                                 |----Slot Description................................Chassis slot 5.00

                                 |----Passthru Capable................................true

                                 |----Parent Device...................................PCI 0:128:1:0

                                 |----Dependent Device................................PCI 0:129:0:0

                                 |----Reset Method....................................1

                                 |----FPT Shareable...................................true

                              \==+PCI Device :

                                 |----Segment.........................................0x0000

                                 |----Bus.............................................0x81

                                 |----Slot............................................0x00

                                 |----Function........................................0x01

                                 |----Runtime Owner...................................vmkernel

                                 |----Has Configured Owner............................false

                                 |----Configured Owner................................

                                 |----Vendor Id.......................................0x8086

                                 |----Device Id.......................................0x10fb

                                 |----Sub-Vendor Id...................................0x8086

                                 |----Sub-Device Id...................................0x0003

                                 |----Vendor Name.....................................Intel Corporation

                                 |----Device Name.....................................82599EB 10-Gigabit SFI/SFP+ Network Connection

                                 |----Device Class....................................512

                                 |----Device Class Name...............................Ethernet controller

                                 |----PIC Line........................................10

                                 |----Old IRQ.........................................10

                                 |----Vector..........................................57

                                 |----PCI Pin.........................................0

                                 |----Spawned Bus.....................................0

                                 |----Flags...........................................513

                                 \==+BAR Info :

                                    \==+BAR0 :

                                       |----Type......................................0x00000003

                                       |----Address...................................0x00000000fb800000

                                       |----Size......................................524288

                                       |----Flags.....................................0x0000000c

                                    \==+BAR1 :

                                       |----Type......................................0x00000004

                                       |----Address...................................0

                                       |----Size......................................0

                                       |----Flags.....................................0

                                    \==+BAR2 :

                                       |----Type......................................0x00000001

                                       |----Address...................................0x000000000000f000

                                       |----Size......................................32

                                       |----Flags.....................................0x00000001

                                    \==+BAR3 :

                                       |----Type......................................0

                                       |----Address...................................0

                                       |----Size......................................0

                                       |----Flags.....................................0

                                    \==+BAR4 :

                                       |----Type......................................0x00000003

                                       |----Address...................................0x00000000fb900000

                                       |----Size......................................16384

                                       |----Flags.....................................0x0000000c

                                    \==+BAR5 :

                                       |----Type......................................0x00000004

                                       |----Address...................................0

                                       |----Size......................................0

                                       |----Flags.....................................0

                                 |----Module Id.......................................4120

                                 |----Chassis.........................................0

                                 |----Physical Slot...................................5

                                 |----VmKernel Device Name............................vmnic3

                                 |----Slot Description................................Chassis slot 5.01

                                 |----Passthru Capable................................true

                                 |----Parent Device...................................PCI 0:128:1:0

                                 |----Dependent Device................................PCI 0:129:0:1

                                 |----Reset Method....................................1

                                 |----FPT Shareable...................................true

                  • 7. Re: Server Crash/Reboot: vnetflt.sys
                    chistv Novice

                    There is a memory leak caused by VmWare tools VMCI driver VMware vShield Endpoint TDI manager:

                    The problem is still there after update to ESXi5.5 U1

                    vnetflt.sys 5.5.0.0 build-1191373

                     

                    Solution: Uninstall this driver via vmware tools setup change

                     

                    detected by windows support tools poolmon.exe (press p util type= NonP, press b to sort on Bytes) the Tag VNet is the vnetflt.sys driver

                    Tag  Type     Allocs            Frees            Diff   Bytes       Per Alloc

                     

                    VNet Nonp     486393 ( 875)    347795 ( 625)   138598 5561088 ( 10000)     40

                    MFEm Nonp     251716 (   0)    251683 (   0)       33 5245248 (     0) 158946

                     

                    hope this helps

                    • 8. Re: Server Crash/Reboot: vnetflt.sys
                      sofasurfer Lurker

                      Hi guys,

                       

                      We're also experiencing a memory leak with this driver on Windows Server 2012 - though we're running build 1397926. Does anyone know if there is a fix for this - other than uninstalling the driver.

                      • 9. Re: Server Crash/Reboot: vnetflt.sys
                        venugs Novice

                        Hi,

                         

                        We had a similar issue, it definitely related to the VMCI driver. However, the temporary workaround for the Guest OS to stop crashing repeatedly is to disable the VMCI driver in Device Manger by loggin into Safe Mode with Networking.

                         

                        I am not sure about the adverse impacts of disabling it, but the guest OS recovered and we have migrated it to a different host which is earlier version of 5.1 and things were fine. ( not to mention we had a critical application that needs 2 E1000 and 2 VMXNET3 n/w adapters. )

                         

                        hope it helps someone, somewhere.

                         

                        Regards,

                        Venu

                        • 10. Re: Server Crash/Reboot: vnetflt.sys
                          Bleeder Hot Shot

                          Well, you could uninstall the 5.5 tools and install the latest 5.1 tools if you really need the vShield drivers.  Otherwise, there's no fix for the 5.5 tools yet.

                           

                          For reference: http://kb.vmware.com/kb/2077302

                          • 11. Re: Server Crash/Reboot: vnetflt.sys
                            Bleeder Hot Shot

                            Here's another VMware KB article regarding this problem:

                            http://kb.vmware.com/kb/2081616

                            • 12. Re: Server Crash/Reboot: vnetflt.sys
                              DrNickT Novice

                              I think I may be having this same issue.  I have some 2003 VM's, when they first boot up they say that vnetflt will not start.  Then they run for a while and then go completely unresponsive.  No errors in the event log.  they just need to be power cycled to get back online.

                               

                              I'm running Cisco UCS B200 M3 Blades, ESXi 5.1 1612806

                              • 13. Re: Server Crash/Reboot: vnetflt.sys
                                Wh33ly Hot Shot

                                On Windows 2008 Server we had the same problems on a few machines where the Full VMware tools install was done, which include the vShield drivers.

                                We use ESXi version 5.1

                                 

                                I noticed that after removing/uninstalling the VMware tools the VMware vShield driver is still available and not completely removed.

                                 

                                Removed them manually:

                                Open device manager -> View -> Show Hidden Devices

                                Now an item appears : Non-Plug and play drivers, when you collapse this one you find the vNetFilter driver, this is the one that causes the problem. You can manually uninstall it here to be sure it isn't installed anymore. (right click uninstall)

                                 

                                My workflow

                                - Note IP settings (as vmxnet driver will also be removed which causes network disconnection)

                                - Remove VMware tools

                                - Remove vNetFilter in Device manager

                                - Reboot

                                - Install VMware Tools automatic or typical

                                - Reboot

                                - Reconfigure NIC

                                - Reboot (optional) because some application services require an active connection when starting etc.

                                 

                                Haven't found a similar workaround for Windows 2012

                                 

                                So to wrap it up

                                Option 1) Remove the leaky driver manually or through the VMware tools change option

                                VMware KB: Removing modules for VMware Tools during unattended install or upgrade

                                Option 2) Leave leaky driver, update VMware tools, driver is patched this is what Bleeder mentioned

                                For more information, see VMware ESXi 5.1, Patch Release ESXi510-201404001 (2070666) which fixes this leak.

                                Option 3) Option 1 + Install "typical" VMware tools without vShield drivers.

                                 

                                Currently there is no resolution for ESXi 5.5.

                                 

                                Because we don't use anything from vShield I preferred option 3 to be sure no non-essential components are installed which can cause problems in the future.

                                If you use vShield drivers, it's best to upgrade as soon as possible.

                                • 14. Re: Server Crash/Reboot: vnetflt.sys
                                  yzennezy Novice

                                  Thanks Christv,

                                   

                                  Solution: Uninstall this driver via vmware tools setup change

                                  Uninstalling VmWare tools VMCI driver VMware vShield Endpoint TDI manager fixed the problem

                                   

                                  Regards,

                                  Tom

                                  1 person found this helpful
                                  1 2 Previous Next