1 2 Previous Next 21 Replies Latest reply on Sep 19, 2014 5:23 AM by vmware088

Server Crash/Reboot: vnetflt.sys

yzennezy Nov 28, 2013 3:57 PM

Hi,

Does anyone know or has anyone heard or seen any issue using the VMXNET3 driver with Windows 2003 R2?

We're running ESXi 5.5 with a mix of Windows and Linux guests. We have migrated everything from VMware Server 2 using the VMware vCenter Converter to bring everything up to VM version 10. VMware Tools have been installed.

Since the migration, the Windows guests have been very unstable:

* Windows 7 guest has 'locked' or 'frozen' once and we had to reboot the host to get it back up and running

* Windows 2003 Server R2 guest has also 'frozen' requiring a reboot of the host

* A different Windows 2003 R2 guest is crashing and rebooting with a TCPIP issue.

All three guests are configured with the VMXNET3 Adapter Type. All three machines have experienced 'lockup'. Last night both the Windows Server 2003 R2 guests have crashed and rebooted on STOP ERROR 7f.

I have analysed the crash memory dump for both Windows Server 2003 R2 guests and they both point the finger directly at vnetflt.sys. The guests are configured with Adapter Type VMXNET3 and the underlying hardware on the ESXi host is an Intel 82599EB 10-Gigabit SFI/SFP+. The guests' interfaces come up using the vmxnet3 Ethernet Adapter driver version 1.5.1.0 (vmxnet3n51x86.sys) and report 10Gbps speed. Also the check-box for power management is on ('Allow the computer to turn off this device to save power' option). Perhaps that should be switched off?

Can anyone say or does anyone know or has heard about problems with the vnetflt.sys. Is it a component of the VMXNET3? Should I be using a different driver?

Any clues are much appreciated.

Kind regards,
Tom

1. Re: Server Crash/Reboot: vnetflt.sys

Josh26 Nov 28, 2013 9:24 PM (in response to yzennezy)

The obvious question here is going to be, is your server AND network card on the HCL?
Like Show 0 Likes(0)

Actions
2. Re: Server Crash/Reboot: vnetflt.sys

yzennezy Nov 28, 2013 10:02 PM (in response to Josh26)

Hi Josh,

Thanks for your feedback.

I had a look and found this:

VMware Compatibility Guide: I/O Device Search

It lists the following properties for a compatible device:
VID:    8086
DID:    10FB
SVID:    8086
SSID:    0000

I'm not sure how to match these on the host but I've tried the following:

# vmkchdev -l | grep -i 10fb 0000:81:00.0 8086:10fb 8086:0003 vmkernel vmnic2 0000:81:00.1 8086:10fb 8086:0003 vmkernel vmnic3 # esxcfg-nics -l | grep 82599EB vmnic2    0000:81:00.00 ixgbe       Up   10000Mbps Full   90:e2:ba:3f:cf:30 1500   Intel Corporation 82599EB 10-Gigabit SFI/SFP+ Network Connection vmnic3    0000:81:00.01 ixgbe       Down 0Mbps     Half   90:e2:ba:3f:cf:31 1500   Intel Corporation 82599EB 10-Gigabit SFI/SFP+ Network Connection

I'm assuming that I have the following:
VID:    8086 DID:    10FB SVID:    8086 SSID:    0003

If I've got that right (please confirm), what is the significance of the SSID difference (HCL SSID=0000, my SSID=0003)?

Regards,
Tom
Like Show 0 Likes(0)

Actions
3. Re: Server Crash/Reboot: vnetflt.sys

ScreamingSilence Nov 28, 2013 10:50 PM (in response to yzennezy)

Have you checked the supported GOS list at Intel� Server Adapters &mdash; Guest Operating System Support for SR-IOV in VMware vSphere* 5.1 and 5.5
Like Show 0 Likes(0)

Actions
4. Re: Server Crash/Reboot: vnetflt.sys

john23 Nov 28, 2013 11:02 PM (in response to yzennezy)

If you want to get the information about VID,DID etc from your system, run this command.. lspci -vvvvvv

It will show the information, and then do the matches from compatibility guide.

-A
Like Show 0 Likes(0)

Actions
5. Re: Server Crash/Reboot: vnetflt.sys

john23 Nov 28, 2013 11:13 PM (in response to john23)

esxcfg-info command also provide VID,DID information...
Like Show 0 Likes(0)

Actions
6. Re: Server Crash/Reboot: vnetflt.sys

yzennezy Dec 11, 2013 5:16 PM (in response to john23)

Hi,

Thanks for all your feedback. Sorry for not responding for so long, I have been unwell.

I changed the guest's configured Network Adapter Type to 'Flexible' and Installed the device as 'VMware Accelerated AMD PCNet Adapter', 'Driver Version 2.2.0.0'. The adapter shows up in the guest at 1Gbps speed. Since this change, it has been more stable but I still get the occasional 'lock-up' or crash (stop 7f).

Apart from the Sub-Device Id (mine is 0x0003), everything else matches up for the supported cards being supported. Is this purely a guest compatibility issue?

Kind regards,
Tom

esxcfg-info shows the following (it's dual port so there are two entries)
           \==+PCI Device :
               |----Segment.........................................0x0000
               |----Bus.............................................0x81
               |----Slot............................................0x00
               |----Function........................................0x00
               |----Runtime Owner...................................vmkernel
               |----Has Configured Owner............................false
               |----Configured Owner................................
               |----Vendor Id.......................................0x8086
               |----Device Id.......................................0x10fb
               |----Sub-Vendor Id...................................0x8086
               |----Sub-Device Id...................................0x0003
               |----Vendor Name.....................................Intel Corporation
               |----Device Name.....................................82599EB 10-Gigabit SFI/SFP+ Network Connection
               |----Device Class....................................512
               |----Device Class Name...............................Ethernet controller
               |----PIC Line........................................11
               |----Old IRQ.........................................11
               |----Vector..........................................56
               |----PCI Pin.........................................0
               |----Spawned Bus.....................................0
               |----Flags...........................................513
               \==+BAR Info :
                  \==+BAR0 :
                     |----Type......................................0x00000003
                     |----Address...................................0x00000000fb880000
                     |----Size......................................524288
                     |----Flags.....................................0x0000000c
                  \==+BAR1 :
                     |----Type......................................0x00000004
                     |----Address...................................0
                     |----Size......................................0
                     |----Flags.....................................0
                  \==+BAR2 :
                     |----Type......................................0x00000001
                     |----Address...................................0x000000000000f020
                     |----Size......................................32
                     |----Flags.....................................0x00000001
                  \==+BAR3 :
                     |----Type......................................0
                     |----Address...................................0
                     |----Size......................................0
                     |----Flags.....................................0
                  \==+BAR4 :
                     |----Type......................................0x00000003
                     |----Address...................................0x00000000fb904000
                     |----Size......................................16384
                     |----Flags.....................................0x0000000c
                  \==+BAR5 :
                     |----Type......................................0x00000004
                     |----Address...................................0
                     |----Size......................................0
                     |----Flags.....................................0
               |----Module Id.......................................4120
               |----Chassis.........................................0
               |----Physical Slot...................................5
               |----VmKernel Device Name............................vmnic2
               |----Slot Description................................Chassis slot 5.00
               |----Passthru Capable................................true
               |----Parent Device...................................PCI 0:128:1:0
               |----Dependent Device................................PCI 0:129:0:0
               |----Reset Method....................................1
               |----FPT Shareable...................................true
            \==+PCI Device :
               |----Segment.........................................0x0000
               |----Bus.............................................0x81
               |----Slot............................................0x00
               |----Function........................................0x01
               |----Runtime Owner...................................vmkernel
               |----Has Configured Owner............................false
               |----Configured Owner................................
               |----Vendor Id.......................................0x8086
               |----Device Id.......................................0x10fb
               |----Sub-Vendor Id...................................0x8086
               |----Sub-Device Id...................................0x0003
               |----Vendor Name.....................................Intel Corporation
               |----Device Name.....................................82599EB 10-Gigabit SFI/SFP+ Network Connection
               |----Device Class....................................512
               |----Device Class Name...............................Ethernet controller
               |----PIC Line........................................10
               |----Old IRQ.........................................10
               |----Vector..........................................57
               |----PCI Pin.........................................0
               |----Spawned Bus.....................................0
               |----Flags...........................................513
               \==+BAR Info :
                  \==+BAR0 :
                     |----Type......................................0x00000003
                     |----Address...................................0x00000000fb800000
                     |----Size......................................524288
                     |----Flags.....................................0x0000000c
                  \==+BAR1 :
                     |----Type......................................0x00000004
                     |----Address...................................0
                     |----Size......................................0
                     |----Flags.....................................0
                  \==+BAR2 :
                     |----Type......................................0x00000001
                     |----Address...................................0x000000000000f000
                     |----Size......................................32
                     |----Flags.....................................0x00000001
                  \==+BAR3 :
                     |----Type......................................0
                     |----Address...................................0
                     |----Size......................................0
                     |----Flags.....................................0
                  \==+BAR4 :
                     |----Type......................................0x00000003
                     |----Address...................................0x00000000fb900000
                     |----Size......................................16384
                     |----Flags.....................................0x0000000c
                  \==+BAR5 :
                     |----Type......................................0x00000004
                     |----Address...................................0
                     |----Size......................................0
                     |----Flags.....................................0
               |----Module Id.......................................4120
               |----Chassis.........................................0
               |----Physical Slot...................................5
               |----VmKernel Device Name............................vmnic3
               |----Slot Description................................Chassis slot 5.01
               |----Passthru Capable................................true
               |----Parent Device...................................PCI 0:128:1:0
               |----Dependent Device................................PCI 0:129:0:1
               |----Reset Method....................................1
               |----FPT Shareable...................................true
Like Show 0 Likes(0)

Actions
7. Re: Server Crash/Reboot: vnetflt.sys

chistv Apr 3, 2014 12:00 AM (in response to yzennezy)

There is a memory leak caused by VmWare tools VMCI driver VMware vShield Endpoint TDI manager:
The problem is still there after update to ESXi5.5 U1
vnetflt.sys 5.5.0.0 build-1191373

Solution: Uninstall this driver via vmware tools setup change

detected by windows support tools poolmon.exe (press p util type= NonP, press b to sort on Bytes) the Tag VNet is the vnetflt.sys driver
Tag Type     Allocs            Frees            Diff   Bytes       Per Alloc

VNet Nonp     486393 ( 875)    347795 ( 625)   138598 5561088 ( 10000)     40
MFEm Nonp     251716 (   0)    251683 (   0)       33 5245248 (     0) 158946

hope this helps
Like Show 0 Likes(0)

Actions
8. Re: Server Crash/Reboot: vnetflt.sys

sofasurfer May 8, 2014 12:47 AM (in response to chistv)

Hi guys,

We're also experiencing a memory leak with this driver on Windows Server 2012 - though we're running build 1397926. Does anyone know if there is a fix for this - other than uninstalling the driver.
Like Show 0 Likes(0)

Actions
9. Re: Server Crash/Reboot: vnetflt.sys

venugs May 13, 2014 12:59 AM (in response to yzennezy)

Hi,

We had a similar issue, it definitely related to the VMCI driver. However, the temporary workaround for the Guest OS to stop crashing repeatedly is to disable the VMCI driver in Device Manger by loggin into Safe Mode with Networking.

I am not sure about the adverse impacts of disabling it, but the guest OS recovered and we have migrated it to a different host which is earlier version of 5.1 and things were fine. ( not to mention we had a critical application that needs 2 E1000 and 2 VMXNET3 n/w adapters. )

hope it helps someone, somewhere.

Regards,
Venu
Like Show 0 Likes(0)

Actions
10. Re: Server Crash/Reboot: vnetflt.sys

Bleeder Jun 3, 2014 10:32 AM (in response to sofasurfer)

Well, you could uninstall the 5.5 tools and install the latest 5.1 tools if you really need the vShield drivers. Otherwise, there's no fix for the 5.5 tools yet.

For reference: http://kb.vmware.com/kb/2077302
Like Show 0 Likes(0)

Actions
11. Re: Server Crash/Reboot: vnetflt.sys

Bleeder Jun 25, 2014 9:49 AM (in response to yzennezy)

Here's another VMware KB article regarding this problem:
http://kb.vmware.com/kb/2081616
Like Show 0 Likes(0)

Actions
12. Re: Server Crash/Reboot: vnetflt.sys

DrNickT Jul 2, 2014 11:23 AM (in response to yzennezy)

I think I may be having this same issue. I have some 2003 VM's, when they first boot up they say that vnetflt will not start. Then they run for a while and then go completely unresponsive. No errors in the event log. they just need to be power cycled to get back online.

I'm running Cisco UCS B200 M3 Blades, ESXi 5.1 1612806
Like Show 0 Likes(0)

Actions
13. Re: Server Crash/Reboot: vnetflt.sys

Wh33ly Jul 4, 2014 6:03 AM (in response to DrNickT)

On Windows 2008 Server we had the same problems on a few machines where the Full VMware tools install was done, which include the vShield drivers.
We use ESXi version 5.1

I noticed that after removing/uninstalling the VMware tools the VMware vShield driver is still available and not completely removed.

Removed them manually:
Open device manager -> View -> Show Hidden Devices
Now an item appears : Non-Plug and play drivers, when you collapse this one you find the vNetFilter driver, this is the one that causes the problem. You can manually uninstall it here to be sure it isn't installed anymore. (right click uninstall)

My workflow
- Note IP settings (as vmxnet driver will also be removed which causes network disconnection)
- Remove VMware tools
- Remove vNetFilter in Device manager
- Reboot
- Install VMware Tools automatic or typical
- Reboot
- Reconfigure NIC
- Reboot (optional) because some application services require an active connection when starting etc.

Haven't found a similar workaround for Windows 2012

So to wrap it up
Option 1) Remove the leaky driver manually or through the VMware tools change option
VMware KB: Removing modules for VMware Tools during unattended install or upgrade
Option 2) Leave leaky driver, update VMware tools, driver is patched this is what Bleeder mentioned
For more information, see VMware ESXi 5.1, Patch Release ESXi510-201404001 (2070666) which fixes this leak.
Option 3) Option 1 + Install "typical" VMware tools without vShield drivers.

Currently there is no resolution for ESXi 5.5.

Because we don't use anything from vShield I preferred option 3 to be sure no non-essential components are installed which can cause problems in the future.
If you use vShield drivers, it's best to upgrade as soon as possible.
Like Show 1 Likes(1)

Actions
14. Re: Server Crash/Reboot: vnetflt.sys

yzennezy Jul 10, 2014 4:47 PM (in response to chistv)

Thanks Christv,

Solution: Uninstall this driver via vmware tools setup change
Uninstalling VmWare tools VMCI driver VMware vShield Endpoint TDI manager fixed the problem

Regards,
Tom
1 person found this helpful
Like Show 0 Likes(0)

Actions