Server Crash/Reboot: vnetflt.sys

[yzennezy]

Hi,

Does anyone know or has anyone heard or seen any issue using the VMXNET3 driver with Windows 2003 R2?

We're running ESXi 5.5 with a mix of Windows and Linux guests. We have migrated everything from VMware Server 2 using the VMware vCenter Converter to bring everything up to VM version 10. VMware Tools have been installed.

Since the migration, the Windows guests have been very unstable:

* Windows 7 guest has 'locked' or 'frozen' once and we had to reboot the host to get it back up and running

* Windows 2003 Server R2 guest has also 'frozen' requiring a reboot of the host

* A different Windows 2003 R2 guest is crashing and rebooting with a TCPIP issue.

All three guests are configured with the VMXNET3 Adapter Type. All three machines have experienced 'lockup'. Last night both the Windows Server 2003 R2 guests have crashed and rebooted on STOP ERROR 7f.

I have analysed the crash memory dump for both Windows Server 2003 R2 guests and they both point the finger directly at vnetflt.sys. The guests are configured with Adapter Type VMXNET3 and the underlying hardware on the ESXi host is an Intel 82599EB 10-Gigabit SFI/SFP+. The guests' interfaces come up using the vmxnet3 Ethernet Adapter driver version 1.5.1.0 (vmxnet3n51x86.sys) and report 10Gbps speed. Also the check-box for power management is on ('Allow the computer to turn off this device to save power' option). Perhaps that should be switched off?

Can anyone say or does anyone know or has heard about problems with the vnetflt.sys. Is it a component of the VMXNET3? Should I be using a different driver?

Any clues are much appreciated.

Kind regards,
Tom

[capsoc]

We tried the VM hardware version 10 uplift, but that did not fix it for us.

The fix for us was to receive the updated vShield driver from Vmware. We are now awaiting them to release a public ESX update / tools update so we can roll it out to all our Windows VM's.

[vmware088]

Just wanted to provide our feedback as we have had issues with crashes.  BSOD debug points to vnetflt.sys as well.  ESXi 5.1 1900470, Trend Micro Deep Security v9.0.  We could not cleanly uninstall all vmware tools until we deactivated the client from Deep Security Manager.  Only then could we properly remove the vShield drivers.   We could have found this sooner, however received very poor support from VMware initially.  We suspected something with the VMware Tools because it started going crazy after an upgrade.  Tried to uninstall but reminents of vmware tools still there.  VMware Tech's solution was to disable VMware Tools service.  It was days later when I looked in system drivers list in system tools and found that vnetflt.sys was still there and in fact running!  Disabling Tools did not fix.

Good news no more crashes...bad news...what now for malware protection.  No confidence in agentless with vshield drivers.

Anyway just wanted to give my 2 cents.

[Akim_khan]

i have the same issue, let me know if have any concrete solution for it, or recommend to reinstall Vm tool?

find the below bugcheck details.

*******************************************************************************

*                                                                             *

*                        Bugcheck Analysis                                    *

*                                                                             *

*******************************************************************************

SYSTEM_THREAD_EXCEPTION_NOT_HANDLED_M (1000007e)

This is a very common bugcheck.  Usually the exception address pinpoints

the driver/function that caused the problem.  Always note this address

as well as the link date of the driver/image that contains this address.

Some common problems are exception code 0x80000003.  This means a hard

coded breakpoint or assertion was hit, but this system was booted

/NODEBUG.  This is not supposed to happen as developers should never have

hardcoded breakpoints in retail code, but ...

If this happens, make sure a debugger gets connected, and the

system is booted /DEBUG.  This will let us see why this breakpoint is

happening.

Arguments:

Arg1: ffffffffc0000005, The exception code that was not handled

Arg2: fffff880015c0478, The address that the exception occurred at

Arg3: fffff8800230f7e8, Exception Record Address

Arg4: fffff8800230f050, Context Record Address

Debugging Details:

------------------

DUMP_CLASS: 1

DUMP_QUALIFIER: 400

BUILD_VERSION_STRING:  7601.23864.amd64fre.win7sp1_ldr.170707-0600

SYSTEM_MANUFACTURER:  VMware, Inc.

VIRTUAL_MACHINE:  VMware

SYSTEM_PRODUCT_NAME:  VMware Virtual Platform

SYSTEM_VERSION:  None

BIOS_VENDOR:  Phoenix Technologies LTD

BIOS_VERSION:  6.00

BIOS_DATE:  04/14/2014

BASEBOARD_MANUFACTURER:  Intel Corporation

BASEBOARD_PRODUCT:  440BX Desktop Reference Platform

BASEBOARD_VERSION:  None

DUMP_TYPE:  2

BUGCHECK_P1: ffffffffc0000005

BUGCHECK_P2: fffff880015c0478

BUGCHECK_P3: fffff8800230f7e8

BUGCHECK_P4: fffff8800230f050

EXCEPTION_CODE: (NTSTATUS) 0xc0000005 - The instruction at 0x%08lx referenced memory at 0x%08lx. The memory could not be %s.

FAULTING_IP:

vsepflt+13478

fffff880`015c0478 450fb7435c      movzx   r8d,word ptr [r11+5Ch]

EXCEPTION_RECORD:  fffff8800230f7e8 -- (.exr 0xfffff8800230f7e8)

ExceptionAddress: fffff880015c0478 (vsepflt+0x0000000000013478)

   ExceptionCode: c0000005 (Access violation)

  ExceptionFlags: 00000000

NumberParameters: 2

   Parameter[0]: 0000000000000000

   Parameter[1]: ffffffffffffffff

Attempt to read from address ffffffffffffffff

CONTEXT:  fffff8800230f050 -- (.cxr 0xfffff8800230f050)

rax=ffff0000036e1811 rbx=fffffa800f754a10 rcx=0000000000000010

rdx=fffff880015dfac0 rsi=0000000000000002 rdi=0000000000000010

rip=fffff880015c0478 rsp=fffff8800230fa20 rbp=0000000000000000

r8=0000000000380036  r9=0000000000000002 r10=fffff880009cec40

r11=00300032002e002a r12=fffff880013ecc80 r13=fffff880015c0838

r14=0000000000000000 r15=fffff880013e6000

iopl=0         nv up ei pl nz na pe nc

cs=0010  ss=0000  ds=002b  es=002b  fs=0053  gs=002b             efl=00010202

vsepflt+0x13478:

fffff880`015c0478 450fb7435c      movzx   r8d,word ptr [r11+5Ch] ds:002b:00300032`002e0086=????

Resetting default scope

CPU_COUNT: 4

CPU_MHZ: a28

CPU_VENDOR:  GenuineIntel

CPU_FAMILY: 6

CPU_MODEL: 3e

CPU_STEPPING: 4

CPU_MICROCODE: 6,3e,4,0 (F,M,S,R)  SIG: 427'00000000 (cache) 427'00000000 (init)

CUSTOMER_CRASH_COUNT:  1

DEFAULT_BUCKET_ID:  WIN7_DRIVER_FAULT_SERVER

PROCESS_NAME:  System

CURRENT_IRQL:  0

ERROR_CODE: (NTSTATUS) 0xc0000005 - The instruction at 0x%08lx referenced memory at 0x%08lx. The memory could not be %s.

EXCEPTION_CODE_STR:  c0000005

EXCEPTION_PARAMETER1:  0000000000000000

EXCEPTION_PARAMETER2:  ffffffffffffffff

FOLLOWUP_IP:

vsepflt+13478

fffff880`015c0478 450fb7435c      movzx   r8d,word ptr [r11+5Ch]

BUGCHECK_STR:  0x7E

READ_ADDRESS: GetPointerFromAddress: unable to read from fffff800020af100

Unable to get MmSystemRangeStart

GetUlongPtrFromAddress: unable to read from fffff800020af2e8

GetUlongPtrFromAddress: unable to read from fffff800020af498

GetPointerFromAddress: unable to read from fffff800020af0b8

ffffffffffffffff

ANALYSIS_SESSION_HOST:  INOTIFYPROD

ANALYSIS_SESSION_TIME:  11-14-2017 09:03:13.0818

ANALYSIS_VERSION: 10.0.15063.468 amd64fre

LAST_CONTROL_TRANSFER:  from 0000000000000000 to fffff880015c0478

STACK_TEXT: 

fffff880`0230fa20 00000000`00000000 : fffff880`015dfac0 00000000`00380036 00000000`00000002 fffffa80`1b6fc4e0 : vsepflt+0x13478

THREAD_SHA1_HASH_MOD_FUNC:  436176c8c3db348c9a9ce1d32fc516db20c6ed24

THREAD_SHA1_HASH_MOD_FUNC_OFFSET:  6158c2b9990c53b8693e16f616e7af005ec00417

THREAD_SHA1_HASH_MOD:  436176c8c3db348c9a9ce1d32fc516db20c6ed24

FAULT_INSTR_CODE:  43b70f45

SYMBOL_STACK_INDEX:  0

SYMBOL_NAME:  vsepflt+13478

FOLLOWUP_NAME:  MachineOwner

MODULE_NAME: vsepflt

IMAGE_NAME:  vsepflt.sys

DEBUG_FLR_IMAGE_TIMESTAMP:  53aa728a

STACK_COMMAND:  .cxr 0xfffff8800230f050 ; kb

FAILURE_BUCKET_ID:  X64_0x7E_vsepflt+13478

BUCKET_ID:  X64_0x7E_vsepflt+13478

PRIMARY_PROBLEM_CLASS:  X64_0x7E_vsepflt+13478

TARGET_TIME:  2017-11-14T01:13:01.000Z

OSBUILD:  7601

OSSERVICEPACK:  1000

SERVICEPACK_NUMBER: 0

OS_REVISION: 0

SUITE_MASK:  272

PRODUCT_TYPE:  3

OSPLATFORM_TYPE:  x64

OSNAME:  Windows 7

OSEDITION:  Windows 7 Server (Service Pack 1) TerminalServer SingleUserTS

OS_LOCALE: 

USER_LCID:  0

OSBUILD_TIMESTAMP:  2017-07-07 20:26:26

BUILDDATESTAMP_STR:  170707-0600

BUILDLAB_STR:  win7sp1_ldr

BUILDOSVER_STR:  6.1.7601.23864.amd64fre.win7sp1_ldr.170707-0600

ANALYSIS_SESSION_ELAPSED_TIME:  5ca

ANALYSIS_SOURCE:  KM

FAILURE_ID_HASH_STRING:  km:x64_0x7e_vsepflt+13478

FAILURE_ID_HASH:  {67429edc-8db0-adaf-cbf3-2c17995cb15e}