VMware Cloud Community
Travis_83
Enthusiast
Enthusiast

Win SRVs 2012 R2 VMs Crashing/Rebooting Since Upgrade to ESXi 6.0 (Dump file: Probably caused by : vsepflt.sys)

Hi,

We upgraded hosts to esxi 6.0 over the weekend and the vmware tools were upgraded on the vms, since monday a few of our file servers have been rebooting/crashing randomly. To stop the behavior I have to unload vsepflt.sys. I thought initially it was an issue with vShield but after contacting their support they say the problem lies with vmware tools and i believe it is the introspection driver that gets installed.

Below is the crashdump - has anyone else experienced this issue and what did you do to resolve it, ofcourse we need to have this particular driver installed as vShield with Sophos is part of our setup.

*******************************************************************************

*                                                                             *

*                        Bugcheck Analysis                                    *

*                                                                             *

*******************************************************************************

Use !analyze -v to get detailed debugging information.

BugCheck 1000007E, {ffffffffc0000005, fffff803d1ced478, ffffd00021085368, ffffd00021084b70}

*** WARNING: Unable to verify timestamp for vsepflt.sys

*** ERROR: Module load completed but symbols could not be loaded for vsepflt.sys

Probably caused by : vsepflt.sys ( vsepflt+cc8c )

Followup:     MachineOwner

---------

0: kd> !analyze -v

*******************************************************************************

*                                                                             *

*                        Bugcheck Analysis                                    *

*                                                                             *

*******************************************************************************

SYSTEM_THREAD_EXCEPTION_NOT_HANDLED_M (1000007e)

This is a very common bugcheck.  Usually the exception address pinpoints

the driver/function that caused the problem.  Always note this address

as well as the link date of the driver/image that contains this address.

Some common problems are exception code 0x80000003.  This means a hard

coded breakpoint or assertion was hit, but this system was booted

/NODEBUG.  This is not supposed to happen as developers should never have

hardcoded breakpoints in retail code, but ...

If this happens, make sure a debugger gets connected, and the

system is booted /DEBUG.  This will let us see why this breakpoint is

happening.

Arguments:

Arg1: ffffffffc0000005, The exception code that was not handled

Arg2: fffff803d1ced478, The address that the exception occurred at

Arg3: ffffd00021085368, Exception Record Address

Arg4: ffffd00021084b70, Context Record Address

Debugging Details:

------------------

DUMP_CLASS: 1

DUMP_QUALIFIER: 400

BUILD_VERSION_STRING:  9600.18185.amd64fre.winblue_ltsb.151230-0600

SYSTEM_MANUFACTURER:  VMware, Inc.

VIRTUAL_MACHINE:  VMware

SYSTEM_PRODUCT_NAME:  VMware Virtual Platform

SYSTEM_VERSION:  None

BIOS_VENDOR:  Phoenix Technologies LTD

BIOS_VERSION:  6.00

BIOS_DATE:  09/21/2015

BASEBOARD_MANUFACTURER:  Intel Corporation

BASEBOARD_PRODUCT:  440BX Desktop Reference Platform

BASEBOARD_VERSION:  None

DUMP_TYPE:  2

BUGCHECK_P1: ffffffffc0000005

BUGCHECK_P2: fffff803d1ced478

BUGCHECK_P3: ffffd00021085368

BUGCHECK_P4: ffffd00021084b70

EXCEPTION_CODE: (NTSTATUS) 0xc0000005 - The instruction at 0x%p referenced memory at 0x%p. The memory could not be %s.

FAULTING_IP:

nt!IoRetrievePriorityInfo+104

fffff803`d1ced478 8b4040          mov     eax,dword ptr [rax+40h]

EXCEPTION_RECORD:  ffffd00021085368 -- (.exr 0xffffd00021085368)

ExceptionAddress: fffff803d1ced478 (nt!IoRetrievePriorityInfo+0x0000000000000104)

   ExceptionCode: c0000005 (Access violation)

  ExceptionFlags: 00000000

NumberParameters: 2

   Parameter[0]: 0000000000000000

   Parameter[1]: ffffffffffffffff

Attempt to read from address ffffffffffffffff

CONTEXT:  ffffd00021084b70 -- (.cxr 0xffffd00021084b70)

rax=31c29d836eec0337 rbx=ffffe00177d57880 rcx=ffffe00178b516b0

rdx=ffffe0017826ed20 rsi=ffffd00021085638 rdi=0000000000000002

rip=fffff803d1ced478 rsp=ffffd000210855a0 rbp=0000000000000000

r8=ffffe00177d57880  r9=ffffd00021085638 r10=0000000000000000

r11=ffffe00178b519c0 r12=ffffe0017635a580 r13=0000000000000000

r14=0000000000100000 r15=0000000000000000

iopl=0         nv up ei pl nz na pe nc

cs=0010  ss=0018  ds=002b  es=002b  fs=0053  gs=002b             efl=00010202

nt!IoRetrievePriorityInfo+0x104:

fffff803`d1ced478 8b4040          mov     eax,dword ptr [rax+40h] ds:002b:31c29d83`6eec0377=????????

Resetting default scope

CPU_COUNT: 2

CPU_MHZ: 95a

CPU_VENDOR:  GenuineIntel

CPU_FAMILY: 6

CPU_MODEL: 2d

CPU_STEPPING: 7

CPU_MICROCODE: 6,2d,7,0 (F,M,S,R)  SIG: 710'00000000 (cache) 710'00000000 (init)

CUSTOMER_CRASH_COUNT:  1

DEFAULT_BUCKET_ID:  WIN8_DRIVER_FAULT_SERVER

PROCESS_NAME:  System

CURRENT_IRQL:  0

ERROR_CODE: (NTSTATUS) 0xc0000005 - The instruction at 0x%p referenced memory at 0x%p. The memory could not be %s.

EXCEPTION_CODE_STR:  c0000005

EXCEPTION_PARAMETER1:  0000000000000000

EXCEPTION_PARAMETER2:  ffffffffffffffff

READ_ADDRESS: GetPointerFromAddress: unable to read from fffff803d1f67138

Unable to get MmSystemRangeStart

ffffffffffffffff

FOLLOWUP_IP:

vsepflt+cc8c

fffff801`c2454c8c ??              ???

BUGCHECK_STR:  AV

ANALYSIS_SESSION_HOST:  LS-DH-DESK12

ANALYSIS_SESSION_TIME:  02-24-2016 11:20:42.0830

ANALYSIS_VERSION: 10.0.10586.567 amd64fre

LAST_CONTROL_TRANSFER:  from fffff801c20a3140 to fffff803d1ced478

STACK_TEXT: 

ffffd000`210855a0 fffff801`c20a3140 : ffffe001`7857fd00 ffffd000`21085619 ffffe001`7857fdd8 00000000`00000000 : nt!IoRetrievePriorityInfo+0x104

ffffd000`210855d0 fffff801`c20cba72 : ffffe001`7857fd00 ffffe001`7857fdd8 ffffe001`787f7000 ffffe001`7635a580 : fltmgr!FltPerformSynchronousIo+0x270

ffffd000`21085680 fffff801`c2454c8c : 00000000`00001000 ffffd000`21085790 ffffe001`7826ed00 00000000`00000000 : fltmgr!FltQuerySecurityObject+0x52

ffffd000`210856c0 00000000`00001000 : ffffd000`21085790 ffffe001`7826ed00 00000000`00000000 ffffe001`00001000 : vsepflt+0xcc8c

ffffd000`210856c8 ffffd000`21085790 : ffffe001`7826ed00 00000000`00000000 ffffe001`00001000 ffffd000`210856f0 : 0x1000

ffffd000`210856d0 ffffe001`7826ed00 : 00000000`00000000 ffffe001`00001000 ffffd000`210856f0 ffffd000`00000000 : 0xffffd000`21085790

ffffd000`210856d8 00000000`00000000 : ffffe001`00001000 ffffd000`210856f0 ffffd000`00000000 00000009`00000010 : 0xffffe001`7826ed00

THREAD_SHA1_HASH_MOD_FUNC:  db0c543414dac3ac330ffd31977941191bbf4355

THREAD_SHA1_HASH_MOD_FUNC_OFFSET:  9ddaef6e8d93eab022d6b0a345c3611f71b42281

THREAD_SHA1_HASH_MOD:  fe7fdb831712cbe5617d7c84a311747127b2b6dc

SYMBOL_STACK_INDEX:  3

SYMBOL_NAME:  vsepflt+cc8c

FOLLOWUP_NAME:  MachineOwner

MODULE_NAME: vsepflt

IMAGE_NAME:  vsepflt.sys

DEBUG_FLR_IMAGE_TIMESTAMP:  55b60757

STACK_COMMAND:  .cxr 0xffffd00021084b70 ; kb

BUCKET_ID_FUNC_OFFSET:  cc8c

FAILURE_BUCKET_ID:  AV_vsepflt!Unknown_Function

BUCKET_ID:  AV_vsepflt!Unknown_Function

PRIMARY_PROBLEM_CLASS:  AV_vsepflt!Unknown_Function

TARGET_TIME:  2016-02-24T03:30:29.000Z

OSBUILD:  9600

OSSERVICEPACK:  0

SERVICEPACK_NUMBER: 0

OS_REVISION: 0

SUITE_MASK:  272

PRODUCT_TYPE:  3

OSPLATFORM_TYPE:  x64

OSNAME:  Windows 8.1

OSEDITION:  Windows 8.1 Server TerminalServer SingleUserTS

OS_LOCALE: 

USER_LCID:  0

OSBUILD_TIMESTAMP:  2015-12-30 14:49:56

BUILDDATESTAMP_STR:  151230-0600

BUILDLAB_STR:  winblue_ltsb

BUILDOSVER_STR:  6.3.9600.18185.amd64fre.winblue_ltsb.151230-0600

ANALYSIS_SESSION_ELAPSED_TIME: 3bb

ANALYSIS_SOURCE:  KM

FAILURE_ID_HASH_STRING:  km:av_vsepflt!unknown_function

FAILURE_ID_HASH:  {c13bc55f-f0e5-ed3d-0c0d-0f4115ea2f82}

Followup:     MachineOwner

---------

Tags (1)
0 Kudos
4 Replies
Travis_83
Enthusiast
Enthusiast

Hi guys,

Anyone experiencing a similar issue?

Is it possible to load an older version of the vsepflt file - we have a platform still on esxi 5.5 which runs an older version of file and wondering if can unload existing problem sys file and load older version or does it need to be an installer of some kind..

0 Kudos
MKguy
Virtuoso
Virtuoso

Have you tried the latest 6.0 U1 Tools or the standalone 10.x Tools?

https://my.vmware.com/group/vmware/details?downloadGroup=VMTOOLS1005&productId=491

http://pubs.vmware.com/Release_Notes/en/vmwaretools/1005/vmware-tools-1005-release-notes.html

You can rollback to an older Tools version on the problematic VMs, for example to 5.5 Tools and see if that helps. The VM Tools status may show as outdated, but it's supported by VMware:

http://partnerweb.vmware.com/comp_guide2/sim/interop_matrix.php#interop&1=&39=

-- http://alpacapowered.wordpress.com
0 Kudos
joeyalex82
Contributor
Contributor

I have the same problem after upgrading to the latest release of 6.0.  The latest tools release doesn't fix it.

Be careful rolling back the tools, as I ran into this lovely problem (VMware KB: Cannot save Microsoft Office files to a shared directory on a virtual machine protected b...) on our file server.  (People won't be able to save Excel files on network shares.)

I currently have the AV off on the server, waiting to hear back from VMware support...

0 Kudos
Travis_83
Enthusiast
Enthusiast

Thanks Joey,

We are currently in the midst of trying to resolve this issue. We have also had to resort to turning off AV.

Problem is we don't have VMWare Support. It would be greatly appreciated if you could relay any solution VMWare provide.

Kind Regards,

Travis

0 Kudos