Hi,
We upgraded hosts to esxi 6.0 over the weekend and the vmware tools were upgraded on the vms, since monday a few of our file servers have been rebooting/crashing randomly. To stop the behavior I have to unload vsepflt.sys. I thought initially it was an issue with vShield but after contacting their support they say the problem lies with vmware tools and i believe it is the introspection driver that gets installed.
Below is the crashdump - has anyone else experienced this issue and what did you do to resolve it, ofcourse we need to have this particular driver installed as vShield with Sophos is part of our setup.
*******************************************************************************
* *
* Bugcheck Analysis *
* *
*******************************************************************************
Use !analyze -v to get detailed debugging information.
BugCheck 1000007E, {ffffffffc0000005, fffff803d1ced478, ffffd00021085368, ffffd00021084b70}
*** WARNING: Unable to verify timestamp for vsepflt.sys
*** ERROR: Module load completed but symbols could not be loaded for vsepflt.sys
Probably caused by : vsepflt.sys ( vsepflt+cc8c )
Followup: MachineOwner
---------
0: kd> !analyze -v
*******************************************************************************
* *
* Bugcheck Analysis *
* *
*******************************************************************************
SYSTEM_THREAD_EXCEPTION_NOT_HANDLED_M (1000007e)
This is a very common bugcheck. Usually the exception address pinpoints
the driver/function that caused the problem. Always note this address
as well as the link date of the driver/image that contains this address.
Some common problems are exception code 0x80000003. This means a hard
coded breakpoint or assertion was hit, but this system was booted
/NODEBUG. This is not supposed to happen as developers should never have
hardcoded breakpoints in retail code, but ...
If this happens, make sure a debugger gets connected, and the
system is booted /DEBUG. This will let us see why this breakpoint is
happening.
Arguments:
Arg1: ffffffffc0000005, The exception code that was not handled
Arg2: fffff803d1ced478, The address that the exception occurred at
Arg3: ffffd00021085368, Exception Record Address
Arg4: ffffd00021084b70, Context Record Address
Debugging Details:
------------------
DUMP_CLASS: 1
DUMP_QUALIFIER: 400
BUILD_VERSION_STRING: 9600.18185.amd64fre.winblue_ltsb.151230-0600
SYSTEM_MANUFACTURER: VMware, Inc.
VIRTUAL_MACHINE: VMware
SYSTEM_PRODUCT_NAME: VMware Virtual Platform
SYSTEM_VERSION: None
BIOS_VENDOR: Phoenix Technologies LTD
BIOS_VERSION: 6.00
BIOS_DATE: 09/21/2015
BASEBOARD_MANUFACTURER: Intel Corporation
BASEBOARD_PRODUCT: 440BX Desktop Reference Platform
BASEBOARD_VERSION: None
DUMP_TYPE: 2
BUGCHECK_P1: ffffffffc0000005
BUGCHECK_P2: fffff803d1ced478
BUGCHECK_P3: ffffd00021085368
BUGCHECK_P4: ffffd00021084b70
EXCEPTION_CODE: (NTSTATUS) 0xc0000005 - The instruction at 0x%p referenced memory at 0x%p. The memory could not be %s.
FAULTING_IP:
nt!IoRetrievePriorityInfo+104
fffff803`d1ced478 8b4040 mov eax,dword ptr [rax+40h]
EXCEPTION_RECORD: ffffd00021085368 -- (.exr 0xffffd00021085368)
ExceptionAddress: fffff803d1ced478 (nt!IoRetrievePriorityInfo+0x0000000000000104)
ExceptionCode: c0000005 (Access violation)
ExceptionFlags: 00000000
NumberParameters: 2
Parameter[0]: 0000000000000000
Parameter[1]: ffffffffffffffff
Attempt to read from address ffffffffffffffff
CONTEXT: ffffd00021084b70 -- (.cxr 0xffffd00021084b70)
rax=31c29d836eec0337 rbx=ffffe00177d57880 rcx=ffffe00178b516b0
rdx=ffffe0017826ed20 rsi=ffffd00021085638 rdi=0000000000000002
rip=fffff803d1ced478 rsp=ffffd000210855a0 rbp=0000000000000000
r8=ffffe00177d57880 r9=ffffd00021085638 r10=0000000000000000
r11=ffffe00178b519c0 r12=ffffe0017635a580 r13=0000000000000000
r14=0000000000100000 r15=0000000000000000
iopl=0 nv up ei pl nz na pe nc
cs=0010 ss=0018 ds=002b es=002b fs=0053 gs=002b efl=00010202
nt!IoRetrievePriorityInfo+0x104:
fffff803`d1ced478 8b4040 mov eax,dword ptr [rax+40h] ds:002b:31c29d83`6eec0377=????????
Resetting default scope
CPU_COUNT: 2
CPU_MHZ: 95a
CPU_VENDOR: GenuineIntel
CPU_FAMILY: 6
CPU_MODEL: 2d
CPU_STEPPING: 7
CPU_MICROCODE: 6,2d,7,0 (F,M,S,R) SIG: 710'00000000 (cache) 710'00000000 (init)
CUSTOMER_CRASH_COUNT: 1
DEFAULT_BUCKET_ID: WIN8_DRIVER_FAULT_SERVER
PROCESS_NAME: System
CURRENT_IRQL: 0
ERROR_CODE: (NTSTATUS) 0xc0000005 - The instruction at 0x%p referenced memory at 0x%p. The memory could not be %s.
EXCEPTION_CODE_STR: c0000005
EXCEPTION_PARAMETER1: 0000000000000000
EXCEPTION_PARAMETER2: ffffffffffffffff
READ_ADDRESS: GetPointerFromAddress: unable to read from fffff803d1f67138
Unable to get MmSystemRangeStart
ffffffffffffffff
FOLLOWUP_IP:
vsepflt+cc8c
fffff801`c2454c8c ?? ???
BUGCHECK_STR: AV
ANALYSIS_SESSION_HOST: LS-DH-DESK12
ANALYSIS_SESSION_TIME: 02-24-2016 11:20:42.0830
ANALYSIS_VERSION: 10.0.10586.567 amd64fre
LAST_CONTROL_TRANSFER: from fffff801c20a3140 to fffff803d1ced478
STACK_TEXT:
ffffd000`210855a0 fffff801`c20a3140 : ffffe001`7857fd00 ffffd000`21085619 ffffe001`7857fdd8 00000000`00000000 : nt!IoRetrievePriorityInfo+0x104
ffffd000`210855d0 fffff801`c20cba72 : ffffe001`7857fd00 ffffe001`7857fdd8 ffffe001`787f7000 ffffe001`7635a580 : fltmgr!FltPerformSynchronousIo+0x270
ffffd000`21085680 fffff801`c2454c8c : 00000000`00001000 ffffd000`21085790 ffffe001`7826ed00 00000000`00000000 : fltmgr!FltQuerySecurityObject+0x52
ffffd000`210856c0 00000000`00001000 : ffffd000`21085790 ffffe001`7826ed00 00000000`00000000 ffffe001`00001000 : vsepflt+0xcc8c
ffffd000`210856c8 ffffd000`21085790 : ffffe001`7826ed00 00000000`00000000 ffffe001`00001000 ffffd000`210856f0 : 0x1000
ffffd000`210856d0 ffffe001`7826ed00 : 00000000`00000000 ffffe001`00001000 ffffd000`210856f0 ffffd000`00000000 : 0xffffd000`21085790
ffffd000`210856d8 00000000`00000000 : ffffe001`00001000 ffffd000`210856f0 ffffd000`00000000 00000009`00000010 : 0xffffe001`7826ed00
THREAD_SHA1_HASH_MOD_FUNC: db0c543414dac3ac330ffd31977941191bbf4355
THREAD_SHA1_HASH_MOD_FUNC_OFFSET: 9ddaef6e8d93eab022d6b0a345c3611f71b42281
THREAD_SHA1_HASH_MOD: fe7fdb831712cbe5617d7c84a311747127b2b6dc
SYMBOL_STACK_INDEX: 3
SYMBOL_NAME: vsepflt+cc8c
FOLLOWUP_NAME: MachineOwner
MODULE_NAME: vsepflt
IMAGE_NAME: vsepflt.sys
DEBUG_FLR_IMAGE_TIMESTAMP: 55b60757
STACK_COMMAND: .cxr 0xffffd00021084b70 ; kb
BUCKET_ID_FUNC_OFFSET: cc8c
FAILURE_BUCKET_ID: AV_vsepflt!Unknown_Function
BUCKET_ID: AV_vsepflt!Unknown_Function
PRIMARY_PROBLEM_CLASS: AV_vsepflt!Unknown_Function
TARGET_TIME: 2016-02-24T03:30:29.000Z
OSBUILD: 9600
OSSERVICEPACK: 0
SERVICEPACK_NUMBER: 0
OS_REVISION: 0
SUITE_MASK: 272
PRODUCT_TYPE: 3
OSPLATFORM_TYPE: x64
OSNAME: Windows 8.1
OSEDITION: Windows 8.1 Server TerminalServer SingleUserTS
OS_LOCALE:
USER_LCID: 0
OSBUILD_TIMESTAMP: 2015-12-30 14:49:56
BUILDDATESTAMP_STR: 151230-0600
BUILDLAB_STR: winblue_ltsb
BUILDOSVER_STR: 6.3.9600.18185.amd64fre.winblue_ltsb.151230-0600
ANALYSIS_SESSION_ELAPSED_TIME: 3bb
ANALYSIS_SOURCE: KM
FAILURE_ID_HASH_STRING: km:av_vsepflt!unknown_function
FAILURE_ID_HASH: {c13bc55f-f0e5-ed3d-0c0d-0f4115ea2f82}
Followup: MachineOwner
---------
Hi guys,
Anyone experiencing a similar issue?
Is it possible to load an older version of the vsepflt file - we have a platform still on esxi 5.5 which runs an older version of file and wondering if can unload existing problem sys file and load older version or does it need to be an installer of some kind..
Have you tried the latest 6.0 U1 Tools or the standalone 10.x Tools?
https://my.vmware.com/group/vmware/details?downloadGroup=VMTOOLS1005&productId=491
http://pubs.vmware.com/Release_Notes/en/vmwaretools/1005/vmware-tools-1005-release-notes.html
You can rollback to an older Tools version on the problematic VMs, for example to 5.5 Tools and see if that helps. The VM Tools status may show as outdated, but it's supported by VMware:
http://partnerweb.vmware.com/comp_guide2/sim/interop_matrix.php#interop&1=&39=
I have the same problem after upgrading to the latest release of 6.0. The latest tools release doesn't fix it.
Be careful rolling back the tools, as I ran into this lovely problem (VMware KB: Cannot save Microsoft Office files to a shared directory on a virtual machine protected b...) on our file server. (People won't be able to save Excel files on network shares.)
I currently have the AV off on the server, waiting to hear back from VMware support...
Thanks Joey,
We are currently in the midst of trying to resolve this issue. We have also had to resort to turning off AV.
Problem is we don't have VMWare Support. It would be greatly appreciated if you could relay any solution VMWare provide.
Kind Regards,
Travis