Hi,
we have faulting application errors every ~5-10 minutes on our 2016 servers with vm tools:
Faulting application name: WmiApSrv.exe, version: 10.0.14393.0, time stamp: 0x57899ac8
Faulting module name: vmStatsProvider.dll, version: 10.0.9.55972, time stamp: 0x57426eff
Exception code: 0xc0000005
Fault offset: 0x0000000000001d9a
Faulting process id: 0x14a4
Faulting application start time: 0x01d2f499b7b2dbde
Faulting application path: C:\WINDOWS\system32\wbem\WmiApSrv.exe
Faulting module path: C:\Program Files\VMware\VMware Tools\vmStatsProvider\win64\vmStatsProvider.dll
Report Id: 42c0409f-03cd-4033-bedc-9194b77373b4
Faulting package full name:
Faulting package-relative application ID:
Tried different vmtools versions including newest 10.1.7, still the same problem.
Server has latest security and feature updates, also .net 4.7
I've tried to look at procdump, however I don't have any experience with windbg therefore I'm unable to conclude anything from the output. Some info from the dump:
AULTING_IP:
vmStatsProvider+1d9a
00007ffd`1bf41d9a ff5010 call qword ptr [rax+10h]
EXCEPTION_RECORD: ffffffffffffffff -- (.exr 0xffffffffffffffff)
ExceptionAddress: 00007ffd1bf41d9a (vmStatsProvider+0x0000000000001d9a)
ExceptionCode: c0000005 (Access violation)
ExceptionFlags: 00000000
NumberParameters: 2
Parameter[0]: 0000000000000000
Parameter[1]: 00007ffd241fe520
Attempt to read from address 00007ffd241fe520
PROCESS_NAME: WmiApSrv.exe
ERROR_CODE: (NTSTATUS) 0xc0000005 - The instruction at "0x%08lx" referenced memory at "0x%08lx". The memory could not be "%s".
EXCEPTION_CODE: (NTSTATUS) 0xc0000005 - The instruction at "0x%08lx" referenced memory at "0x%08lx". The memory could not be "%s".
EXCEPTION_PARAMETER1: 0000000000000000
EXCEPTION_PARAMETER2: 00007ffd241fe520
READ_ADDRESS: 00007ffd241fe520
FOLLOWUP_IP:
vmStatsProvider+1d9a
00007ffd`1bf41d9a ff5010 call qword ptr [rax+10h]
NTGLOBALFLAG: 0
APPLICATION_VERIFIER_FLAGS: 0
LAST_CONTROL_TRANSFER: from 00007ffd1bf428c5 to 00007ffd1bf41d9a
ADDITIONAL_DEBUG_TEXT: Followup set based on attribute [UnloadedModule_Arch_AX] from Frame:[0] on thread:[188c] ; Followup set based on attribute [Is_ChosenCrashFollowupThread] from Frame:[0] on thread:[PSEUDO_THREAD] ; Followup set based on attribute [ip_is_call_value_Arch_ax] from Frame:[0] on thread:[188c] ; Followup set based on attribute [Is_OriginalExceptionThread] from Frame:[0] on thread:[188c]
FAULTING_THREAD: ffffffffffffffff
BUGCHECK_STR: APPLICATION_FAULT_INVALID_POINTER_READ_CALL_LEAK
PRIMARY_PROBLEM_CLASS: INVALID_POINTER_READ_CALL_LEAK
DEFAULT_BUCKET_ID: INVALID_POINTER_READ_CALL_LEAK
STACK_TEXT:
00000000`00000000 00000000`00000000 fastprox!CWbemInstance::_vftable_+0x0
STACK_COMMAND: .ecxr ; ~~[188c] ; .frame 0 ; ** Pseudo Context ** ; kb
SYMBOL_STACK_INDEX: 0
SYMBOL_NAME: fastprox!CWbemInstance::_vftable_
FOLLOWUP_NAME: MachineOwner
MODULE_NAME: fastprox
IMAGE_NAME: fastprox.dll
DEBUG_FLR_IMAGE_TIMESTAMP: 0
FAILURE_BUCKET_ID: INVALID_POINTER_READ_CALL_LEAK_c0000005_fastprox.dll!CWbemInstance::_vftable_
BUCKET_ID: X64_APPLICATION_FAULT_INVALID_POINTER_READ_CALL_LEAK_fastprox!CWbemInstance::_vftable_
WATSON_STAGEONE_URL: http://watson.microsoft.com/StageOne/WmiApSrv_exe/6_2_14393_0/57899ac8/vmStatsProvider_dll/10_0_9_55...
and some more:
This dump file has an exception of interest stored in it.
The stored exception information can be accessed via .ecxr.
(688.188c): Access violation - code c0000005 (first/second chance not available)
*** ERROR: Symbol file could not be found. Defaulted to export symbols for vmStatsProvider.dll -
vmStatsProvider+0x1d9a:
00007ffd`1bf41d9a ff5010 call qword ptr [rax+10h] ds:00007ffd`241fe520=????????????????
0:004> k
# Child-SP RetAddr Call Site
00 0000001c`d977f500 00007ffd`1bf428c5 vmStatsProvider+0x1d9a
01 0000001c`d977f540 00007ffd`1bf42403 vmStatsProvider+0x28c5
02 0000001c`d977f580 00007ffd`1bf56190 vmStatsProvider+0x2403
03 0000001c`d977f5d0 00007ffd`1bf5171f vmStatsProvider!DllUnregisterServer+0x10830
04 0000001c`d977f600 00007ffd`1bf51965 vmStatsProvider!DllUnregisterServer+0xbdbf
05 0000001c`d977f640 00007ffd`378ea35f vmStatsProvider!DllUnregisterServer+0xc005
06 0000001c`d977f680 00007ffd`378d3d2d ntdll!LdrpCallInitRoutine+0x4b
07 0000001c`d977f6e0 00007ffd`378d1621 ntdll!LdrpProcessDetachNode+0xf5
08 0000001c`d977f7b0 00007ffd`3790eefc ntdll!LdrpUnloadNode+0x49
09 0000001c`d977f800 00007ffd`3790ee24 ntdll!LdrpDecrementModuleLoadCountEx+0xc4
0a 0000001c`d977f850 00007ffd`347602cd ntdll!LdrUnloadDll+0x94
0b 0000001c`d977f880 00007ffd`367f5b38 KERNELBASE!FreeLibrary+0x1d
0c 0000001c`d977f8b0 00007ffd`367f5a56 combase!CClassCache::CDllPathEntry::CFinishObject::Finish+0x28 [d:\rs1\onecore\com\combase\objact\dllcache.cxx @ 3368]
0d 0000001c`d977f8e0 00007ffd`3679f1b4 combase!CClassCache::CFinishComposite::Finish+0x56 [d:\rs1\onecore\com\combase\objact\dllcache.cxx @ 3478]
0e 0000001c`d977f910 00007ffd`3679c4cb combase!CClassCache::CleanUpDllsForProcess+0x124 [d:\rs1\onecore\com\combase\objact\dllcache.cxx @ 6969]
0f (Inline Function) --------`-------- combase!CCCleanUpDllsForProcess+0xe [d:\rs1\onecore\com\combase\objact\dllcache.cxx @ 8680]
10 0000001c`d977fb10 00007ffd`367ded00 combase!ProcessUninitialize+0x1c3 [d:\rs1\onecore\com\combase\class\compobj.cxx @ 2236]
11 0000001c`d977fb50 00007ffd`367deb23 combase!DecrementProcessInitializeCount+0x44 [d:\rs1\onecore\com\combase\class\compobj.cxx @ 992]
12 0000001c`d977fb80 00007ffd`367db835 combase!wCoUninitialize+0x87 [d:\rs1\onecore\com\combase\class\compobj.cxx @ 4115]
13 0000001c`d977fbc0 00007ff7`5bd44351 combase!CoUninitialize+0x85 [d:\rs1\onecore\com\combase\class\compobj.cxx @ 3946]
14 0000001c`d977fbf0 00007ff7`5bd45472 WmiApSrv!WinRun+0x89
15 0000001c`d977fc30 00007ffd`35223dd2 WmiApSrv!WmiAdapterService::_ServiceMain+0x52
16 0000001c`d977fc60 00007ffd`37728364 sechost!ScSvcctrlThreadA+0x22
17 0000001c`d977fc90 00007ffd`379270d1 kernel32!BaseThreadInitThunk+0x14
18 0000001c`d977fcc0 00000000`00000000 ntdll!RtlUserThreadStart+0x21
What is the version and build number of ESXI you are using.
Hi,
it is ESXi, 6.0.0, 5224934
I believe you already had correlated tools version with the installed build number (10.0.9)
Uninstall and reinstall VMware tools with Custom Install, Uncheck WMI performance logging while choosing feature and monitor for sometime.
https://packages.vmware.com/tools/esx/6.0u3/windows/x64/VMware-tools-windows-10.0.9-3917699.iso
What if we want to use WMI performance logging? This does not sound like a solution in that case.
Of Course this is not a solution, neither there is much documentation for Server 2016 yet so this can help to reproduce the cause.
I can confirm that the error is not logged when wmi performance logging is not installed.
Were you always having this issue since the server was deployed and VMware Tools installed or did something change recently? I've been testing with a 2016 server with WMI and haven't been able to reproduce the error.
We were not using server 2016 up to this point, therefore we never had it working previously.
Install with VMWareToolsSetup.exe incase if you are installing with VMwareTools64.msi and check.
I was using Setup64.exe.
Setup.exe does not run on 64bit OS.
Hi,
we have on serveral servers (around 50) the same error.
Have anyone found a solution?
It would be great, if there is another option.
Disabling wmi in the setup dialog of VMware Tools, could only be a workaround, but no solution!
Thanks + Best regards
Stefan
We have few servers that do not have this issue, the only difference I was able to identify between these servers are that VMs with errors are running in BIOS mode, and those without errors are running in UEFI, anyone else having the problem can check if they are running UEFI or BIOS?
My VM with no errors is running in BIOS.
It seems this problem affects only the VMs that had OS installed via SCCM (or cloned from template which was isntalled via SCCM). If you install from ISO, this problem does not appear.
Do anyone have a fix for this ?
I have tried all the solutions I could find on this, but it keeps going on with WMI appcrash and spams the log with eventcode 1001. I have this on all the win 2016 servers, and yes they are deployed from a template, but I still need a solution 🙂
This issue is fixed since VMware Tools 10.1.10, please refer to VMware Tools 10.1.10 Release Notes
Please download the latest available VMware Tools 10.1.15 to get the fix.
We have the same issue,
Server 2016; vmstatsprovider.dll is crashing as shown in WER logs and application log.
Other OS do not exhibit this issue.
We updated to tools 10.1.10 from 10.1.7 to no avail - still causing an issue.
We also tried a full uninstall of 10.1.7, restart and install of 10.1.10 on one Server.
Oddly both in event log and WER logs vmstatsprovider.dll looks to still be registered as 10.1.7 despite the file version being 10.1.10 on the machine - this is the same on all we have tried to update.
Sig[0].Name=Application Name
Sig[0].Value=WmiApSrv.exe
Sig[1].Name=Application Version
Sig[1].Value=10.0.14393.0
Sig[2].Name=Application Timestamp
Sig[2].Value=57899ac8
Sig[3].Name=Fault Module Name
Sig[3].Value=vmStatsProvider.dll
Sig[4].Name=Fault Module Version
Sig[4].Value=10.1.7.61298
Sig[5].Name=Fault Module Timestamp
Sig[5].Value=59163c3a
Sig[6].Name=Exception Code
Sig[6].Value=c0000005
Sig[7].Name=Exception Offset
Sig[7].Value=0000000000001d9a
DynamicSig[1].Name=OS Version
DynamicSig[1].Value=10.0.14393.2.0.0.272.7
Should I log with VMware and if so has anyone else done this and had a resolution?