VMware Cloud Community
complexxL9
Contributor
Contributor

VMtools error in Application events on server 2016

Hi,

we have faulting application errors every ~5-10 minutes on our 2016 servers with vm tools:

Faulting application name: WmiApSrv.exe, version: 10.0.14393.0, time stamp: 0x57899ac8
Faulting module name: vmStatsProvider.dll, version: 10.0.9.55972, time stamp: 0x57426eff
Exception code: 0xc0000005
Fault offset: 0x0000000000001d9a
Faulting process id: 0x14a4
Faulting application start time: 0x01d2f499b7b2dbde
Faulting application path: C:\WINDOWS\system32\wbem\WmiApSrv.exe
Faulting module path: C:\Program Files\VMware\VMware Tools\vmStatsProvider\win64\vmStatsProvider.dll
Report Id: 42c0409f-03cd-4033-bedc-9194b77373b4
Faulting package full name:
Faulting package-relative application ID:

Tried different vmtools versions including newest 10.1.7, still the same problem.

Server has latest security and feature updates, also .net 4.7

I've tried to look at procdump, however I don't have any experience with windbg therefore I'm unable to conclude anything from the output. Some info from the dump:

AULTING_IP:
vmStatsProvider+1d9a
00007ffd`1bf41d9a ff5010          call    qword ptr [rax+10h]

EXCEPTION_RECORD:  ffffffffffffffff -- (.exr 0xffffffffffffffff)
ExceptionAddress: 00007ffd1bf41d9a (vmStatsProvider+0x0000000000001d9a)
   ExceptionCode: c0000005 (Access violation)
  ExceptionFlags: 00000000
NumberParameters: 2
   Parameter[0]: 0000000000000000
   Parameter[1]: 00007ffd241fe520
Attempt to read from address 00007ffd241fe520

PROCESS_NAME:  WmiApSrv.exe

ERROR_CODE: (NTSTATUS) 0xc0000005 - The instruction at "0x%08lx" referenced memory at "0x%08lx". The memory could not be "%s".

EXCEPTION_CODE: (NTSTATUS) 0xc0000005 - The instruction at "0x%08lx" referenced memory at "0x%08lx". The memory could not be "%s".

EXCEPTION_PARAMETER1:  0000000000000000

EXCEPTION_PARAMETER2:  00007ffd241fe520

READ_ADDRESS:  00007ffd241fe520

FOLLOWUP_IP:
vmStatsProvider+1d9a
00007ffd`1bf41d9a ff5010          call    qword ptr [rax+10h]

NTGLOBALFLAG:  0

APPLICATION_VERIFIER_FLAGS:  0

LAST_CONTROL_TRANSFER:  from 00007ffd1bf428c5 to 00007ffd1bf41d9a

ADDITIONAL_DEBUG_TEXT:  Followup set based on attribute [UnloadedModule_Arch_AX] from Frame:[0] on thread:[188c] ; Followup set based on attribute [Is_ChosenCrashFollowupThread] from Frame:[0] on thread:[PSEUDO_THREAD] ; Followup set based on attribute [ip_is_call_value_Arch_ax] from Frame:[0] on thread:[188c] ; Followup set based on attribute [Is_OriginalExceptionThread] from Frame:[0] on thread:[188c]

FAULTING_THREAD:  ffffffffffffffff

BUGCHECK_STR:  APPLICATION_FAULT_INVALID_POINTER_READ_CALL_LEAK

PRIMARY_PROBLEM_CLASS:  INVALID_POINTER_READ_CALL_LEAK

DEFAULT_BUCKET_ID:  INVALID_POINTER_READ_CALL_LEAK

STACK_TEXT: 
00000000`00000000 00000000`00000000 fastprox!CWbemInstance::_vftable_+0x0


STACK_COMMAND:  .ecxr ; ~~[188c] ; .frame 0 ; ** Pseudo Context ** ; kb

SYMBOL_STACK_INDEX:  0

SYMBOL_NAME:  fastprox!CWbemInstance::_vftable_

FOLLOWUP_NAME:  MachineOwner

MODULE_NAME: fastprox

IMAGE_NAME:  fastprox.dll

DEBUG_FLR_IMAGE_TIMESTAMP:  0

FAILURE_BUCKET_ID:  INVALID_POINTER_READ_CALL_LEAK_c0000005_fastprox.dll!CWbemInstance::_vftable_

BUCKET_ID:  X64_APPLICATION_FAULT_INVALID_POINTER_READ_CALL_LEAK_fastprox!CWbemInstance::_vftable_

WATSON_STAGEONE_URL:  http://watson.microsoft.com/StageOne/WmiApSrv_exe/6_2_14393_0/57899ac8/vmStatsProvider_dll/10_0_9_55...

and some more:

This dump file has an exception of interest stored in it.
The stored exception information can be accessed via .ecxr.
(688.188c): Access violation - code c0000005 (first/second chance not available)
*** ERROR: Symbol file could not be found.  Defaulted to export symbols for vmStatsProvider.dll -
vmStatsProvider+0x1d9a:
00007ffd`1bf41d9a ff5010          call    qword ptr [rax+10h] ds:00007ffd`241fe520=????????????????
0:004> k
# Child-SP          RetAddr           Call Site
00 0000001c`d977f500 00007ffd`1bf428c5 vmStatsProvider+0x1d9a
01 0000001c`d977f540 00007ffd`1bf42403 vmStatsProvider+0x28c5
02 0000001c`d977f580 00007ffd`1bf56190 vmStatsProvider+0x2403
03 0000001c`d977f5d0 00007ffd`1bf5171f vmStatsProvider!DllUnregisterServer+0x10830
04 0000001c`d977f600 00007ffd`1bf51965 vmStatsProvider!DllUnregisterServer+0xbdbf
05 0000001c`d977f640 00007ffd`378ea35f vmStatsProvider!DllUnregisterServer+0xc005
06 0000001c`d977f680 00007ffd`378d3d2d ntdll!LdrpCallInitRoutine+0x4b
07 0000001c`d977f6e0 00007ffd`378d1621 ntdll!LdrpProcessDetachNode+0xf5
08 0000001c`d977f7b0 00007ffd`3790eefc ntdll!LdrpUnloadNode+0x49
09 0000001c`d977f800 00007ffd`3790ee24 ntdll!LdrpDecrementModuleLoadCountEx+0xc4
0a 0000001c`d977f850 00007ffd`347602cd ntdll!LdrUnloadDll+0x94
0b 0000001c`d977f880 00007ffd`367f5b38 KERNELBASE!FreeLibrary+0x1d
0c 0000001c`d977f8b0 00007ffd`367f5a56 combase!CClassCache::CDllPathEntry::CFinishObject::Finish+0x28 [d:\rs1\onecore\com\combase\objact\dllcache.cxx @ 3368]
0d 0000001c`d977f8e0 00007ffd`3679f1b4 combase!CClassCache::CFinishComposite::Finish+0x56 [d:\rs1\onecore\com\combase\objact\dllcache.cxx @ 3478]
0e 0000001c`d977f910 00007ffd`3679c4cb combase!CClassCache::CleanUpDllsForProcess+0x124 [d:\rs1\onecore\com\combase\objact\dllcache.cxx @ 6969]
0f (Inline Function) --------`-------- combase!CCCleanUpDllsForProcess+0xe [d:\rs1\onecore\com\combase\objact\dllcache.cxx @ 8680]
10 0000001c`d977fb10 00007ffd`367ded00 combase!ProcessUninitialize+0x1c3 [d:\rs1\onecore\com\combase\class\compobj.cxx @ 2236]
11 0000001c`d977fb50 00007ffd`367deb23 combase!DecrementProcessInitializeCount+0x44 [d:\rs1\onecore\com\combase\class\compobj.cxx @ 992]
12 0000001c`d977fb80 00007ffd`367db835 combase!wCoUninitialize+0x87 [d:\rs1\onecore\com\combase\class\compobj.cxx @ 4115]
13 0000001c`d977fbc0 00007ff7`5bd44351 combase!CoUninitialize+0x85 [d:\rs1\onecore\com\combase\class\compobj.cxx @ 3946]
14 0000001c`d977fbf0 00007ff7`5bd45472 WmiApSrv!WinRun+0x89
15 0000001c`d977fc30 00007ffd`35223dd2 WmiApSrv!WmiAdapterService::_ServiceMain+0x52
16 0000001c`d977fc60 00007ffd`37728364 sechost!ScSvcctrlThreadA+0x22
17 0000001c`d977fc90 00007ffd`379270d1 kernel32!BaseThreadInitThunk+0x14
18 0000001c`d977fcc0 00000000`00000000 ntdll!RtlUserThreadStart+0x21

Reply
0 Kudos
17 Replies
vijayrana968
Virtuoso
Virtuoso

What is the version and build number of ESXI you are using.

Reply
0 Kudos
complexxL9
Contributor
Contributor

Hi,

it is ESXi, 6.0.0, 5224934

Reply
0 Kudos
vijayrana968
Virtuoso
Virtuoso

I believe you already had correlated tools version with the installed build number (10.0.9)

Uninstall and reinstall VMware tools with Custom Install, Uncheck WMI performance logging while choosing feature and monitor for sometime.

pastedImage_1.png

https://packages.vmware.com/tools/esx/6.0u3/windows/x64/VMware-tools-windows-10.0.9-3917699.iso

Reply
0 Kudos
complexxL9
Contributor
Contributor

What if we want to use WMI performance logging? This does not sound like a solution in that case.

Reply
0 Kudos
vijayrana968
Virtuoso
Virtuoso

Of Course this is not a solution, neither there is much documentation for Server 2016 yet so this can help to reproduce the cause.

Reply
0 Kudos
complexxL9
Contributor
Contributor

I can confirm that the error is not logged when wmi performance logging is not installed.

Reply
0 Kudos
ITaaP
Enthusiast
Enthusiast

Were you always having this issue since the server was deployed and VMware Tools installed or did something change recently? I've been testing with a 2016 server with WMI and haven't been able to reproduce the error.

https://tactsol.com https://vmware.solutions
Reply
0 Kudos
complexxL9
Contributor
Contributor

We were not using server 2016 up to this point, therefore we never had it working previously.

Reply
0 Kudos
vijayrana968
Virtuoso
Virtuoso

Install with VMWareToolsSetup.exe incase if you are installing with VMwareTools64.msi and check.

Reply
0 Kudos
complexxL9
Contributor
Contributor

I was using Setup64.exe.

Setup.exe does not run on 64bit OS.

Reply
0 Kudos
0x73746566616e
Contributor
Contributor

Hi,

we have on serveral servers (around 50) the same error.

Have anyone found a solution?

It would be great, if there is another option.

Disabling wmi in the setup dialog of VMware Tools, could only be a workaround, but no solution!

Thanks + Best regards

Stefan

Reply
0 Kudos
complexxL9
Contributor
Contributor

We have few servers that do not have this issue, the only difference I was able to identify between these servers are that VMs with errors are running in BIOS mode, and those without errors are running in UEFI, anyone else having the problem can check if they are running UEFI or BIOS?

Reply
0 Kudos
ITaaP
Enthusiast
Enthusiast

My VM with no errors is running in BIOS.

https://tactsol.com https://vmware.solutions
Reply
0 Kudos
complexxL9
Contributor
Contributor

It seems this problem affects only the VMs that had OS installed via SCCM (or cloned from template which was isntalled via SCCM). If you install from ISO, this problem does not appear.

Reply
0 Kudos
Elkjer
Contributor
Contributor

Do anyone have a fix for this ?
I have tried all the solutions I could find on this, but it keeps going on with WMI appcrash and spams the log with eventcode 1001. I have this on all the win 2016 servers, and yes they are deployed from a template, but I still need a solution 🙂

Reply
0 Kudos
mhzhou
VMware Employee
VMware Employee

This issue is fixed since VMware Tools 10.1.10, please refer to VMware Tools 10.1.10 Release Notes

  • WMI performance adapter service fails on windows guest operating systems
    WMI performance adapter service (wmiapsrv.exe) fails on virtual machines running Windows 10 and Windows Server 2016. This issue is resolved in this release.

Please download the latest available VMware Tools 10.1.15 to get the fix.

Reply
0 Kudos
kmacl02
Contributor
Contributor

We have the same issue,

Server 2016; vmstatsprovider.dll is crashing as shown in WER logs and application log.

Other OS do not exhibit this issue.

We updated to tools 10.1.10 from 10.1.7 to no avail - still causing an issue.

We also tried a full uninstall of 10.1.7, restart and install of 10.1.10 on one Server.

Oddly both in event log and WER logs vmstatsprovider.dll looks to still be registered as 10.1.7 despite the file version being 10.1.10 on the machine - this is the same on all we have tried to update.

Sig[0].Name=Application Name

Sig[0].Value=WmiApSrv.exe

Sig[1].Name=Application Version

Sig[1].Value=10.0.14393.0

Sig[2].Name=Application Timestamp

Sig[2].Value=57899ac8

Sig[3].Name=Fault Module Name

Sig[3].Value=vmStatsProvider.dll

Sig[4].Name=Fault Module Version

Sig[4].Value=10.1.7.61298

Sig[5].Name=Fault Module Timestamp

Sig[5].Value=59163c3a

Sig[6].Name=Exception Code

Sig[6].Value=c0000005

Sig[7].Name=Exception Offset

Sig[7].Value=0000000000001d9a

DynamicSig[1].Name=OS Version

DynamicSig[1].Value=10.0.14393.2.0.0.272.7

Should I log with VMware and if so has anyone else done this and had a resolution?

Reply
0 Kudos