VMware Communities
JSNT007
Contributor
Contributor

Windows 10 crashed with bugcheck at vmx86.sys

Windows crashed many time, all with vmx86.sys. Uninstalled and re-installed back, same issue.

Windows:     Version 1809 (OS Build 17763.253)

VMWare:     15.0.2 build-10952284

crash dump file: C:\WINDOWS\MEMORY.DMP

uptime: 16:01:39

This was probably caused by the following module: vmx86.sys (vmx86+0x100a)

Bugcheck code: 0x101 (0x5, 0x0, 0xFFFFC781FAAA6180, 0x20)

Error: CLOCK_WATCHDOG_TIMEOUT

file path: C:\WINDOWS\system32\drivers\vmx86.sys

product: VMware kernel driver

company: VMware, Inc.

description: VMware kernel driver

Bug check description: This indicates that an expected clock interrupt on a secondary processor, in a multi-processor system, was not received within the allocated interval. This can be caused by non-responding hardware or by a overheated CPU (thermal issue).

A third party driver was identified as the probable root cause of this system error. It is suggested you look for an update for the following driver: vmx86.sys (VMware kernel driver, VMware, Inc.).

Google query: vmx86.sys VMware, Inc. CLOCK_WATCHDOG_TIMEOUT

Reply
0 Kudos
11 Replies
hahakiki2010
VMware Employee
VMware Employee

thanks for your report. We have been aware of this issue, and will look into this issue internally.

Reply
0 Kudos
JSNT007
Contributor
Contributor

Thanks hahakiki2010for the quick reply.

What's the recommended workaround? What version/edition of Windows 10, and/or what version of VWS should I use to avoid such crash? I have bug check/blue screen every one or two days.

Reply
0 Kudos
hs777
Contributor
Contributor

Any news on this?  One year later, still seeing same problem, which prevents installation of VMWare Workstation Pro 15.5.1.

Thanks.

Reply
0 Kudos
hs777
Contributor
Contributor

Looks like 15.5.2 fixed this problem (at least for me).

Reply
0 Kudos
ChaosEnergy
Contributor
Contributor

Same herewith latest 15.5.2
*******************************************************************************
* *
* Bugcheck Analysis *
* *
*******************************************************************************

CLOCK_WATCHDOG_TIMEOUT (101)
An expected clock interrupt was not received on a secondary processor in an
MP system within the allocated interval. This indicates that the specified
processor is hung and not processing interrupts.
Arguments:
Arg1: 0000000000000010, Clock interrupt time out interval in nominal clock ticks.
Arg2: 0000000000000000, 0.
Arg3: ffffc380677e5180, The PRCB address of the hung processor.
Arg4: 000000000000000a, The index of the hung processor.

Debugging Details:
------------------


KEY_VALUES_STRING: 1

  Key : Analysis.CPU.Sec
  Value: 4

  Key : Analysis.DebugAnalysisProvider.CPP
  Value: Create: 8007007e on CHAOS10

  Key : Analysis.DebugData
  Value: CreateObject

  Key : Analysis.DebugModel
  Value: CreateObject

  Key : Analysis.Elapsed.Sec
  Value: 15

  Key : Analysis.Memory.CommitPeak.Mb
  Value: 66

  Key : Analysis.System
  Value: CreateObject


ADDITIONAL_XML: 1

BUGCHECK_CODE: 101

BUGCHECK_P1: 10

BUGCHECK_P2: 0

BUGCHECK_P3: ffffc380677e5180

BUGCHECK_P4: a

FAULTING_PROCESSOR: a

PROCESS_NAME: vmware-vmx.exe

FAULTING_THREAD: ffff8287faf52080

STACK_TEXT: 
ffff9385`ba8844a8 fffff806`156d8308 : ffffc380`677e5180 fffff806`0ff21385 ffff8287`faf59250 00000000`00000000 : vmx86+0x100a
ffff9385`ba8844b0 fffff806`156dabd3 : ffff8287`fab3c2e0 00000000`00000000 ffffc380`68f00000 00000000`0000006b : vmx86+0x8308
ffff9385`ba8845f0 fffff806`156d1f4e : 00000000`000001b0 00000000`00000001 ffff9385`ba884978 ffff8288`014eb750 : vmx86+0xabd3
ffff9385`ba8846c0 fffff806`156d298e : 00000000`00000000 fffff806`101543a9 00000000`00000000 fffff806`0ff132e4 : vmx86+0x1f4e
ffff9385`ba8848a0 fffff806`104b24eb : 00000000`000003f4 00000000`00000001 00000000`00000000 ffff8287`fab7b510 : vmx86+0x298e
ffff9385`ba884900 fffff806`104b1db6 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : nt!IopXxxControlFile+0x71b
ffff9385`ba884a20 fffff806`0ffd3c18 : 00000000`00000000 ffff9385`00000001 00000000`000003f4 ffff9385`ba884b80 : nt!NtDeviceIoControlFile+0x56
ffff9385`ba884a90 00007fff`fa59c1a4 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : nt!KiSystemServiceCopyEnd+0x28
00000067`3c5ffaa8 00000000`00000000 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : 0x00007fff`fa59c1a4


STACK_COMMAND: .thread 0xffff8287faf52080 ; kb

SYMBOL_NAME: vmx86+100a

MODULE_NAME:
vmx86

IMAGE_NAME: vmx86.sys

BUCKET_ID_FUNC_OFFSET: 100a

FAILURE_BUCKET_ID: CLOCK_WATCHDOG_TIMEOUT_INTERRUPTS_DISABLED_vmx86!unknown_function

OS_VERSION: 10.0.18362.1

BUILDLAB_STR: 19h1_release

OSPLATFORM_TYPE: x64

OSNAME: Windows 10

FAILURE_ID_HASH: {e206f958-3924-e3cb-af92-ab296be9b5e9}

Followup: MachineOwner
---------

Reply
0 Kudos
ChaosEnergy
Contributor
Contributor

Its so annoying..loosing data cause of this

Microsoft (R) Windows Debugger Version 10.0.19528.1000 AMD64
Copyright (c) Microsoft Corporation. All rights reserved.


Loading Dump File [C:\WINDOWS\MEMORY.DMP]
Kernel Bitmap Dump File: Kernel address space is available, User address space may not be available.


************* Path validation summary **************
Response Time (ms) Location
Deferred srv*
Symbol search path is: srv*
Executable search path is:
Windows 10 Kernel Version 18362 MP (12 procs) Free x64
Product: WinNt, suite: TerminalServer SingleUserTS
18362.1.amd64fre.19h1_release.190318-1202
Machine Name:
Kernel base = 0xfffff802`55000000 PsLoadedModuleList = 0xfffff802`55448150
Debug session time: Sat Apr 11 15:12:43.332 2020 (UTC + 2:00)
System Uptime: 2 days 4:52:55.718
Loading Kernel Symbols
...............................................................
................................................................
................................................................
....................
Loading User Symbols

Loading unloaded module list
.....................
For analysis of this file, run
!analyze -v

nt!KeBugCheckEx:
fffff802`551c2380 48894c2408 mov qword ptr [rsp+8],rcx ss:0018:fffff802`5a075b10=0000000000000101
0: kd> !analyze -v
*******************************************************************************
* *
* Bugcheck Analysis *
* *
*******************************************************************************

CLOCK_WATCHDOG_TIMEOUT (101)
An expected clock interrupt was not received on a secondary processor in an
MP system within the allocated interval. This indicates that the specified
processor is hung and not processing interrupts.
Arguments:
Arg1: 0000000000000010, Clock interrupt time out interval in nominal clock ticks.
Arg2: 0000000000000000, 0.
Arg3: ffffa48079820180, The PRCB address of the hung processor.
Arg4: 0000000000000001, The index of the hung processor.

Debugging Details:
------------------


KEY_VALUES_STRING: 1

  Key : Analysis.CPU.Sec
  Value: 4

  Key : Analysis.DebugAnalysisProvider.CPP
  Value: Create: 8007007e on CHAOS10

  Key : Analysis.DebugData
  Value: CreateObject

  Key : Analysis.DebugModel
  Value: CreateObject

  Key : Analysis.Elapsed.Sec
  Value: 7

  Key : Analysis.Memory.CommitPeak.Mb
  Value: 66

  Key : Analysis.System
  Value: CreateObject


ADDITIONAL_XML: 1

BUGCHECK_CODE: 101

BUGCHECK_P1: 10

BUGCHECK_P2: 0

BUGCHECK_P3: ffffa48079820180

BUGCHECK_P4: 1

FAULTING_PROCESSOR: 1

PROCESS_NAME: vmware-vmx.exe

FAULTING_THREAD: ffffb588a7940080

STACK_TEXT: 
ffff9f84`164bc4a8 fffff802`5b148308 : ffffa480`79820180 fffff802`55121385 ffffb588`98a68220 00000000`00000000 : vmx86+0x100a
ffff9f84`164bc4b0 fffff802`5b14abd3 : ffffb588`a8050160 00000000`00000000 ffffa480`7bd28000 00000000`0000006b : vmx86+0x8308
ffff9f84`164bc5f0 fffff802`5b141f4e : ffff9f84`164bc978 00000000`00000000 ffff9f84`164bc978 ffffb588`86e02000 : vmx86+0xabd3
ffff9f84`164bc6c0 fffff802`5b14298e : 00000000`00000000 fffff802`553543a9 ffffb588`abe58cc0 fffff802`551132e4 : vmx86+0x1f4e
ffff9f84`164bc8a0 fffff802`556b24eb : 00000000`000003e4 00000000`00000001 00000000`00000000 ffffb588`ab9b07a0 : vmx86+0x298e
ffff9f84`164bc900 fffff802`556b1db6 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : nt!IopXxxControlFile+0x71b
ffff9f84`164bca20 fffff802`551d3c18 : 00000000`00000000 ffff9f84`00000001 00000000`000003e4 ffff9f84`164bcb80 : nt!NtDeviceIoControlFile+0x56
ffff9f84`164bca90 00007ffc`d5bdc1a4 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : nt!KiSystemServiceCopyEnd+0x28
000000c0`c61ffcd8 00000000`00000000 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : 0x00007ffc`d5bdc1a4


STACK_COMMAND: .thread 0xffffb588a7940080 ; kb

SYMBOL_NAME: vmx86+100a

MODULE_NAME:
vmx86

IMAGE_NAME: vmx86.sys

BUCKET_ID_FUNC_OFFSET: 100a

FAILURE_BUCKET_ID: CLOCK_WATCHDOG_TIMEOUT_INTERRUPTS_DISABLED_vmx86!unknown_function

OS_VERSION: 10.0.18362.1

BUILDLAB_STR: 19h1_release

OSPLATFORM_TYPE: x64

OSNAME: Windows 10

FAILURE_ID_HASH: {e206f958-3924-e3cb-af92-ab296be9b5e9}

Followup: MachineOwner

Reply
0 Kudos
ChaosEnergy
Contributor
Contributor

with version 16, it know happens daily

Reply
0 Kudos
gbohn
Enthusiast
Enthusiast

I don't have Version 16, but was seeing a similar BSOD intermittently on my 15.5.6 system. (Same vmx86+100a location, same "CLOCK_WATCHDOG_TIMEOUT_INTERRUPTS_DISABLED_vmx86!unknown_function").

(In case it's relevant, I am using a Xeon E5-1650 v4 CPU and ECC memory on an Asus X99-E WS/USB 3.1 motherboard).

Replacing the CPU and motherboard didn't help. It hasn't happened (knock on wood) in over 9 weeks, but I changed multiple settings trying to see if any of them helped.

That's in addition to less controlled changes like Windows updates, Guests going from RHEL 7.7 to 7.8, I'm not using video in the guest nearly as much, etc.

It may or may not have had anything to do with it, but some things I changed (sadly several at once) between seeing this and not  were:

*) My RHEL Guest used to have the processor settings for 'Virtualize Intel VT-X/EPT' and 'Virtualize CPU Performance counters' set.

 

   I unchecked them since I didn't need them.

*) I disabled Hyper-threading. (I later re-enabled hyper-threading without ill effect, but I replaced that with the next change

    so not a definitive change as such).

*) Manually set the Turbo limit to x38 in BIOS for 1-4 cores. (Since this is a Xeon, I have no idea if that actually

   accomplished anything since they normally ignore some changes to speed settings).

   x38 is applicable for this CPU (Limit to 3.8 GHz turbo), but would vary based on the specific CPU.

Just F.Y.I in case any of those is helpful to you.

I've been reluctant to re-enable some settings on the chance it will ruin the current peace...

Reply
0 Kudos
Mikero
Community Manager
Community Manager

If this is happening daily with a new purchase of Workstation 16, the best advice is to file a support ticket so we can formally help.

The communities aren't an official support channel, but purchases come with 30 days of support.

Support can also file bugs back to the engineering in the cases where we need to make code changes.

-
Michael Roy - Product Marketing Engineer: VCF
Reply
0 Kudos
tanshen_daniel
Contributor
Contributor

hi gbohn . thank's for you share info, these days my windows server 2019 and vmware 15 happend this problem . so  i want understand you analysis result. thank you very much.

Reply
0 Kudos
nathanh0
Contributor
Contributor

Time to find something other than VMWare - At this rate, it's going to cost me $2500 in hardware replacements from so many hard restarts and priceless data.

VMWare has had over a year to work on this, you just lost a customer.

Reply
0 Kudos