Workstation 12.5.0 crash 0xC0000374 (heap corruption) in VMX

[JC_Morris]

Running a setup build of Windows 10 build 1607 ("Anniversary Edition") that worked under 12.1.1 I'm seeing crashes, but not always at the same point.  The event log for the most recent crash reads:

Faulting application name: vmware-vmx.exe, version: 12.5.0.11529, time stamp: 0x57cf7a1d

Faulting module name: ntdll.dll, version: 6.1.7601.23539, time stamp: 0x57c99b8f

Exception code: 0xc0000374

Fault offset: 0x00000000000bf262

Faulting process id: 0x1b20

Faulting application start time: 0x01d20f53c7869ff3

Faulting application path: C:\Program Files (x86)\VMware\VMware Workstation\x64\vmware-vmx.exe

Faulting module path: C:\Windows\SYSTEM32\ntdll.dll

Report Id: 0d9c27fd-7b4f-11e6-8cab-005056c00008

The run producing the above event log was made immediately after rebooting the machine  - no other VM was invoked before the failure.

FWIW the system I'm generating uses UEFI boot mode; the host system (Windows 7 Enterprise) uses legacy boot. The host does run EMET 5.5 but I don't see any EMET alerts.

Any suggestions?

Joe Morris

[jmtella]

Same here:

I tried USB compatibility 2.0 : nothing changed.

When I use a VM with EFI support, VM often crashed at startup or when I reboot.

When I use a VM with BIOS support, no problem, even with Windows 10.

[dariusd]

Hmmmm.  Any chance I could obtain a dumpfile (vmware-vmx.dmp) and corresponding vmware.log from one of those failures with USB2 only?

Thanks,

--

Darius

[JC_Morris]

I've not been able (so far) to convince the but to crawl out of its nest with USB2 compatibility set but I did see another crash with the default options (i.e., USB3 and only default debugging info).  The Event Log entry was the same (other than the timestamp and dump ID) as the earlier one I listed when I started this discussion.

The crash occurred while building a new Win10/1607 image on a Win7 host running VMware 12.5; the setup program (driven by autounattend.xml) had completed and scripts run to configure the Windows 10 client; I rebooted the machine and had just started the installation of the first application (ActivClient 7.0.2.460); VMware announced a failure, but the VMware UI did *not* close.  As in all of the crashes I've seen with Workstation 12.5 *no* VMware-vmx*.dmp file was created - I checked before running the "collect support data..." menu option.

Also as before, at the time of the crash VM Tools had not yet been installed.

The support data has been uploaded to ftpsite as "vmsupoport-2016-09-27-09-34.zip.

Joe Morris

[ralish]

Just in case it has any bearing on the troubleshooting being performed by VMware I can reproduce the crash even with a fully updated VMware Tools installation on the VM (v10.0.10 Build 4301679).

[freebeing]

Sorry, I don't understand why but with USB 2.0 today it didn't crashed... it's very weird.

I rebooted 40 times : no crash.

[JC_Morris]

I can't speak for the other users here who are having the crash problem with a Win10 build 1607 disk, but so far I've had numerous (but irregular) heap corruption events on a standard 12.5 host.  I have *not* seen any problems (despite making several runs) wben the virtual machine is configured to either (a) run in full debug mode, or (b) run USB ports at USB2 specs.

One question: given that the VMware folk are consistently asking for .DMP files, and my system (at least; how about others?) has never created one...is there a reason why no dump file is showing up?  Is there a parameter somewhere that needs to be set to allow it to be created?  Or, since it's VMWARE-VMX.EXE that's failing does that also terminate the thread that should be creating the dump?

Joe

[hanness_rdu]

I attempted to reproduce the error and now can no longer seem to do so. My suspicion is that the Windows Cumulative Update (KB3194496) from just a few days ago may have resolved the issue. I cannot seem to reproduce the problem on either of the machines where it was occurring before.

I have not hammered on it really extensively yet but it seems that the issue occurred pretty regularly before and I've booted my VMs far more often than I would think necessary to see the problem.

I'll keep at it and if I have any reoccurrences of the problem I'll report back here, but so far it is looking good for me.

NOTE: Some people have reported problems with KB3194496 not installing on their machines. Microsoft is apparently aware of that issue and is working on it.

- Hannes

[JC_Morris]

Are you saying that you installed KB3194496 on the *client* or to the *host*?

If it's to the host the problem is still open since I'm seeing it on a development machine running Windows 7 Enterprise, and 3194496 applies only to Windows 10 build 1611.  OTOH, the only system on which I've (so far) seen the heap corruption panic is a 1607 client - but I've seen the failures during setup, before patches can be installed.

Joe

[ralish]

Just advising that I have just had a crash while running a Windows 10 VM with full debugging information (i.e. using the vmware-vmx-debug.exe process). So it would seem my earlier potential workaround isn't guaranteed to work, but it does seem to reduce the occurrence of the crash. This is the first time I've seen a crash in debug mode and it happened while rebooting to install Windows Updates, so perhaps there's some sort of race condition at play, and high load during early boot makes the crash more likely?

[ralish]

I have KB3194496 installed on my host system and just witnessed the crash so can say with reasonable confidence this update does not have a bearing on the issue. Ironically, the Windows 10 VM crashed while rebooting to complete installation of KB3194496.

[ralish]

Is there any news on this issue from VMware? This thread has gone a bit quiet of late but the issue remains.

I can also confirm I'm seeing heap corruption crashes on a Windows 8.1 x64 VM, so either the issue isn't specific to Windows 10 guests, or there's another heap corruption regression present in Workstation 12.5.

[freebeing]

And you are in EFI mode, isn't it ?

With EFI mode I have some crashs (more when virtual USB is set to USB 3.0) but not any crash with BIOS mode.

[ralish]

Yes, that is correct. I guess the catalyst is EFI with some additional impact when using a USB 3.0 virtual controller.

[dariusd]

Sorry for not following up on this thread earlier.  The issue with EFI virtual machines with virtual USB controllers on Windows hosts crashing with 0xc0000374 (Heap Corruption) has been fixed in Workstation 12.5.1 and newer.  Refer to the Release Notes for more information: VMware Workstation 12 Pro Version 12.5.1 Release Notes

If you are still encountering this 0xc0000374 (Heap Corruption) crash with Workstation 12.5.1 and newer, please let us know right away!

Thanks,

--

Darius