VMware Communities
JC_Morris
Enthusiast
Enthusiast

Workstation 12.5.0 crash 0xC0000374 (heap corruption) in VMX

Running a setup build of Windows 10 build 1607 ("Anniversary Edition") that worked under 12.1.1 I'm seeing crashes, but not always at the same point.  The event log for the most recent crash reads:

Faulting application name: vmware-vmx.exe, version: 12.5.0.11529, time stamp: 0x57cf7a1d

Faulting module name: ntdll.dll, version: 6.1.7601.23539, time stamp: 0x57c99b8f

Exception code: 0xc0000374

Fault offset: 0x00000000000bf262

Faulting process id: 0x1b20

Faulting application start time: 0x01d20f53c7869ff3

Faulting application path: C:\Program Files (x86)\VMware\VMware Workstation\x64\vmware-vmx.exe

Faulting module path: C:\Windows\SYSTEM32\ntdll.dll

Report Id: 0d9c27fd-7b4f-11e6-8cab-005056c00008

The run producing the above event log was made immediately after rebooting the machine  - no other VM was invoked before the failure.

FWIW the system I'm generating uses UEFI boot mode; the host system (Windows 7 Enterprise) uses legacy boot. The host does run EMET 5.5 but I don't see any EMET alerts.

Any suggestions?

Joe Morris

Reply
0 Kudos
33 Replies
jmtella
Enthusiast
Enthusiast

Same here  Smiley Sad

Reply
0 Kudos
ralish
Enthusiast
Enthusiast

I'm seeing the same issue with the vmware-vmx.exe process crashing with a 0xc0000374 (Heap Corruption) exception. I can reliably reproduce the crash on a snapshot of a Windows 10 Build 1607 ("Anniversary Edition") VM. The point of the crash is during the Windows 10 loading screen, immediately after the BIOS, but before what becomes the logon screen. Like JC Morris, the VM is configured in UEFI mode, though so is the underlying host system. The same system ran flawlessly under Workstation v12.1.1.

Interestingly, if I set the VM configuration to gather full debugging information, resulting in the vmware-vmx-debug.exe process being used, I can't reproduce the crash. This may suggest some sort of optimisation in the regular build may be the culprit?

Can someone from VMware please comment?

Reply
0 Kudos
TracyHuang
Enthusiast
Enthusiast

Thanks for the posting!

Would you please help to provide more information so that can help us to investigate:

1. vm-support package. Would you please collected the data via menu -> help -> support -> Collect Support Data.

2. The dmp file and vmmcores file which are located in the VM directory.

Reply
0 Kudos
JC_Morris
Enthusiast
Enthusiast

There are no .dmp or vmmcores files.  Note that the VM has been run since the failure, so if it would normally cleanup the VM folders that may be expected. Note also that in one of the failures the entire system was locked, requiring a power-off cycle to recover control of the host...and possibly explaining the absence of those files.

I have created a data collection ZIP file...it's just under 10 MB. How and where do you want it sent - and given both the subsequent use of the VM and the absence of .dmp and vmmcores files, is it likely to be useful?

Incidentally, after turning on full debugging, performance has been lousy but (as noted by ralish in the earlier note) I've not seen the heap corruption failure. I'll try turning debug off again and see if I can get a data collection immediately following the failure.

Joe

Reply
0 Kudos
JC_Morris
Enthusiast
Enthusiast

Update: I just now changed the debugging option back to "default" and ran a Windows 10 build1607 setup...and VMX blew up again. The system raised the following dialog:

Title: VMware Workstation VMX has stopped working

Body:  A problem caused the program to stop working

           correctly.  Please close the program.

Buttons:  "Close the Program"

               "Debug the program"

The symptom is an apparent CPU loop - I say "apparent" because the system is nonresponsive, but there is no physical disk activity; the only recovery is by pressing the POWER button.  Note: when the POWER button is pressed (it's assigned the function of "power off") the VMware task immediately terminated and a clean shutdown followed, so the message loop *is* operational.

At reboot (and without invoking any virtual machine) there is no .dmp or vmmcores file in either C:\Program Files (x86)\VMware, or in the folder containing the virtual machine.

Also before attempting to run the VM I ran HELP -> Collect Support Data tool (~10 MB).  Where do you want it uploaded?

Joe

Reply
0 Kudos
jmtella
Enthusiast
Enthusiast

It is impossible to work with version 12.5. In my company we have returned to the version 12.1

Reply
0 Kudos
TracyHuang
Enthusiast
Enthusiast

Thanks for your support! Would you please upload the logs to here:

FTP Server: ftpsite.vmware.com

User:   inbound

Pwd:    inbound

Port:   21

Reply
0 Kudos
JC_Morris
Enthusiast
Enthusiast

Uploaded file “vmsupport-2016-09-19-14-56.zip” into the default folder on ftpsite. Unlike the crash I described upthread in this case the host was NOT locked and performed a clean shutdown – so (hopefully) any buffers for the log files should have been flushed.

And also as noted earlier – there was neither a .dmp nor a vmmcores file the VMware folders.

Other than the event timestamp the Application event log entry for the crash is the same as what I quoted at the start of this thread.

Incidentally, because of the problems with Windows 10 I would have taken the action that "jmtella" noted - returning to version 12.1 - but our INFOSEC department has declared anything older than 12.5 to be prohibited software.  OTOH, my own personal copy of VMware - which at this time is being used only with Windows 7 - has exhibited no problems whatever after upgrading to 12.5.

Joe Morris

 

Reply
0 Kudos
ralish
Enthusiast
Enthusiast

Just adding that I've opened a support case with VMware regarding this issue. I'll post here anything pertinent (e.g. new workarounds) and hopefully an eventual solution.

Reply
0 Kudos
ElCoyote_
Enthusiast
Enthusiast

Funny you should mention that.. I am also seeing tons of 'malloc/free' errors with 12.5 on RHEL6 (Linux).

These look like this:

*** glibc detected *** /usr/lib/vmware-installer/2.1.0/vmis-launcher: free(): invalid pointer: 0x00007fd30fe4bf50 ***

*** glibc detected *** /usr/lib/vmware-installer/2.1.0/vmis-launcher: free(): invalid pointer: 0x00007fd30fe4c570 ***

*** glibc detected *** /usr/lib/vmware-installer/2.1.0/vmis-launcher: free(): invalid pointer: 0x00007fd30fe4cb90 ***

*** glibc detected *** /usr/lib/vmware-installer/2.1.0/vmis-launcher: free(): invalid pointer: 0x00007fd30fe9af40 ***

*** glibc detected *** /usr/lib/vmware-installer/2.1.0/vmis-launcher: free(): invalid pointer: 0x00007fd30fe5db30 ***

*** glibc detected *** /usr/lib/vmware-installer/2.1.0/vmis-launcher: free(): invalid pointer: 0x00007fd30fe9b340 ***

*** glibc detected *** /usr/lib/vmware-installer/2.1.0/vmis-launcher: free(): invalid pointer: 0x00007fea229ec100 ***

I'm also in the process of reverting back to 12.1.1 on RHEL6.

Too bad, that was the first version that didn't need patching on Fedora24.. Smiley Sad

Vincent

Reply
0 Kudos
hanness_rdu
Enthusiast
Enthusiast

I too am seeing the 0xc0000374 (Heap Corruption) exception. I've tested this on two very different machines now where Workstation 12.1.1 worked fine. In my case it happens frequently, but not every time I boot the host. Sometimes it boots just fine.

On both of my systems the host OS is Windows 10 Pro ver 1607. One system is a desktop system, the other is a Dell XPS 15 9550 laptop.

Seems that this is an easily reproducible issue so hopefully that will result in a fix very soon!

- Hannes

Reply
0 Kudos
TracyHuang
Enthusiast
Enthusiast

Thanks JC for uploading packages!

In the logs we cannot find relative information of the VMX crash as you mentioned. So would you please help to collect logs again? Really appreciated!

1. Host support bundle: from Workstation toolbar Help menu -> support -> Collect support data, choose the related guest.
2. Guest support bundle:
In guest OS, go to VMware Tools program directory(C:\Program Files\VMware\VMware Tools), run vm-support.vbs. The script will be finished with below message:

---------------------------
Windows Script Host
---------------------------
Support information has been uploaded to the Virtual Machine's log file, please run vm-support on the host to send the information to VMware support.
---------------------------
OK  
---------------------------

Under %TMP% directory of guest OS, you’ll find a *.zip file like vmsupport-8-30-2016-16-33.zip, rename it to guest-vmsupport-8-30-2016-16-33.zip.

3. Collect VMware Tools log files: In guest OS, collect every log file under C:\tmp.

Reply
0 Kudos
TracyHuang
Enthusiast
Enthusiast


Hi hanness,

Thanks for the posting! To help us find the issue ASAP, would you please collect the logs for the crashed vm?

1. Host support bundle: from Workstation toolbar Help menu -> support -> Collect support data, choose the related guest.
2. Guest support bundle:
In guest OS, go to VMware Tools program directory(C:\Program Files\VMware\VMware Tools), run vm-support.vbs. The script will be finished with below message:

---------------------------
Windows Script Host
---------------------------
Support information has been uploaded to the Virtual Machine's log file, please run vm-support on the host to send the information to VMware support.
---------------------------
OK  
---------------------------

Under %TMP% directory of guest OS, you’ll find a *.zip file like vmsupport-8-30-2016-16-33.zip, rename it to guest-vmsupport-8-30-2016-16-33.zip.

3. Collect VMware Tools log files: In guest OS, collect every log file under C:\tmp.

Would you please upload the logs to here, and tell us the name you uploaded. Really appreciated!

FTP Server: ftpsite.vmware.com
User:   inbound
Pwd:    inbound
Port:   21

Reply
0 Kudos
JC_Morris
Enthusiast
Enthusiast

In most cases (including the one for which I uploaded the log files) I'm preparing a master image for use on bare-metal machines, so the installation of VMtools would be inappropriate.  I'll try getting through the initial script (autounattend.xml), add VMtools, and see if I can experience another crash.

As I've noted earlier, the point of failure and the immediate impact of the crash is neither predictable nor reproducible in detail on demand; some of the crashes are during the processing of the XML script, long before I have an opportunity to install VMtools.  Fingers crossed...

Joe

Reply
0 Kudos
hanness_rdu
Enthusiast
Enthusiast

I'll see if I can collect logs over the weekend. In the meantime, I have already reverted back to Workstation 12.1.1 so I'll need to go back to 12.5, duplicate the problem, then gather logs.

- Hannes

Reply
0 Kudos
ralish
Enthusiast
Enthusiast

Purely for the reference of the VMware employees participating in this thread, the support case I've opened is SR16245102909. May prove useful to help avoid duplication of effort as we all work toward getting this resolved.

Reply
0 Kudos
dariusd
VMware Employee
VMware Employee

If you are affected by this problem, I'd appreciate if you could try downgrading your virtual USB controller to USB 2.0 to see if the problem goes away.  Power off your VM, go into VM > Settings, and on the Hardware tab choose the USB Controller, and change USB Compatibility to USB 2.0 then choose the Save button.

I appreciate that this might not be an acceptable long-term solution for many; There's a chance that attempting this may provide a workaround for some, and may provide us with useful information towards resolving this crash.

Thanks for your patience,

--

Darius

Reply
0 Kudos
JC_Morris
Enthusiast
Enthusiast

Would changing the USB compatibility be significant if no USB devices have been connected when the failures occurred?

FWIW, I'm currently stepping through my 1607 build kit with USB2 compatibility set and haven't (yet) seen the failure - but I've never been able to trigger a failure on demand.

Joe

Reply
0 Kudos
freebeing
Enthusiast
Enthusiast

I tried USB compatibility 2.0 : nothing changed.

When I use a VM with EFI support, VM often crashed at startup or when I reboot.

When I use a VM with BIOS support, no problem, even with Windows 10.

Reply
0 Kudos