Bug: VMware OpenGL Crash Between vm3dgl64.dll and opengl32.dll on Second DLL Run

[NoelC1]

Hello,

We have an application that loads as a graphic plug-in (DLL) and does OpenGL context creation and displays, etc.

In a Windows guest if we are run from a host application that does not itself use OpenGL, when our plug-in DLL loads the opengl32.dll from Windows AND the vm3dgl64.dll load - as expected.  Everything works right on that run.

The problem starts to become apparent when our plug-in DLL finishes and unloads.  At that time the opengl32.dll library unloads.

Notably at this point vm3dgl64.dll does NOT unload.

The crash happens when we are loaded/run a second time and opengl32.dll loads again at a different address (as you would expect due to Address Space Layout Randomization).

At this point the vm3dgl64.dll continues to call addresses in opengl32.dll based on the base address it loaded at the FIRST time, not the second time.

The crash comes, specifically, when we call the wglCreateContextAttribsARB function.  When we call only functions directly from within opengl32.dll the problem does not occur.

We have observed that the unload of opengl32.dll does not occur when Windows is running on real hardware with drivers from nVidia, ATI, or Intel.   However, the root cause seems to be that the vm3dgl64.dll does not properly re-estabilish its linkages with the opengl32.dll library when opengl32.dll is reloaded.

This is occurring with the latest version of VMware Workstation 14 and the latest VMware tools.

-Noel Carboni

ProDigital Software

[parmarr]

Please see VMware Knowledge Base

[NoelC1]

Thanks for trying to help, but no, that's not the issue at all.  We are correctly requesting and assessing the OpenGL capabilities with the context.

The problem is that the VMware OpenGL implementation is allowing the system's opengl32.dll library to exit/be unloaded, and is NOT re-estabilishing its linkages when the driver is re-loaded.

In fact, when we work around the problem by increasing the reference count to keep the library loaded after our DLL is unloaded the problem does not occur and our OpenGL operations work perfectly run after run.

This appears to be an implementation problem with the VMware driver's implementation of OpenGL not coordinating its operations properly with the system library.

-Noel

[vandentr01]

Can confirm there is an issue with the latest release of the VMware Tools.

When AutoCAD is run on a VM with any of the more recent VMTools installed it will not close and it appears to hinge on not releasing the vm3dgl64.dll file.

Our fix was to roll back to the 10.3.5 version of VMTools to correct the issue.

[anil2016]

Can you attach vmware.log file for your VM?

[TiArno]

I have the same issue with VMWare Tools 12.1 and also had to downgrade to 10.3.5. Is there any chance of seeing a solution to this problem?

[joserfonseca]

@NoelC1 thank you for this detailed report.  Sorry it went unnoticed for so long.   But thanks to your remarks, we have good idea of how to address it.

First we'll need to reproduce it so we can verify it, as it's unclear the exact mechanism why opengl32.dll unloads leaving our driver loaded, and why it became more widespread recently.

[NoelC1]

At this point it's been so long since we saw the original issue that I can't even recall the workaround we had implemented. I do recall that we found one, I just don't remember what it was. Thanks for your attention, and good luck with the fixes! -Noel