To start with I'd suggest you try disabling "Accelerate 3D graphics" for the two VMs you ran into this issue with and see if it still happens after you power them on. It's under Virtual Machine Settings > Hardware > Display
It seems that regardless of what limit you place on 'Graphics Memory' for each VM the new mksSandbox.exe process will still commit 4GB. So say you wanted to give a VM 4GB of normal RAM and told it to use 1GB of RAM to the GPU and would in turn only expect it to use just over a total of about 5GB (with slightly more overhead)...it would instead be using 8GB (with slightly more overhead)! Though between 3-4 GB of this only shows up as "Shared Commit" or "System Commit" in Process Explorer and "Commited" in Task Manager
So upon running two similar machines that you would expect to only take about 10GB total, they would instead be using 16GB+ total...and if you're like me with only 16GB to start with this quickly translated into an unusable system followed by a lockup or a random BSOD (with a different error each time) as two running machines plus windows could only function if you are using a huge pagefile.sys and even then it would get very very slow.
I managed a workaround by applying the Windows 10 Anti-Exploit Setting: "Do not allow child processes" to vmware-vmx.exe so that it could not spawn the mksSandbox.exe process used in the new GPU sandboxing. This in turn forced VMWare to revert to the old way of handling things and avoids the constant 4GB to GPU alone being applied for each VM (despite setting it lower) allowing me to once again run two or three appropriately set VMs at once without running into the issues it caused before. Note that you could also just disable 'Accelerate 3D graphics' for each VM as well but as the issue brings my system to its knees should I forget to to uncheck it after creating a new VM I've found the Anti Exploit option to be more reliable as it prevents the mksSandbox.exe from ever getting launched and commiting the extra RAM despite having lower settings. You could also try allowing the usage of a MUCH larger pagefile.
All the above assumes you are seeing the same issue I was and I can't be sure that you are. It sounds similar enough on the face that I suspect it could be so I figured I'd let you know what I ran into and what has worked for me though obviously you get none of the benefits of the new gpu sandboxing and it may even impact the new DX11 compatibility (I'm not sure as I don't use it)
