Multiple VM Unrecoverable Errors (vcpu-0)

[MS230000751]

I have had this issue on two different machines at two different times now and I have no idea why.

I have a bunch of VMs that I have setup with different OSs to simulate test environments. After after a couple months of not using them, I come back to power them on and get "VMWare Workstation Unrecoverable Error: (vcpu-0)" errors. This is sometimes immediately on boot, sometimes they power on and run for a second and then throw the message.

On the first machine it happened on - I just erased the VMs and started over. No big deal.

But on these machines - there are a bunch of them (not all, but a lot) and they have purchased license keys to activate different versions of Windows on them! If I start over, I'm wasting both time and money. It's happening on both windows and linux VMs which tells me its not OS specific - but something about the config files. I have attached the error screenshots and logs for a couple of them. All pointing to invalid memory access locations.

I have no idea why they just stopped working. My computer is fine and the software can create and run other VMs and it can run other VMs that were in the same drive, folder, and group with no problem.

The drive hosting the VMs is an internal M.2 NVME SSD (in good health)

The system RAM is 128GB (Giving no issues with any other applications or OS features)

The CPU is not overclocked - neither is the RAM - nothing is

Virtualization is enabled - as I can still run other VMs in the program fine.

VMWare Workstation is the only hypervisor software I have installed

I could really use some help here. If I can't get this to work then I'll be looking at buying new license keys for the OS version - rebuilding all the broken VMs and cloning backups to a RAID server to restoring if this happens again. But the fact that this just randomly happened on two different PCs for me is not comforting. I want to figure out what is going on so I can fix it and prevent it from happening again.

Again - any help is greatly appreciated!

[MS230000751]

So after further investigation - this only happens when I power on additional VMs. If I just run one VM - everything is fine. So the VMs themselves do not appear to be broken. But once I switch to a new VM tab and power it one - it's not long before the other open VM(s) fail and shutdown with the error.

[MS230000751]

Just did some more troubleshooting. This is getting very annoying.... The same thing happened to the Windows 10 VM I had on my other desktop. Completely different storage device and PC than these, but same issue. Can't run it.

I went back to the original desktop with the 5+ failing VMs and did the following

1. Uninstalled VMWare Workstation - then Reinstalled.                           FAIL
2. Right clicked each VM - Manage - Clean Up Disks.                               FAIL
3. Created new empty VM and linked the vmdk to it.                              FAIL
4. Reverted to farthest back snapshot and deleted newer snapshot.     FAIL
5. Removed unnecessary hardware (CD Rom - Printer)                           FAIL
6. Moved VMs to Different PC.                                                                  FAIL
7. Added smc.version = "0" to vm configs                                                FAIL
8. Updated to latest VMWare Workstation 16.2.4                                     FAIL

Now it seems I've spoken too soon. The VMs seem to be failing at random. I thought it was a pattern but I thought I had the issue fixed as they were not shutting down, but then random ones would just shut down and give me that error again.

I have no clue what is going on here. Does anyone have any clue as to what is going on?

[MS230000751]

Okay. So this is not just with the pre-existing VMs.

I am creating new virtual machines and they are experiencing the same issue as well. Have not even installed an OS on them and they are getting the error (vcpu-0) invalid memory access location.

This is beyond just the vm configurations. Something bigger is jacked up here.

I have no idea how to decipher these vm logs to figure this out. But I know my hardware is fine - otherwise, both my computers would have to be experiencing hardware (RAM) issues - and I would definitely be seeing errors in more places than just VMWare.

I have no clue where to go from here.

[MS230000751]

I know I'm getting no responses here, but I'm updating the log anyway just in case someone happens to suffer from the same issue.

I got nowhere with my own troubleshooting.

This is happening on two PCs running VMWare Workstation. Both computers are experiencing the same issue.

I have just contacted VMWare Support and they created business ticket with me. Kind gentlemen on the phone established a zoom meeting with me - shared the screen - and collected all the information and logs for the issue. He is going to do some research and get back with me on Monday.

I may have one more suspicion while I wait though....... Going to test my theory.....

[MS230000751]

My theory was wrong. It just kicked rocks in my face. I thought maybe it was my antivirus because both machines received an update to the antivirus before I noticed this - but I completely uninstalled the antivirus and the issue is still persisting. 

Man, what in the world.

[MS230000751]

This is officially insane. I have gone through every troubleshooting step I know to go through short of wiping the entire system and starting from scratch.

I got off the phone call with VMWare today and they had some basic troubleshooting to go over with me like hyperthreading, hyper-v, vm configs, hardware configurations - but nothing we tried worked.

I have attached a text file with all of the troubleshooting steps I have taken so far. Nothing is working.

I have uninstalled - reinstalled - verified the installer is good because it works and the VMs work on two out of 4 machines in the office. All same updates - same OS version - same app versions - same antivirus.

One machine is setup with Dual Boot (Two Windows Installations) and VMWare works on one boot drive but not the other for some reason? Which means its not BIOS or hardware. Again - these two windows installations have the exact same updates and versions on it.

And VMWare is the ONLY program I am having issues with.

I reinstalled Windows (keeping the programs and files but reinstalling all Windows and System files) and it still does not work.

If I go to clean install windows on here and wipe out all the programs and setups, this is going to take one or two weeks to set everything back up the way it is, and I still won't know the cause so I won't know the solution - which means I am taking a risk that it will happen again and have to go through wiping everything all over again. Seems ridiculous. There has to be something here we are missing.

I copied all the program files for VMWare, the program data, the appdata, from a working machine to this machine - and it still does not work.

Uninstalled Antivirus, disabled firewall, disabled hyper-v, uninstalled vmware, performed Norton 360 registry cleaner to remove invalid registries associated to vmware, reinstalled vmware, moved same vms to different machine and they work, do not work on this one, reinstalled windows but kept programs and user data, ran sfc /scannow for repairs, checked Windows Event Viewer for errors - no logs generated, confirmed it works on the same system hardware and BIOS configuration through different boot device, changed vm config files with IOMMU - VTD-X - hardware config to 16.2.x, and banged my head against the wall.

None of the steps above have fixed anything.

I am quickly going insane here. Somebody rescue me

[louyo · Accepted solution]

Tried turning off 3D acceleration?

Another user with similar but he doesn't know for sure what "fixed" it:

https://communities.vmware.com/t5/VMware-Workstation-Pro/Error-and-crash-on-16-2-x-with-vcpu-0-PANIC...

Lou 

[MS230000751]

Louyo you are exactly right.

I had one more Zoom session with a VMWare Tech and he disabled that on the VMs which resolved the issue.

I thought it strange because I had been using that for over a year, so i dug deeper.

3D Acceleration uses the graphics card - so I gave myself a hefty facepalm and checked the graphics driver.

After updating the NVIDIA Graphics Driver to the latest release - everything is working as normal!

Thanks for the response. So glad I can finally put this behind me and get back to work.