Workstation 9 crashes ugly on ASUS P8Z77-V Pro based machine, Linux host

[kemida]

I've been trying to solve this for the past five days to no avail. I've posted all the details, tests, logs, and a video of the crash here: http://www.kevindanenberg.com/vmware/

Basically, a few seconds after I start up a guest machine (any OS), the host screen goes blank and the host reboots. This happens after a clean install of Ubuntu 12.04.1 64-bit on the host, and several other Linux flavors (see the link above for all the various trials...) This happens with Workstation 9 as well as Player.

It seems hardware-related, because Ubuntu 12.04.1 64-bit works on a clean install on an old machine just fine.

Hardware in question:

ASUS P8Z77-V Pro

Intel i7-3770S @ 3.10GHz

G.SKILL Ripjaws X Series 16 GB (4 x 4 GB) F3-12800CL9Q-16GBXL

WD Black 1 TB (ATA WDC WD1002FAEX-0)

I've stripped down the machine as bare as possible to test.

I put in a support request to VMWare and someone is trying to help, but they haven't figured anything out yet, either.

Any ideas?

[admin]

On the host, "cat /proc/interrupts | grep NMI".  Wait ten seconds and do it again.  If the numbers are going up, you may have a hardware issue that's generating non-maskable interrupts.  This can cause the kind of problem you've described.

[kemida]

Thanks, Jim

Ten seconds doesn't show any change:

kevin@Umbreon ~ $ cat /proc/interrupts | grep NMI
NMI:         17          2          2          1          2          1          0          0   Non-maskable interrupts
kevin@Umbreon ~ $ cat /proc/interrupts | grep NMI
NMI:         17          2          2          1          2          1          0          0   Non-maskable interrupts

Three minutes later shows some increases, but I don't know if that's significant:

kevin@Umbreon ~ $ cat /proc/interrupts | grep NMI

NMI:         26          4          4          2          2          1          1          0   Non-maskable interrupts

[admin]

That shouldn't be significant.

What if you try to configure a 32-bit guest to use binary translation  (VM->Settings->Processors->Preferred mode)?

[kemida]

Tried binary translation before, and just tried it again. No improvement.

[admin]

What happens if you specify Intel VT-x/EPT?

[admin · Accepted solution]

I suspect that this is a Workstation 9 bug regarding SMEP on Ivy Bridge systems.  If so, it only affects VMs using binary translation.  The bug should be fixed in Workstation 9.0.1.  In the meantime, you should be able to work around the problem by specifically requesting VT-x/EPT as the preferred mode--assuming that VT-x is enabled in your BIOS.

[kemida]

Brilliant! Yes, using VT-x/EPT allowed me to start my VM as a workaround! So does VT-x.

Also, VT-x is not enabled in my BIOS by default. I had to turn it on.

Glad to hear a fix is in the works. Thanks so much!

[markdean]

FWIW, I'm getting this on an older Dell 470 Precision Workstation running F17 (LXDE) that I still use, which doesn't have the VT extensions (P4 Xeon 2004) and can only use binary translation. This means it is broader than just SMEP on Ivy Bridge issue and may be a problem on other implementations of execute disable. Having to back down to Workstation 8 for now...

[admin]

Mark Dean wrote:

FWIW, I'm getting this on an older Dell 470 Precision Workstation running F17 (LXDE) that I still use, which doesn't have the VT extensions (P4 Xeon 2004) and can only use binary translation. This means it is broader than just SMEP on Ivy Bridge issue and may be a problem on other implementations of execute disable. Having to back down to Workstation 8 for now...

If "F17" means Fedora 17, you may be running into a different problem, which is that Workstation 9 doesn't work with Linux kernels 3.5 and later.  I believe there is a patch in javascript:;.

[markdean]

"If "F17" means Fedora 17, you may be running into a different problem, which is that Workstation 9 doesn't work with Linux kernels 3.5 and later.  I believe there is a patch in Re: vmplayer-5.0.0 terminates immediately after start."

I believe you may be correct-although it was the the underlying architecture that was the problem. I have 32 bit CentOS 6.3 running just fine now. I had to drop all the way down to Workstation 7.0.1 just to install Workstation and run 32 bit VMs (I know that without the VT extensions 64 bit VMs were not going to run on this systems). This Pentium 4 Xeon system, although having gobs of RAM, does not have the CPU features that Intel based systems need to run even binary translation in VMware Workstation versions later than 7.x. I'm sure it's a check the installer does since I'm not aware that it uses VT or DEP for 32 bit VMs. This is the last Intel based system I have (aside from my laptop) and will be chucking it next year for an Opteron based Workstation like my other systems. I was just shocked when I couldn't even run 32 bit VMs on WS8.