Because we use ACE we're tied to Using Workstation 7.1.6 / Player 3.1.6.
We've got several PCs with latest-generation intel procesors (i7-3770, i5-3450, i5-3570) and on all of them (but not older Core2 Duos, etc). If I create a new VM and try to use a stock Ubuntu install ISO for 11.10 or newer then boot fails almost instantaneously, with a message saying "The CPU has been disabled by the guest operating system. You will need to power off or reset the virtual machine at this point". The VM console is completely black.
Looking inside the VM log files, everything is as normal, then there're these lines;
Jun 27 15:35:39.398: vcpu-0| X86Fault_Warning: vmcore/vmm64/cpu/interp.c:398: cs:eip=0x60:0xc0135838 fault=13 Jun 27 15:35:39.398: vcpu-0| Vix: [3416 vmxCommands.c:9705]: VMAutomation_HandleCLIHLTEvent. Do nothing. Jun 27 15:35:39.398: vcpu-0| Exiting on CLI;HLT at 0x60:0xc0685ff5
After which the log shows a clean shutdown.
Looking at the system.map for the kernel in question, the 0xc0135838 address corresponds to the "native_write_cr4", which I deduce is triggering the 'early_protection_fault' routine in the linux kernel, which eventually double-faults while trying to call 'printk' and enters the 'hlt_loop' function, which in the kernel we're using is at the 0xc0685ff5 address as listed in the VM log.
I've tried every single VM parameter I can think of or find on the net in the VMX file, including systematicaly flipping every single cpuid bit. Running the install on another PC and then copying the built VM over hits the exact same problem (with different cs:eip offsets depending on the kernel used.) As far as I can tell, this happens with any Linux kernel newer than about 3.0. I've also tried flipping every remotely relevant setting in the host PCs BIOS (disabling VT-X, CPU cores, etc.)
I've also tried pretty much every linux kernel parameter for disabling subsystems or drivers, changing earlyprintk, etc with no joy.
There's nothing relevant looking in the host OS (Win 7/x64) logs. I've reinstalled everything more times than I care to think. This also happens with the previous versions of WS and player (7.1.5/3.1.5)
Does anyone have any ideas, or has seen anything like this before?