This issue happens first time when I try to start 64-bit guest on 32-bit host, so I initially think it's related to 32/64-bit. But later it become clear it's more about VT-x than about 32/64-bit.
My host OS is 32-bit Hardened Gentoo Linux, kernel 2.6.28-hardened-r9. I've tried both VMware Workstation 6.5.3.185404 and 7.0.1.227600. My hardware is Core 2 Duo 6600 on ASUS P5B-Deluxe with latest stable BIOS 1101. I've power-cycled system, then enabled Vanderpool in BIOS. My CPU doesn't support Trusted Execution Technology, and there no way to disable it in BIOS. I've rebooted several times after that, sometimes with power-cycled, and ensure Vandertool is enabled in BIOS.
I've run VMware-guest64check-5.5.0-18463 tool, and it report "This host is capable of running a 64-bit guest operating system under this VMware product.". I've boot using vt.iso from http://communities.vmware.com/docs/DOC-8978 and here is it report:
CPU 0: VT is enabled on this core
CPU 1: VT is enabled on this core
When I start any (32/64-bit) guest in "Intel VT-x or AMD-V" mode, my host OS immediately reboot (right after guest "Power On", I don't even see guest's BIOS POST output). When I start 64-bit guest in "Intel VT-x/EPT or AMD-V/RVI" I see guest BIOS POST, then guest OS start booting (Windows7 install dvd completes progressbar "Loading files"), and then host OS reboot. And "reboot" actually mean skipping normal host OS shutdown procedure and immediately reset system to host BIOS POST.
I've tried to boot 32-bit guests (Windows7, Ubuntu9.04 and Gentoo) using all possible virtualization modes. In "Automatic", "Automatic with Replay", "Binary translation" everything works, in "Intel VT-x/EPT or AMD-V/RVI" I got message "This host does not support EPT. Using software virtualization with a software MMU." and everything works ok too (only 64-bit guests reboot host when this mode used).
About a year ago I tried to disable hardened in kernel to ensure this isn't because of PaX/GrSecurity, but that doesn't help. ASUS provide "beta" BIOS updates, but according to their descriptions these updates doesn't fix this issue, so I'm not sure is it good idea to try it.
My best guess now it's motherboard/BIOS bug. Any ideas?
