What's the model ID of the host, as shown on the ESXi screen? I expect it's a Macmini5,2, Macmini5,3 or Macmini6,2. If it's a Macmini6,2, see my posts in this thread: ESXi 5.x on new Apple Mac Mini 6,2 Late 2012 *NOT* working
If that doesn't help, could you please attach the vmware.log from a VM which has faield to boot into Mac OS? (Just use the Browse... button below when composing a reply.)
It's the newest Mini, "Macmini6,2".
That thread is huge, so maybe I missed something, but I didn't see any mentions of something resembling my problem. Is there a specific post you are referring to? In general, the Mini is working fine and I am running several Linux and one Windows VM on it.
Attached is the vmware.log file. Note that there are some messages about the CD drive because the install DMG was attached. I have since removed that and rebooted, but the behavior is still the same.
vmware.log 103.7 K
You are having the same SMC problem as the other users in that thread. Your vmware.log contains
2012-12-02T06:41:43.325Z| vcpu-0| I120: HostSMCRead: Unable to open the driver.
2012-12-02T06:41:43.340Z| vcpu-0| I120: HostSMCRead: Unable to open the driver.
... which means that, as far as ESXi is concerned, it thinks it's not really a Mac. This is due to a change that Apple made to their latest hardware, which ESXi is not yet able to handle.
Look for zer010gic's post in that other thread, where there's a link to an ESXi 5.1 ISO image specifically with SMC in its name. That is the same release of ESXi as you are running there, but preconfigured with the network drivers for the Macmini, with the iovDisableIR=TRUE config option that stops the PSOD at boot, and also includes an engineering fix for the SMC problem you're hitting (not an official fix, just a quick workaround we put together to get your Mac OS guests booting again).
Some users have reported that they can't "upgrade" their existing installed ESXi to the patched one, so you might need to do a fresh installation before things start to work.
Hope this helps!