There are actually two issues in this case. Firstly the ESXi virtual machine needs at least 4GB virtual memory provisioned in the VM's settings, and secondly you may need to upgrade to at least VMware Workstation 8 in order to present the required CPU features to the VM itself. For further details, please take a look at Running Nested VMs
André
PS: Discussion moved from VMware ESXi 5 to Nested Virtualization
Hello,
as commented, you must assign 4 GB to the virtual esxi.
To be able to run nested virtual machines, some requirements must be accomplished.
I would recommend you to check this to gain acknowledge about this topic.
Best regards,
Pablo