No, you would have to backup your VMs, install ESXi and then import your VMs back to the host. ESX and ESXi have different partition layout that make them incompatible for switching between the 2 installs.
Is there a good technical reference on this question? I appreciate the clarity of the question and answer here, but before trekking off into the wilderness I'd like to see that VMware has considered this and worked up a set of instructions for me. Thanks!
The VMware doc (page 47 - http://www.vmware.com/pdf/vi3_35/esx_3/r35u2/vi3_35_25_u2_upgrade_guide.pdf) assumes you have VirtualCenter. If you don't then many are using VMware Converter to move the VMs from ESX to temporany storage and then back to ESXi (I'd also suggest an extra backup just to be safe).