If you want to follow the document provided from VMware, be sure that to use PXELinux 3.x, not 4.0+ for ESXi. So far, ESXi 5.1a only works with PXELinux 3.x.
Or, if you want to do it simply, I wrote a tool for this which is running on Windows providing an all-in-one boot server for installing ESXi. Check here: http://wp.secretnest.info/?p=812
