See my updated description for this post. ESXi requires an older version of pxelinux, I think probably because they forked off of the Redhat, i.e. in ESXi 4, it was a lot more compatible with Linux, and I think was based on RHEL. Not anymore.
I got around this by having a menu choice in my non-ESXi PXE install point to the directory for the ESXi install, i.e.
LABEL Vmware installs and upgrade
MENU LABEL ^VMware ESXi
COM32 pxechn.c32
APPEND /vmware/pxelinux.0 -p /vmware/
TEXT HELP
All vmware install/upgrade
ENDTEXT
I wouldn't know about compiling against the new code base. I looked into that, and it seems I'd run into too many dependencies to be worth my time.
