It looks like you are inadvertently using the mboot.c32 plugin that comes with syslinux/pxelinux. That's the wrong plugin; you need the mboot.c32 that comes with ESXi. Get it from the latest version of ESXi that you're trying to boot -- newer versions of mboot are backward compatible with older versions of ESXi. You will find our mboot.c32 in the root directory of the ESXi install CD.

The reason our documentation says to use syslinux 3.86 is that ESXi's mboot.c32 plugin is built against the syslinux v3/v4 plugin API and specifically tested against 3.86. The newer plugin API from syslinux v5 or v6 is incompatible with plugins built against the older API.

I am trying to get someone to work on porting our mboot to the newer syslinux API to make it easier for customers like you who are PXE booting multiple different OSes and want to use the latest syslinux. But no promises when/if that will happen.

Could you please comment in this thread if you have success with the correct mboot.c32?