Can you confirm that the ESXi hosts are not syncing the time from a VM which is running on these hosts, and that the NTP server (and port) is reachable from the hosts?
the esxi hosts ARE syncing to a time server located on an ESXI host.
The NTP server is reachable and all ports open.
the time sync does work but it seems like the time falls behind by a minute or 2.
Why not point your ESXi hosts to an external time source or something external to the ESXi host at least? You're syncing your host to a guest located on the host. I could see a loop of time skew here.
I sync my hosts with a Cisco router that sync's with an external time source (ntp pool etc..). My VM's sync to AD and/or through VMtools down to the ESXi host. I hope this helps.
I would recommend you to review the ntp logs located in /var/log and see if you see any strange event that can help you where to start looking, and maybe it could help you to see the hostd.log file in /var/log , sometimes it has useful information for troubleshooting purposes.
Let us know how it goes and good luck!