As you may have guessed this is quite important to get fixed. If the ESXi host is not using NTP then it gets it's time from the CMOS / BIOS of the physical server which the hypervisor runs on. This could easily cause a denial of service for your VMs that depend on Kerberos (max 5 minute skew before ticket denial). For example AD uses this. Also some applications are even less resilient to clock skew and requirements for accuracy of 2 minutes or less is not uncommon.
What will end up happening in your situation is that Guest Operating Systems will likely adjust their own time by participating in the domain hierarchy (AD) for example, but when they are vMotioned, rebooted, snapshotted, etc. the Guest OS time instantly changes to that of the ESXi host, then the cycle repeats of them trying to correct their time again. This can cause undesirable affects and should be remediated by correcting the ESXi hosts to use NTP as linked by another poster above (see the official VMware KB).
For more info on this phenomenon review the whitepaper http://www.vmware.com/files/pdf/techpaper/Timekeeping-In-VirtualMachines.pdf and you should know that the option exists to disable that setting but it's really not recommended or popular, but KB 1189 explains how (don't do this). Simply fix your ESXi host NTP and you'll be fine.
For detailed review of NTP health, I use the following PowerCLI function:
http://psvmware.wordpress.com/2013/09/03/get-vmhosttimereport-reporting-time-from-vmhost-system/
For a quick review of time (PowerCLI one-liner):
Get-VMHost |Sort Name|Select Name, @{N=“NTPServer“;E={$_ |Get-VMHostNtpServer}}, @{N=“ServiceRunning“;E={(Get-VmHostService -VMHost $_ |Where-Object {$_.key-eq “ntpd“}).Running}} | Out-GridView
In general, it's considered safe to change the NTP settings of ESXi hosts (except for extremely rare edge cases such as CSCtu05038 – "Time change may result in unnecessary transmission of LACP PDUs", which only affected very old Cisco 1000v versions). Typically it's safe to make the change, but since timesync is one of the most critical infrastructure mechanisms you may consider performing the change off hours. If VM time changing during vMotions is a concern then consider setting DRS to manual (or lower priority) until resolved.
