actyler1001
Enthusiast
Enthusiast

NTP broken after ESXi 7u3 upgrade

NTP time sync appears to have broken after the 7U3 upgrade.  Anyone else run into this and have suggestions on how I might fix?  I've deleted and re-created the service, checked FW policy, tried different servers...  Nothing helps.

Original Build: VMware ESXi, 7.0.2, 17867351

actyler1001_1-1635089620144.png

 

New Build: VMware ESXi, 7.0.3, 18644231

actyler1001_0-1635089532365.png

 

Labels (2)
100 Replies
AndrewBorn
Contributor
Contributor

For us, the warning in the GUI appears to be a false positive and it clears after 10-15 minutes. I tested with changing the host's time to be off, and it adjusted itself to match the NTP source.

 

Try checking your NTP via the command line. 

watch "ntpq -p 127.0.0.1"

 

There are some other options you can use as well.  I believe there is a KB for more troubleshooting.

0 Kudos
Srijithk
Enthusiast
Enthusiast

as far as i have heard its a known issue in 7.0 U3 and a false positive

you shall see some 2 warnings

"Host has lost time synchronization" and "Time service is currently not synchronized"

this should ideally be fixed in an upcoming patch

P.S 7.0 U2 and earlier dont have any issues, changing the NTP servers wont help, i have tested in my lab

 

 

0 Kudos
actyler1001
Enthusiast
Enthusiast

Great, at least this issue isn't such a big deal.  VMware just keeps churning out one garbage release after another with no regard for QA.  Not impressed.

Regards,
Adam Tyler

actyler1001
Enthusiast
Enthusiast

Just applied build VMware ESXi, 7.0.3, 18825058, NTP still broken.  Unbelievable.

-Adam

0 Kudos
loungehostmaste
Enthusiast
Enthusiast

i REALLY get sick auf VMware!

the ntp servers are two guests with vmware-tools sync disabled and all other guests sync with the host, restarted ntpd on both timeservers, forced ntpdate and restarted the ntp service on both hosts - guess which one is 7.0.3....... damned can VMware stop messing around with public cloud and what not instead doing their basics homework?

[root@esx1:~] ntpq -p 127.0.0.1
    remote           refid      st t when poll reach   delay   offset  jitter
==============================================================================
+buildserver.the 116.203.119.16   3 u   41   64  377    0.144   +3.152   0.737
*ntp2.thelounge. 217.196.145.42   2 u   50   64  377    0.118   +4.310   0.870


[root@esx2:~] ntpq -p 127.0.0.1
    remote           refid      st t when poll reach   delay   offset  jitter
==============================================================================
buildserver.the 116.203.119.16   3 u    3   64  177    0.113  +1339.8   0.037
ntp2.thelounge. 217.196.145.42   2 u   56   64  177    0.078  +1341.2   0.035

denis-rg
Contributor
Contributor

Hi!

Someone has any update in this issue? Is there a patch or workaround?

Thank you!!

0 Kudos
loungehostmaste
Enthusiast
Enthusiast

after restart and resync all ntpd servers and a lot of time it seems at least to stop sending alerts - however, it#s a bad joke after 12 years that each and every update is a tradeoff between solving problems and bring a ton of new ones

themikem
Contributor
Contributor

On my system, it appears to be an issue with using aggregated uplinks.  The diagnostic output from "Test Services":

Service analysis started on host: esxi.domain.com
Test started at: 2021-11-05T13:36:14Z
Time Service is administratively enabled.
Verifying NTP service.
NTP server: 172.16.4.11 resolves IPv4: XXX.XXX.XXX.XXX
Virtual NIC vmk0 : Admin: Up
IP Interface: vmk0 IPv4 Address: STATIC XXX.XXX.XXX.XXX
IP Interface: vmk0 connected to 43 on distributed vswitch
Virtual Switch distributed vswitch needs at least one configured uplink
IP Network Stack: defaultTcpipStack
No physical NICs are connected to vmk interface
Firewall Rule: ntpClient allows traffic on port: 123
ntpd is running, PID: 2098587
Kernel clock type: ntp
NTP is in sync
Peering with: XXX.XXX.XXX.XXX
Statum: 2
Accuracy to within: 17.196500 msecs
Polling every: 8 secs
Network delay round trip: 0.239000 msecs
Difference from remote clock: -0.095681 msecs
Service analysis completed.

 

So even though the test has the alert of "Configuration is not working normally. " at the top, NTP still is sync'd to within 17 msec. Obviously it has an uplink somewhere - just using LACP uplinks to two different switches for redundancy....

I hope they fix this soon...

0 Kudos
Srijithk
Enthusiast
Enthusiast

  1. On vSphere Client, go to Configure -> System -> Time Configuration tab, select "Network Time Protocol" and click on EDIT button. 
     

     

  2. From the configuration box, uncheck "Enable monitoring events" and click on OK button.            
     

     

    this is the workaround i'm aware of and its supposed to be getting fixed in the upcoming release of vsphere
 

 

actyler1001
Enthusiast
Enthusiast

They might fix it in the next release.  Wonder what new bugs they'll introduce?  Hey guys, its VMware here, we've fixed the NTP problem, but have to apologize that all hosts with USB ports PSOD randomly.  We'll fix that in a few months and introduce something else more awful.  At least you are enjoying the rollercoaster right?

fed up.  Time to look at alternate products.

arangopablo
Contributor
Contributor

Same problem here, after some troubleshooting, we are facing same problem

Hopefully, it will be fixed in the next release. 

 

0 Kudos
plannue
Contributor
Contributor

I too ran into this upon upgrading hosts to 7U3a. I wasn't able to immediately fix it and didn't bother with a reboot because the host clearly had perfect time and was indeed sync'd with the NTP server, as much as that warning wanted be to believe.

 

That was a couple days ago. I went back just now and ran "test services" for NTP, and it resolved itself and showed NTP as synchronized. So I'm not sure how long one would have to wait after updating to this release to run the test services function, but nevertheless, the error is gone now after doing nothing special to the host except that test function.

0 Kudos
sthoen1
Contributor
Contributor

It is STILL broken in 7u3b build # 18905247

bondebond
Contributor
Contributor

Yes, it is still broken in 18905247

actyler555
Enthusiast
Enthusiast

Confirmed, I've also applied 18905247 and my hosts are reporting NTP is still broke.  But I am sure that some kubernetes feature that none of us use or care about got fixed.

SecretSmile
Contributor
Contributor

The same on 18644231. Does anyone know how to treat?

0 Kudos
actyler555
Enthusiast
Enthusiast

Bro, at this point VMware doesn't know how to treat.  They're just churning out terrible software right now.  You have to choose between running stable, but vulnerable older builds or have stuff just not work properly.  It's unfortunate, VMware used to be a decent software company.

0 Kudos
loungehostmaste
Enthusiast
Enthusiast

"VMware used to be a decent software company" - Yeah, *before* Dell put their dirty fingers on it - for the sake of god now that VMware becomes a own company things will get better before it hurts too much any i take money and time in my hands migrationg to KVM/Proxmox

normally you shouldn't recognize that there are hypervisors in the game

0 Kudos
actyler555
Enthusiast
Enthusiast

This just in, VMware has officially pulled the following builds.  Unreal!

https://kb.vmware.com/s/article/86398

  • vSphere ESXi 7.0 Update 3    (build 18644231)
  • vSphere ESXi 7.0 Update 3a  (build 18825058)
  • vSphere ESXi 7.0 Update 3b  (build 18905247)
  • vSphere vCenter 7.0 Update 3b (build 18901211)
0 Kudos