VMware Cloud Community
actyler1001
Enthusiast
Enthusiast

NTP broken after ESXi 7u3 upgrade

NTP time sync appears to have broken after the 7U3 upgrade.  Anyone else run into this and have suggestions on how I might fix?  I've deleted and re-created the service, checked FW policy, tried different servers...  Nothing helps.

Original Build: VMware ESXi, 7.0.2, 17867351

actyler1001_1-1635089620144.png

 

New Build: VMware ESXi, 7.0.3, 18644231

actyler1001_0-1635089532365.png

 

Labels (2)
135 Replies
actyler555
Enthusiast
Enthusiast

I've just deployed Update 3f Build 20036589 into my VMware environment.  This NTP problem appears to be solved.  Wow, that only took VMware 9 months to fix.....

mbuster25
Contributor
Contributor

Yup i upgraded  to 3f too, and no more time sync issues!  Seems to be a silent fix as nothing was mentioned in release notes.

Reply
0 Kudos
LabMasterBeta
Enthusiast
Enthusiast

**** My NTP cleanup/fix on vSphere 7.0 Update 3(g) ****

VMware vSphere update: 7.0u3 (aka 7.0.3)
Patched ESXi version: 7.0u3g.
Patched vCenter version: 7.0u3g.

FIRST: Verify Time Zone are same on ESXi and vCenter.. I only point this out because you can now finally change Zone on VCSA admin port 5480..

===============

 

**NTP KB ARTICLE; NOTE! ONLY FOR 7.0-U3+**

https://kb.vmware.com/s/article/87488

[root@localhost:~] vi /scratch/ntpconfig.txt
[root@localhost:~] cat /scratch/ntpconfig.txt

server time.nist.gov
server pool.ntp.org
tos maxdist 30

[root@localhost:~] esxcli system ntp set -f /scratch/ntpconfig.txt
[root@localhost:~] esxcli system ntp set -e 1

 

===============

**NTP KB ARTICLE; NOTE! ONLY FOR 7.0-U3+**

https://kb.vmware.com/s/article/87176

To display current configuration:

# esxcli system ntp get
# esxcli system ptp get

To change configuration:

# esxcli system ntp set
# esxcli system ptp set

To get a list of ALL new 7.0 Update-3 commands, use help:

# esxcli system ntp --help

 

===============

**HELPFUL OLD NTP KB ARTICLE, FOR SYNTAX COMMANDS ONLY**

((This KB specifically states issues after u3 upgrade))

https://kb.vmware.com/s/article/86255

**WARNING!! "RESOLUTION" at end of article does not do what you think it does at first glance, did not work for me, and in general editing any "conf" file with plain-text in 7.0.3 can be dangerous because if its encrypted it will become corrupted, so.... be sure to check if the "conf" file is still plain-text (safe to edit) or if it's now encrypted (dont edit will corrupt), as VMware transitions towards the configurations being encrypted over time with future updates...  I expand on this in a post further down.

===============
===============

After KP Articles, vCenter was showing as broken (red alerts).

 

To get vCenter to re-sync NTP based on Host, this worked for me:

**Reset NTP settings on ESXi host, back to NTP defaults:**

[root@localhost:~] esxcli system ntp set -r

 

**Reload NTP service on ESXi host:**

[root@localhost:~] /etc/init.d/ntpd restart

 

**Use vCenter GUI to input NTP addresses into ESXi host again**
((I'm using GUI because I want to see it work as designed...)

 

**Reload NTP on ESXi host again**

[root@localhost:~] /etc/init.d/ntpd restart

 

**Poll the NTP server on ESXi host a few times, give it a minute:**

[root@localhost:~] ntpq -p

 

**Wait until NTP server on ESXi host shows Time Sync = True:**

[root@localhost:~] esxcli system ntp test

((Diags output omitted..)

Service analysis completed.
Timeinsync: true

 

[root@localhost:~] esxcli system ntp get
Enabled: true
Loglevel: warning
PID: 2126728
Runtime Seconds: 260
Servers: time.nist.gov, pool.ntp.org
Service Providing Kernel Time: Network Time Protocol
Time Service Enabled: true
Time Synchronized: true

 

**Check ESXi host and vCenter GUI's**
((In vCenter > Host > Configure > Time Config > Test Service))
**Finally, for 1st time since 7.0u2, this test passed for me!**

 

Note-1:
This final NTP bug is annoying but not a deal-breaker:
In ESXi host NTP config GUI (not vCenter) version 7.0u3g:
ESXi GUI: Clicking "Actions" is still broken (it does nothing).
ESXi GUI: Clicking "Refresh" finally works again as expected.

 

Note-2:

After all this cleanup work and fiddling with ESXi and vCenter (both version 7.0.3g), finally now, NTP tests pass in both ESXi and vCenter when vCenter is set to ESXI Host for time sync.

I did not try all this with ESXi version 7.0.3f. but if you read 7.0.3f release notes, you'll want to upgrade to patch 7.0.3g immediately...

 

EDIT: Oct 2, 2022, to remove being overly harsh with use of "destroy" (elaborated below). And, fix typo to clarify **Reset NTP on ESXi host again** should instead read **Reload NTP on ESXi host again**"

Reply
0 Kudos
Kinnison
Commander
Commander

Comment edited because it is not worthy of note and not to cause disruption to others.

Reply
0 Kudos
CMorrisMCC
Contributor
Contributor

How is this a false alert when I'm suffering issues on three of my hosts using build 19482537? Just saying. 

Reply
0 Kudos
Kinnison
Commander
Commander

Comment edited because it is not worthy of note and not to cause disruption to others.

Reply
0 Kudos
actyler555
Enthusiast
Enthusiast

Well, we've started our vSphere 7 upgrades in production.  Deployed ESXi, 7.0.3, 20328353 to our first host and guess what?  Yes, the NTP problem persists.  What a joke.  I've opened a support request with VMware, we'll see where this goes.

Support Request #22363318109

actyler555_0-1663272762273.png

 

jdptechnc
Expert
Expert

Deployed a fresh 7.0.3 Build 20328383 server, and I am seeing this problem as well.

Please consider marking as "helpful", if you find this post useful. Thanks!... IT Guy since 12/2000... Virtual since 10/2006... VCAP-DCA #2222
Reply
0 Kudos
actyler555
Enthusiast
Enthusiast

I just got done running the commands that @LabMasterBeta posted above and it did work...

 

for reference...

**Reset NTP settings on ESXi host, back to defaults:**
esxcli system ntp set -r


**Reload NTP service on ESXi host:**
/etc/init.d/ntpd restart


**Use vCenter GUI to input NTP addresses into ESXi host again**
((I'm using GUI because I want to see it work as designed...)

**Reset NTP on ESXi host again**
/etc/init.d/ntpd restart


**Poll the NTP server on ESXi host a few times, give it a minute:**
ntpq -p


**Wait until NTP server on ESXi host shows Time Sync = True:**
esxcli system ntp test

((Diags output omitted..)

Service analysis completed.
Timeinsync: true

 

**Check ESXi host and vCenter GUI's**
((In vCenter > Host > Configure > Time Config > Test Service))
**Finally, for 1st time since 7.0u2, this test passed for me!**

Reply
0 Kudos
LabMasterBeta
Enthusiast
Enthusiast

@Kinnison 

Regarding my prior comment (Before I edited):

"

**HELPFUL OLD NTP KB ARTICLE, FOR SYNTAX COMMANDS ONLY**

((This KB specifically states issues after u3 upgrade))

https://kb.vmware.com/s/article/86255

**WARNING!! "RESOLUTION" AT END OF ARTICLE DESTROYS 7.0.3 HOST**
**ONLY USE THE NEW KB ARTICLES ABOVE FOR U3 NTP TEXT FILE METHOD**

  ((VMware should insert a warning about this for Update-3!!))

"

/////////////

 

There are several VMware official KB articles that warn not to touch the "conf" files anymore with 7.0 Update3c+ specifically, those edit commands in that KB seem to be for 7.0 Update 1 and 2 only, and did not do anything to the "tox maxdist" value when I tested and in fact did corrupt my NTP configuration.

 

This made me believe those commands in that KB are leftovers from Update 1 and 2, so the article should instead point to the other newer 7.0.3 / update3 KB's that I listed in my prior post: Which is, to make a plain text file and then read the commands for inclusion into the "conf" file - that DID work for the "tox maxdist" value change for me. 

 

Also, there are official KB articles that warn the numerous "conf" files are being encrypted over time by VMware as future 7.0 Updates become available, so issuing ANY command to add a string of plain-text characters to a "conf" file, is now a risky situation and would in effect "corrupt" any encrypted conf file.

 

And finally, if you look at the commands, they make a backup of the ntp.conf as ntp.conf.bak, and then execute the esxcli against the ntp.conf.bak instead of original file. So, this is good to not corrupt the original conf file (eg. in case its encrypted so as to not corrupt it).  Because of this, I will admit that I was a bit too strong trying to get VMware's attention to update all their KBs for 7.0 Update-3 uniqueness VERSUS 7.0 Update-1/2 and have edited my original post to remove "destroy" and further clarified.

 

THAT BEING SAID:

Finding out the hard way there are these subtle yet SIGNIFICANT under-the-hood changes for new commands methods etc, for 7.0 Update 3 (compared to prior Update 1 and 2), that VMware is not Fully Emphasizing in its KB Articles like they should IMHO makes it a tedious support hassle (eg. I feel VMware should revisit EVERY 7.0.0, 7.0.1, 7.0.2 KB article, to test and fully clarify versioning if older 7.0 KB's are applicable to 7.0.3 or not, and if not then link to the all-new 7.0.3 article).

 

This versioning clarification and highlighting more predominately, would greatly reduce the ongoing support efforts - and reduce risk)....

Reply
0 Kudos
Kinnison
Commander
Commander

Comment edited because it is not worthy of note and not to cause disruption to others.

Reply
0 Kudos
WuGeDe
Enthusiast
Enthusiast

hi @actyler1001  are the esxi really not ntp synced and do not sync any longer or does only the UI show them for what ever reason as "not synced"?

regards

Reply
0 Kudos
afzalim
Contributor
Contributor

Resolution Host has lost time synchronization error after upgrading to ESX 7.0.3 build 18644231 (86255)

This issue is resolved in VMware ESXi 7.0 Update 3c (build number 19193900)

ESXi NTP can be forced to accept time from AD time sources root dispersion has a large value

This is described here https://kb.vmware.com/s/article/1035833

Commands to add "tos maxdist" for 7.0U3 builds:
1. Update NTP configuration(in file and configstore)
cp /etc/ntp.conf /etc/ntp.conf.bak && echo "tos maxdist 15" >> /etc/ntp.conf.bak && esxcli system ntp set -f /etc/ntp.conf.bak
2. Restart NTP: esxcli system ntp set -e 0 && esxcli system ntp set -e 1

Note: Please note that the "tos maxdist" config will not persist across reboot .

 

 

 

https://kb.vmware.com/s/article/86255

 

Reply
0 Kudos
maksym007
Expert
Expert

We still facing NTP problems. And we have opened a SR to VMware, where we got a recomendation exactly the same: 

ESXi NTP can be forced to accept time from AD time sources root dispersion has a large value

This is described here https://kb.vmware.com/s/article/1035833

Commands to add "tos maxdist" for 7.0U3 builds:
1. Update NTP configuration(in file and configstore)
cp /etc/ntp.conf /etc/ntp.conf.bak && echo "tos maxdist 15" >> /etc/ntp.conf.bak && esxcli system ntp set -f /etc/ntp.conf.bak
2. Restart NTP: esxcli system ntp set -e 0 && esxcli system ntp set -e 1

Note: Please note that the "tos maxdist" config will not persist across reboot .

Our ESXi hosts are currently having following build: ESXi 7.0 Update 3g 

20328353
Reply
0 Kudos
Kinnison
Commander
Commander

Hi @maksym007,


Would you mind explaining your exact problem better?


The starting point of this thread began with the worrying message on a gold background which read: "Tine service is currently not syncronized", long since removed in the latest versions of the vCenter object, which then evolved with other reports of various problems (IMHO in part) solved from version to version of ESXi.


I can tell you that AFAIK if you insert the line "tos maxdist" option in your "ntp.conf" file the it persist across reboot since ESXi 7.0U3f, at least on my hosts it works like this, and I didn't even need to add it referencing a host built around a domain controller running Microsoft Windows Server 2022 (never synced to an upstream time source) as my "time server". What I have instead noticed is that if the network interface to which the default VMKERNEL interface connects is "saturated" for a long enough time, the NTP service "tends to lose its bearings".


Regards,
Ferdinando

Reply
0 Kudos
maksym007
Expert
Expert

vCenter version in our environment: vCenter Server 7.0 U3i 20845200; 

ESXi hosts: ESXi 7.0 Update 3g 20328353

 

 

We have two clusters, where in Cluster A: 6 ESXi hosts, in Cluster B: 4 ESXi hosts.

They are not in the domain, and VMs are not syncing time from ESXi hosts. The problem persists in both clusters. 

But when I open via vCenter - > ESXi host - Configuration - > Time - there is a well-known Error: Time service is currently not synchronized

My colleague took over the case but called me yesterday and informed that he did as VMware Support requested: 

adjusted parameter: tos maxdist -  no changes. He restarted services via ssh. still the same.

Reply
0 Kudos
Kinnison
Commander
Commander

Hi @maksym007


I'll tell you, I understand, but from my point of view an ESXi host is not a reliable time source, and neither is any other type of virtual machine, and simply adding the line "tos maxdist" blindly because someone says so, as you've found it's (almost) useless. What I can tell you, based on my personal experience, after having deployed over a hundred ESXi / vCenter combinations is that the error: "Time service is currently not synchronized" does not always (IMHO almost neveer) correspond to the concrete "state of things" and that's why I asked what your exact problem is.

 

I would suggest using reliable time sources outside (and unrelated) your vSphere infrastructure for the whole thing.


Regards,
Ferdinando

Reply
0 Kudos
loungehostmaste
Enthusiast
Enthusiast

> ESXi host is not a reliable time source, and neither is any other type of virtual machine

how comes that our ESXi servers are timesources since 2008 and the ntp-servers for both hosts are a virtual machine on each one?
and yes you can easily confirm if the time is correct

> I would suggest using reliable time sources outside (and unrelated) your vSphere infrastructure for the whole thing

running ntpd on each and every guest is nonsense and in a 100% virtualized environment nothing else exists

frankly i disabeld the foolish warning this thread is about and at no point in time any VM had a wrong time - so this topic should come to an end after all that months

Reply
0 Kudos
Kinnison
Commander
Commander

Comment edited because it is not worthy of note and not to cause disruption to others.

Reply
0 Kudos
loungehostmaste
Enthusiast
Enthusiast

so you are the only one which is allwoed to express his opinion? sorry, the game don't work that way!

Reply
0 Kudos