Problem with Time Configuration in ESXi 5, 1157734

[Rnutter0]

I have been fighting a problem for a couple of days now that doesnt make any sense.  All of the clocks on my blades are showing 4/7/2014 as the date.  If I override ntp and set it to 4/29/2014, it will stay there for about a minute or so and will then revert 4/7/2014.  I have tried turning off the NTP client on a particular blade with the same result.  I have changed the NTP server on another blade and had the same problem.  If I enable SSH and ESXi shell, each blade reports the correct date and time but a client session on the blade reports 4/7/2014 just like the vSphere client GUI shows.

I have a case open with Cisco TAC (since we are running the Cisco UCS chassis, that is where the support is handled from).  They looked at it and couldnt find a problem on the hardware.  They have opened a case with VMware and I am waiting to hear back from them at this point.

Anyone seen this problem ?

[lvaibhavt]

I am sure you would have tried this but just wanted to verify. If any of your colleague open the VI Client from their machine or from the VC itself.

Same behavior is seen ?

[Rnutter0]

I am the admin, so no one else would have a reason to be in the vcenter console at this point.

Here are some diagnostics that i have run.  NTP is getting to the host but for some reason isnt being used -

Every 2s: ntpq -p localhost                                                                                             2014-04-08 13:47:03

     remote           refid      st t when poll reach   delay   offset  jitter
==============================================================================
time-a.nist.gov .ACTS.           1 u    -   64    1   44.221   -0.306   0.001
time-b.nist.gov .ACTS.           1 u    -   64    1   46.186    3.620   0.001

~ # /etc/init.d/ntpd restart
Stopping ntpd
Starting ntpd
~ # esxcfg-vmknic -l
Interface  Port Group/DVPort   IP Family IP Address                              Netmask         Broadcast       MAC Address       MTU     TSO MSS   Enabled Type               
vmk0       Management Network  IPv4      10.34.190.101                           255.255.255.0   10.34.190.255   00:25:b5:0a:00:02 1500    65535     true    STATIC             
~ # tcpdump-uw -c 5 -n -i vmk0 host 129.6.15.28 and port 123
tcpdump-uw: verbose output suppressed, use -v or -vv for full protocol decode
listening on vmk0, link-type EN10MB (Ethernet), capture size 96 bytes
13:56:57.425038 IP 10.34.190.101.123 > 129.6.15.28.123: NTPv4, Client, length 48
13:56:57.470939 IP 129.6.15.28.123 > 10.34.190.101.123: NTPv4, Server, length 48
13:58:04.425006 IP 10.34.190.101.123 > 129.6.15.28.123: NTPv4, Client, length 48
13:58:04.470926 IP 129.6.15.28.123 > 10.34.190.101.123: NTPv4, Server, length 48

The part that doesnt make any sense is that the time is right but the date is wrong.  If I change it to today's date, it is reset to 4/8 within about 30 seconds to 1 minute.

[DanielOprea]

Hi,

Please look to this KB (Troubleshooting NTP on ESX and ESXi 4.x/5.x).

[DanielOprea]

And this KB

[Rnutter0]

Thanks.  have already been looking that document.  Even going so far as to looking at the running processes and may have found a possible indication of an issue.The running ntpd process never leaves a wait state.  Also not seeing it try to communicate to the outside unless I do a /etc/init.d/ntpd restart.

[DanielOprea]

Please, upload the system log for the ESXi.

[MKguy]

Please post your /etc/ntp.conf file.

Messages from ntpd are logged to the /var/log/syslog.log file. Take a look at this file after restarting ntpd:

# grep -i ntp /var/log/syslog.log

Have you checked the hardware/BIOS date/clock settings? A large skew between the hardware system clock and the OS might not be accepted or reset.

# hwclock

15:16:32   04/30/2014   UTC

# esxcli hardware clock get

2014-04-30T15:16:34Z

#This is the OS clock:

# esxcli system time get

2014-04-30T15:18:55Z

You can set the system clock with esxcli as well without booting the server, see esxcli hardware clock set --help

Note that ESXi does not use timezones or daylight saving times, it only uses UTC locally. The hardware clock should also always be set to UTC since BIOSes never know about timezones or DST either.

[Rnutter0]

Here is the ntp.conf file contents -

~ # cat /etc/ntp.conf

restrict default kod nomodify notrap nopeer

restrict 127.0.0.1

server 129.6.15.28

server 129.6.15.29

driftfile /etc/ntp.drift

Here is the syslog file grped for ntp -

2014-04-08T16:59:21Z root: ntpd Stopping ntpd

2014-04-08T16:59:21Z ntpd[67017]: ntpd exiting on signal 1

2014-04-08T16:59:21Z root: ntpd Starting ntpd

2014-04-08T16:59:21Z ntpd[83966]: ntpd 4.2.6p2@1.2194-o Fri Dec 16 03:55:00 UTC 2011 (1)

2014-04-08T16:59:21Z ntpd[83967]: proto: precision = 1.227 usec

2014-04-08T16:59:21Z ntpd[83967]: Listen and drop on 0 v4wildcard 0.0.0.0 UDP 123

2014-04-08T16:59:21Z ntpd[83967]: Listen normally on 1 lo0 127.0.0.1 UDP 123

2014-04-08T16:59:21Z ntpd[83967]: Listen normally on 2 vmk0 10.34.190.101 UDP 123

Tried doing hwclock set via cli, which is I am guessing the same as setting it in the vsphere client.  It took the date change to 4/30 but reverted back to 4/8 within about 30 to 45 seconds.

This is the problem I am fighting.  Been waiting on vmware support to get with me since yesterday.

[MKguy]

Tried doing hwclock set via cli, which is I am guessing the same as setting it in the vsphere client.

No, it's not the same. One is the physical hardware clock and one the clock the OS running on the hardware (ESXi) keeps on it's own.

If it doesn't work setting the hardware clock from within ESXi, then reboot the server, enter the BIOS and change the date/time settings there.

[Rnutter0]

Therein lies the problem.  The hardware is showing the correct date/time - I checked that.  It is in ESXi that it isnt showing the correct time.  I am on patch 1157734 and have been hesitant to apply a newer patch since I wasnt having any problems.  Looking at the patches now since I am still waiting on vmware to call back.

[Rnutter0]

Thoought everyone would like an update.  I picked one of the blades in the UCS chassis and vmotioned all server to other blades.  VMware support was able to identify the problem occurred after a module named vmkapei loaded.  So far they havent been able to explain why.  Have heard from them in 24 hours.  I reinstalled ESXi 5.1 on that blade and the problem went away.  For a reason I cant explain the other 3 blades stopped showing the problem after blade 4 was reinstalled.  I have patched of the blades with 1483097 and continue to be problem free.