Why do esx updates turns ntpd daemon off?

[trusted1]

So I'm finally applying the ESX updates for March and again I'm bugged that the ntp daemon gets turned off after the reboot. Yes, I had the daemon set to on for runlevels 345 (chkconfig --level 345 ntpd on), but after a post-update reboot the daemon is set to off.

What gives? One would think that an update should preserve the system's configuration. This coupled with the mind-numbingly, inefficient update process makes me think I should be re-deploying ESX servers with Altiris rather than patching in place. Has anyone else gone down the road of re-deploying an updated ESX server image, rather than patching each box?

[RBY]

Doesn't seem like you got any bites on your issue/comments. 4 1/2 month later ... what have you learned or decided?

[trusted1]

I've learned that I must be nuts because no one else has seen the prob. .

[eahatch]

I'm not sure that it is updates causing it, but I am also finding ntpd shut off on my ESX servers at different times (today for instance).

This is not only annoying, but I'm not sure what happens when my ESX server gets too far out of sync.

I'm going to update our production cluster in the next couple of weeks and if I verify it is the updates (highly suspicious) shutting off nptd, I'll open a ticket.

Alan

[Jae_Ellers]

What does this return?

chkconfig --list ntpd

Should show:

ntpd 0:off 1:off 2:off 3:on 4:on 5:on 6:off

Achieved with:

chkconfig --level 345 ntpd on

[eahatch]

Jay,

It returns as follows (mostly because I already turned the ntpd back on and started the service) :

chkconfig --list ntpd

ntpd 0:off 1:off 2:on 3:on 4:on 5:on 6:off

I've had to do this at least twice in the last few months for each of our ESX servers.

It is always possible that I've lost my mind, but I believe the runlevels are being reset for this service.

I have patched a couple of times in the same period, so I'm thinking that is the culprit.

Thanks for the response.

Alan

[trusted1]

I'm scheduling to do some testing on this tomorrow. I'm going to try and narrow the problem domain a bit but testing plain reboots vs patching-rebooting. Stay tuned.....

[trusted1]

UPDATE - I just stumbled across this thread about ntpd not starting - http://communities.vmware.com/thread/76833

jlanders had this comment that apparently solved the issue:

* You manually enabled the NTP daemon using the Unix*
* command line tools on the service console and disabled*
* the firewall. This will work, but unless the VC host agent*
* gets informed, these changes won't be persistent.*
* On ESX 3.0.1, a better way to enable NTP is to use VC.*
* On the ESX "Configuration" page, select "Security Profile"*
* on the left hand side, then "Properties" in the upper*
* right hand corner of the page. Tick the NTP entry in the*
* "Services" dialog. You'll probably first have to re-enable*
* the firewall in the Service Console to get the changes to*
* stick.*
* If you want, you can then use 'chkconfig' to disable the*
* firewall again. I'm not sure why you'd want to, but that's*
* up to you.*
* As another member indicated, you'll still need to configure*
* the NTP daemon manually. I'm assured that this capability*
* will be in a future version of VC.*
* Joe*

The "solution" seems a little hokey to me, but I'll give it a try tomorrow and see what happens.

[trusted1]

Sorry for the delay, but I did get to test this a week ago and was able to find a resolution. The problem had to do with some firewall setting through "security profile" under a ESX host's configuration tab. If you look under NTP Client, you'll see that the check box is probably not checked.

To fix it, you need to stop ntpd from the command line (service ntpd stop). Then check the ntp client box, and under advanced options you can choose an option to start the daemon if firewall is open. Once you do that it will start up the service, set the appropriate chkconfig levels (2345 on), and open the firewall port. You may actually have to check the box and save 2 or 3 times to get it to "stick" in the GUI.

This whole thing blows my mind because you would think checking box would be enough. The idea of setting runlevels for daemons through a "security" page is not very intuitive. I hope my explanation is clear enough.

[eahatch]

Thanks for the info. I've followed the steps you provided on one of our ESX servers, patched and rebooted and nptd is up and running fine.

I've also tested an ESX server that does not have ntpd configured via the VC interface (only configured in linux) by rebooting. As you may have guessed it was the reboot that cuased ntp service to be turned off at all run levels. Shut it off via the service command and renabled in the GUI and it persists a reboot. The fact that you reboot after updates makes it fun.

I just attended a "dog and pony" regurading ESX3.5 and apparently the Linux console is a thing of the past. As a Linux guy, I'm not sure how I feel about that. Forcing us to the gui or a proprietary CLI will reduce these sorts of issues, but I like the power of being able to script and manage the console in a familiar shell.

Thanks again for the fix.

Alan

[jonathanp]

Is not the same as:

esxcfg-firewall -e ntpClient

esxcfg-firewall -l

??

Jon

[eahatch]

Hmm . . . I would have expected that to do the same thing, but I'm not seeing it. That opens a port in the firewall (which is not typically necessary with an outbound connection and related/established connections being allowed back in) but I don't see the change updated in virtual center.

Don't get me wrong, Virtual Center is one of the most elegant tools I've ever used. This particular issue, however, is counter intuitive as I would never have thought to use a firewall configuration tool (CLI or otherwise) to manage a service.

Of course, I may be missing something obvious here.

Thanks.

Alan

[jlanders]

No, you're not missing anything obvious, it's just a bit confusing. Virtual Center manages ESX, so if you're going to do management underneath VC, you've got to be aware of what VC is trying to do.

The state of the NTP service gets held in VC. But in 3.0.x, you can't configure the NTP client in Virtual Center. You need to get to the Service Console. If you change the state of the service with chkconfig and the firewall configuration with esxcfg-firewall in the Service Console, Virtual Center doesn't get notified about your change. Periodically, Virtual Center checks with the ESX server and notices a difference in the NTP configuration. VC then reverts the NTP service and firewall setting to whichever state VC had cached.

The upgrade issue is a known bug where the NTP state isn't saved across an update. A fix will be in a future release. Also, in the future, you'll be able to configure unicast NTP with Virtual Center and the configuration dialog boxes will make more sense.

Hope this helps,

Joe