NTP time on VMs and Domain Controller in DR scenario

[snowmizer]

In our current production environment we have on physical DC (all servers are Windows 2003) and one DC that is a VM. In the past we had severe issues with time drift on our VM DC. To correct this we were told to set up the physical DC as the time source, set all ESX hosts NTP to point to the physical DC, and then set VMWare tools on each VM to sync with the ESX host. This has worked fine for us over the past 2 years. Now we're testing our DR scenario. Since we don't have a physical DC (nor do we want one to start our domain in DR mode) to act as a time source we have set up a Linux NTP server to go to the Internet for time. Then we pointed each ESX server to the Linux NTP server for time. This seems to work fine however since our virtual DC isn't an NTP time server the physical Windows servers don't have a DC to get time from. Because of the time drift issue I hesitate to setup NTP on the virtual DC.

Has anyone else encountered this problem? What has anyone done to counteract this issue?

Thanks for the help.

[Rockapot]

Why not just point the VM DC at the same Linux NTP time source as the ESX hosts are pointing to?, or alternatively just set VMtools to do the time sync for the VM DC. The VM DC will then always collect time from the ESX hosts.

Personally I just use VMtools all the time to get time information from the ESX hosts and the ESX host collects time from an NTP time source (be that a physical DC or other NTP source)

Whilst I have seen time drift occur before, it usually has only occured on Linux servers however with the basic model above time drift never seems to occur anymore.. (thank goodness)

Carl

[snowmizer]

My problem isn't so much getting the VM DC to sync. It's W32time service is off and VM tools is set to sync to the ESX host f(which point to the Linux NTP server) or time. My issue is that I've only got this one DC in my DR scenario and I've got physical Windows servers expecting to get their time from the DC (which isn't an NTP server). From what I've read I may need to set the NTP source manually on the physical servers. I just wanted to know if anyone else had this scenario. If so I was curious how they handled it. Time is so touchy I don't want to make any mistakes.:8}

Thanks.

[Rockapot]

Ok i see, no probs

Well personally I would set the VM DC to sync with the ESX hosts via VMtools because the ESX hosts are collecting time from the Linux host so in effect the time would all be synced to the same time source.

Interested to see what others have to say on the subject..

Good luck.

Carl

[snowmizer]

Thanks for the reply. I guess my problem ultimately comes down to the fact that my VMs can get time from the ESX hosts but my physical windows servers want a DC. I don't have and don't want W32time configured on my DC because of time drift. It's a circle where I want both to work but don't know (at this moment) how to make it work.:)

[Rockapot]

Ah ok i see.., I never noticed the fact that there where other physical windows servers (which wont run VMtools). I understand your concern in more detail now.

Cheers

Carl

[snowmizer]

So just to clarify...you are using VMtools for the sync on your Windows VMs. Am I correct that you disabled the Windows Time Service on these VMs?

[Mark_Brophy]

There was a great breakout topic at VMWorld2008 talking about this very scenario.

According to the VMWare speakers via this session, it is best practice to have any virtualized DC to get its time from a source outside of the ESX environment. DO NOT USE VMWARE TOOLS ON DOMAIN CONTROLLERS FOR THIS. Heard straight from VMWare themselves.

So here is what to do for Windows.

Follow this:

and point your Authorative DC to any server on your network or Internet that is running NTP.

From there do nothing else with Windows. The remaining domain controllers will get their time from the Authoratative DC. In turn all windows servers and client OS (2000/XP/Vista) get their time from the other DC's using NTDS5.

If you follow this heirarchy, you will be all set.

NTP time server -> Authoritative Time Domain Controller on your network -> All other domain controllers on your network-> Windows member servers/clients.

Windows will continue to get time from the Domain regardless if VMWare tools are used or not. VMWare tools is great for stand alone clients where there is no domain functionality, but otherwise, let Active Directory do its thing or else the Kerberos will become cranky.

For your Linux/ESX stuff you can configure them to get time from the same NTP time server that your Authoratative DC gets its time from therefore your time settings will be consistant.

-Mark

[Rickard_Anderss]

This is getting really confusing. I've searched around for a solution and everyone has a different method of doing this. If what Mark just said is true, the vmware folks really need to update KB 1318:

"The most accurate way to keep guest operating system time

synchronized with real time is to use the VMware Tools time

synchronization function. *You should not use the Windows Time

service or other form of clock synchronization meant for physical

machines to set the time in the guest operating system.*"

The way I see it, there are three possibles when setting up time synchronization in a domain:

1. Sync with external NTP on PDC and disable time synchronization in vmware tools (Mark's suggestion).

2. Enable time synchronization in vmware tools and set the "NoSync" option for windows time service.

3. Sync with external NTP on PDC and enable time synchronization in vmware tools (both at the same time).

As far as I know, there are issues with all three. The issue with the first one is that if the server is powered off for some time and then started up, it will be out of sync for a longer period of time than with VMware tools sync.

The issue with the second one is that the other domain controllers will not "trust" the PDC because it is not syncing with an external source.

And the issue with the last one I'm not sure about, but it makes sense that running two different time synchronization mechanisms at the same time is a bad idea.

What's the verdict on this? Can someone please enlighten me?

Cheers,

Rickard

[Strago]

I'm adding a virtualized DC to my domain and am at this same decision point, Rick or anyone else, any update?

Thanks..

[Strago]

Oh actually it appears the KB referenced above has had the following added to it:

When You Must Run Windows Time Service

If you use a virtual machine as a primary domain controller for a

Windows network, the primary domain controller must run the Windows

Time service as a time server, to provide time to secondary domain

controllers and other hosts on the network. However, that primary

domain controller does not need to use the Windows Time service as a

client to receive time synchronization input for its own clock. You can

still use VMware Tools to synchronize the virtual machine's clock while

running the Windows Time service in a server-only mode. For

instructions on setting up the Windows Time service this way, see the

Microsoft document titled "The Windows Time Service," at download.microsoft.com/download/2/0/f/20f61625-7b2a-4531-b007-1c714f1e51b7/wintimeserv.doc. Search the document for the NoSync registry option.

I will experiment and post results.

[JoshieBoi]

I would appreciate if you post back your findings.

My experience is that with NoSync, the domain controller will log an event explaining that it has not contacted an NTP server for x amount of time, assume it is no longer trusted and stop serving time clients - not a good thing.

I watched the vmworld session mentioned, and using NoSync was the recommended approach (if using vmware tools for time sync).

I experimented with many different w32time reg keys with no change in behavior.

The workaround, as you have alluded, is to let windows time do its thing, alongside vmware tools also performing time sync.

The only problem I can conceive is that the clock is more likely to "jitter" or swing back and fourth by very very small amounts as vmtools and w32time fight each other. However, this is a lot better than the clock becoming "fast", which does happen, and is not corrected by vmware tools automatically (unless you restart the service or toggle the time sync setting manually).

Quite frankly, this issue bugs me (keeps me up at night). Clarification is needed from VMware regarding time sync on Windows domain controllers.

Regards,

Joshua Smith (VCP)

[Strago]

Using their method I still get weird event log stuff like you mentioned, but it looks like it works.

I have 2 DC's in my VMWare AD test bed. I set up the PDC emulator DC using the solution described above (point to esx using vmtools and set NoSync regkey), and I kept my other DC and all member servers with their default time sync options (i.e. leave HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\W32Time\Parameters\Type = NT5DS).

After an hour I got this on my PDC emulator DC:

system event id 50

The time service detected a time difference of greater than 5000 milliseconds for 900 seconds. The time difference might be caused by synchronization with low-accuracy time sources or by suboptimal network conditions. The time service is no longer synchronized and cannot provide the time to other clients or update the system clock. When a valid time stamp is received from a time service provider, the time service will correct itself.

Then 6 days later I got this:

system event id 36

The time service has not synchronized the system time for 86400 seconds because none of the time service providers provided a usable time stamp. The time service is no longer synchronized and cannot provide the time to other clients or update the system clock. Monitor the system events displayed in the Event Viewer to make sure that a more serious problem does not exist.

Then no more time-related events for the past week.

AD seems to be working functionally, and w32tm /monitor shows that the DC's are still in sync, so these event ID's seem like false positives to me.

Does anyone know of a definitive way to test if member servers are indeed correctly getting their time from DC's? Like a resource toolkit utility or something?

[JoshieBoi]

Hi Mark,

You should be able to tell from the event logs on the domain members what is going on, the client's must get their time from a domain controller when using the NT5DS key. W32Time event entries should give you an indication.

However I have come across the following workaround/solution, that I implemented 2 weeks and and have not had a problem or any event log warnings/errors:

+ Create a scheduled task on your virtual domain controllers that runs every 12 hours that restarts the w32time service (net stop w32time && net start w32time). This prevents W32time from not serving clients due to the max clock age being exceeded - 86400 seconds.

Problem fixed.

My environment is as follows:

ESX hosts synching with the same NTP server on the Internet.

VMware tools synchronising time on all VMs - This includes all domain controllers, including the PDC.

NoSync set on domain controllers, including the PDC. (note: all my domain controllers are virtual).

W32time running on domain controllers, including PDC.

W32time disabled on VMs that are domain members (not domain controllers).

W32time running "normally" on all non-VM domain members.

No event log errors, and all computers are happy.

The only time issue I have is with some VMs running "fast" - eg up to 8 seconds ahead over a 2 week period. This can be fixed by restarting the VMware tools service, which causes a one time correction of the clock backwards. VMware tools does not turn the clock backwards during operation, only at service startup, or when the time sync option is toggled off. It is possible that the Descheduled Time Accounting option of VMware tools may resolve this, but I have not tried it due to its current unsupported status.

If you use any major network monitoring software, you should be able to create a VBscript that compares the time of each of your servers against the PDC. You can then setup alerts if things start to drift beyond acceptable limits - eg. I have email alerts setup when a system drifts more than 15 seconds from PDC time.

Regards,

Joshua Smith (VCP)