Linux guest clock too fast

[calaver4]

I'm runnung VMware Server 1.0.4 on Windows XP SP2 32 bit, the CPU is AMD Athlon 64 X2 4800+ 2.51 GHz.

On the guest VM, that is a CentOS 5.1, the clock run twice over fast than in host machine.

Trying to fix this problem I've done the following things:

- On the host

In the BIOS I've disabled Cool and Quiet.

Modified C:\Documents and Settings\All Users\Application Data\VMware\VMware Server\config.ini adding

host.cpukHz = "2510000"

host.noTSC = "TRUE"

ptsc.noTSC = "TRUE"

- On the guest

I've installed the modifed CentOS kernel customized for guest VM.

I boot guest using the following kernel parameters: clocksource=pit nosmp noapic nolapic

I enabled time sync by VMware Tools

None of the above is able to slow down the clock.

Now I've no more ideas, I hope someone can help me.

Thank in advance.

[Chris_Howard]

You might find the Timekeeping in VMware Virtual Machines white paper useful - page 11 describes how linux keeps time, and specific issues running in VMs.

If you found this helpful please consider the use of the Helpful /Correct buttons to award points. Thanks !!

[dxb]

I've found that the clocksource=pit setting can cause the clock to run fast in RHEL5.1. Try clocksource=acpi_pm instead.

We have more trouble with the clock running slowly whenever the VM gets busy.

[AWo]

Enable the VMware TimeTracker logging:

Add

timeTracker.periodicStats="True"

timeTracker.statsIntercal="10" (for 10 sec. tracking intervall"

to the guests .vmx file (guest needs to be restarted) and post the entries from vmware.log (start with "TimeTrackerStats").

General background:

The issue with virtualization and (mostly) Linux guests is, that Linux (and other OS's, too) are using the PIT (Programmable Interrupt Timer) to count the timer ticks. Linux requests a timer interrupt in frequencies of 100 - 1000 Hz per OS instance + per (virtual) processor (n+1). One bad thing about that is, that this will consume ressources, even if the system itself is idle.

The other bad thing is that only the guest which actually owns the processor receives the interrupt, so many interrupts are lost (Lost Timer Tick). That's why the time of a virtualized machine would always run too slow, if there is no compensation.

Windows uses the PIT, too, and other different mechanisms/time sources which can even change if you install software, for example Apple's Quick Time.

As loosing these ticks happens on hardware, as well, there is a Lost Timer Tick correction algorithm within Linux to compensate this. This works well enough on pure hardware.

But there's a second compensation mechanism if you run it onVMware: VMware tracks the interrupts lost and sends them in a timely compressed manner to the guest later (in fact not all are send, some lost ticks needs to be deleted by the Vmware time management). That's the reason why the time doesn't run consistent and seems to "jump" sometimes.

So there are two mechanisms to keep the time accurate and that could lead to a clock running too fast (overcompensation). There is no way to slow it down with the VMware tools, yet, nor do they set the time back. Slowing down the clock is planned for a future release of the tools.

The suboptimal usage of the PIT has changed from Kernel 2.6.18 for 32 Bit / 2.6.21 64 Bit. In newer Kernels Linux uses a different lock source (TSC, Time Stamp Counter) where it can read the time directly (has not to count the ticks to generate the time) and without using an interrupts. Another advantage is that his time device can be read faster that the PIT.

AWo

[calaver4]

Here is the log file.

The time shown in the log file is in sync with the host clock. In the VM the clock runs twice over the host clock.

For this test I have not set the clocksource kernel parameter and the only option set in the config.ini file is host.cpukHz = "2510000".

Looking at this log file I realize that at boot time the VM clock was behind 57 s, about 1 minute later VM clock reach the host clock and 2 minutes later the VM clock is about 4 s behind, but the date command in linux say that the clock is forward.

bye

[AWo]

The time shown in the log file is in sync with the host clock.

Should be as the host has the accurate time and the log is written by the Time Tracker.

In the VM the clock runs twice over the host clock.:-( Unfortunately you can't slow down the clock with the VMware Tools, yet.

Looking at this log file I realize that at boot time the VM clock was behind 57 s,

That's what the line "TimeTrackerStats behind by 144245424544 cycles (57468296 us); running at 300%; 0 stops, 3 giveups" says.

"300%" means, that the virtual machines clock is running faster than normal (catchup is needed).

"3 giveups" means that the Time Tracker is aware that the guests clock is behind and forces the Tools to stop the catchup and set the clock forward (3 times in this intervall). Maybe this leads to overcompensation and thereby the guests clock (not the virtual machine clock) running too fast.

Which kernel version do you use?

Try to limit the catchup process (you can play with the values):

timeTracker.catchupPercentage = 200

; when to start the catchup (in msec)

timeTracker.catchupIfBehindByUsec = 50

; when to stop the catchup and set the clock forward (in msec)

timeTracker.giveupIfBehindByUsec = 60000000

If this doesn't help try to use NTP and switch the VMware Tools timesync off.

AWo

[BUGCHK · Accepted solution]

AMD Athlon? outch! outch!

I've tried a few hints and finally pinned VMs to specific pCPUs:

• How to Run VMware Hosted Products on Systems on Which TSCs Are Not in Sync

http://kb.vmware.com/kb/2039

In the end I traded the system against one with a Intel Core Duo CPU because the notebook's other quirk was a chipset too slow on USB transfers - I have to move lots of GigaBytes...

[calaver4]

BUGCHK you are a myth!

Assigning VM to one core fixed the clock problem.

Are there any drawbacks assigning VM to specific core?

Thank you very much.

[BUGCHK]

The drawback I have seen is that you have to do a "static load balancing" as the virtualization layer can no longer (re-)distribute VMs across physical CPUs. And I guess a multi-vCPU VM is still affected by the processor problem. While it is unsupported, I tried to run a 2-vCPU ESX VM and the bootstrap failed from time to time - I have not seen this on a Intel Core Duo CPU.