VMware
1 2 3 Previous Next 30 Replies Last post: May 24, 2005 8:21 AM by osde.info  

GSX3 and linux kernel 2.6 guest findings posted: Apr 29, 2004 9:17 AM

Click to view image's profile Lurker 3 posts since
Apr 29, 2004
i just spent a good two days hunting why my linux 2.6 guest running on windows xp was giving slow time, and couldnt use ntpd to sync it up. i changed the HZ in include/asm-i386/param.h in the kernel from 1000 to 100 and it started to work fine. it's probably the difference between XP's clock thats causing it, but i would imagine that GSX should be compensating, not the guest itself.

Re: GSX3 and linux kernel 2.6 guest findings

1. Apr 29, 2004 11:20 AM in response to: image
Click to view petr's profile Champion VMware Employees 7,223 posts since
Jul 10, 2003
System has too much work to do with HZ=1000 that there is not enough time left for guest itself to run... By changing host's HZ to 100Hz you effectively decrease VMware's overhead 10 times - for example with HZ=1000 CPU has to switch 2000 times (4000 times if you run SMP kernel on host) between host & guest. With 100Hz there is only 400 task switches, decreasing task switch overhead by factor of 10.

Re: GSX3 and linux kernel 2.6 guest findings

3. Apr 29, 2004 9:26 PM in response to: image
Click to view TimMann's profile Enthusiast VMware Employees 89 posts since
Oct 2, 2003
We don't yet have enough experience with running 2.6 in a guest to put that recommendation in the documentation. In fact, this is the first report I've seen of a slow guest clock in this situation. Petr is right that the 1000 Hz requirement of the guest is a heavy load, and it was good thinking for you to try reducing guest HZ to 100.

Re: GSX3 and linux kernel 2.6 guest findings

4. Apr 30, 2004 11:32 AM in response to: TimMann
Click to view petr's profile Champion VMware Employees 7,223 posts since
Jul 10, 2003
My notebook (Compaq Evo 800N with 1.8GHz P4) makes hearable sound when it runs VMware with 1000Hz clocks on both host and in the guest. Apparently Compaq did not thought about CPU changing its power consumption & power state 2000 times a second... As with 1000Hz gnome cpu monitor signalled that approx. 50% of time is spent in the kernel and 50% of time it is idle, I decreased HZ to get rid of that beeping I heard whenever guest was idle...

As bonus now system has much better latency it seems - though I cannot explain why decreasing HZ also decreases latency... So I believe that problem are ACPI state transitions and not VMware per-se, just VMware's bursty behavior confuses ACPI sufficiently to get mad (as it seems fine if I run 1000Hz guest on 100Hz kernel, only if kernel uses 1000Hz jiffies things go bad).

Re: GSX3 and linux kernel 2.6 guest findings

5. Jun 23, 2004 11:41 AM in response to: image
Click to view samtregar's profile Lurker 4 posts since
Jun 8, 2004
This worked for me too, running a Fedora Core 2 guest. With HZ=1000 the system is unnacceptably slow. After recompiling with HZ=100 everything is zipping along as usual. Thanks!

-sam

Re: GSX3 and linux kernel 2.6 guest findings

6. Oct 16, 2004 3:04 PM in response to: TimMann
Click to view linux_blade_guy's profile Novice 16 posts since
Oct 8, 2004
I ran in to the same problem on my SLES9 (2.6.5) virtual machines. I was unable to keep time synchronized with NTP at all - we use eDirectory which relies heavily on time synchronization so this effectively killed us until I could find a fix.

Like others here I changed guest HZ to 100 and was able to get time synchronization. Is there any negative effect to doing this on the host as well? I am using a PXE boot environment and keeping all the kernels the same simplifies management.

Re: GSX3 and linux kernel 2.6 guest findings

7. Oct 20, 2004 2:19 AM in response to: image
Click to view sindu's profile Lurker 2 posts since
Oct 20, 2004
Hi all,

I'm running Debian Sarge with kernel 2.6.8-1-386 and I had the same problem.

I followed everything in KB 1420, giving noapic, nolapic, apic=off, lapic=off and they all didn't work for me. I tried recompiling the kernel with HZ=100, but the problem still existed.

Then I tried passing "acpi=off", and voila problem is solved.

I wanted to know what's different, so I diff the 'dmesg' output. Here it is:
-Using pmtmr for high-res timesource
+Using tsc for high-res timesource

Ah, so I tried passing clock=tsc and problem solved without turning off ACPI. And even better, I tried using the stock kernel (2.6.8-1-386 & HZ=1000) and the clock is still in sync.

So here I'd like to share my finding. I hope someone with the same problem will not have to work all week long like me. :p

I'd be happy if anyone can explain why it fails with pmtmr clock.

Re: GSX3 and linux kernel 2.6 guest findings

8. Oct 21, 2004 12:01 PM in response to: sindu
Click to view linux_blade_guy's profile Novice 16 posts since
Oct 8, 2004
I tried with clock=tsc but I was unable to get ntp working with SLES9 (2.6.5). I ended up going back to my kernel compiled with HZ=100

:(

I should also add that running a kernel with HZ=100 on the host AND guests causes WAY more clock jitter than running both with HZ=1000. The only combo that seems to work for me is HZ=100 for the guest and HZ=1000 for the host. It seems even with HZ=100 the clock jitter on my guests is much worse than a real clock - at least it is within NTP's spec however.

Message was edited by: linux_blade_guy

Re: GSX3 and linux kernel 2.6 guest findings

9. Oct 25, 2004 5:34 PM in response to: image
Click to view williamsjj's profile Novice 15 posts since
Sep 9, 2004
I've had the same issue with SLES9. Wish I had known the fix back in August when we set this up. Now I've got 12+ VMs running and configured that need to be recompiled. VMware needs to fix this. Not too mention that they need to vet SLES9 for a Guest OS...along with AMD64 guest OS support. The clock thing's killing me.
Click to view linux_blade_guy's profile Novice 16 posts since
Oct 8, 2004
Reliability on our systems has been very poor due to this time synchronization problem. Before NTP would give up after about 20 minutes - now it holds timesync for a few days but eventually looses it.

This is a MAJOR problem for us - we may have to scrap our VMWare rollout. Hopefully they get this fixed or I'm certainly not going to tell anyone to run production 2.6 kernel servers on VMWare.

Re: GSX3 and linux kernel 2.6 guest findings

11. Oct 28, 2004 8:38 PM in response to: image
Click to view sindu's profile Lurker 2 posts since
Oct 20, 2004
About a week ago, I posted my finding about "clock=tsc".

Well, after running it for several days, I found that my clock still got drifted a little bit. Running with HZ=100 didn't help me either.

I found this excellent explanation about the clock in 2.6:
http://groups.google.com/groups?hl=en&lr=&selm=1EoHw-71a-11%40gated-at.bofh.it

So I passed "clock=pit" to the kernel, and now I've been running the 2.6.8 kernel (HZ=100) for 3 days with the clock perfectly in-sync.

FYI, the GSX host is Windows 2000, and the Linux guest is Debian Sarge running 2.6.8-386.

Btw, you also need vmware-guestd to keep the clock in-sync.

Re: GSX3 and linux kernel 2.6 guest findings

12. Nov 15, 2004 3:09 PM in response to: sindu
Click to view TimMann's profile Enthusiast VMware Employees 89 posts since
Oct 2, 2003
On the clock= question, I have some partial info, but it doesn't seem to mesh completely with everything that others have posted here.

The default in a vanilla 2.6 kernel is for Linux to try clock= options in the order cyclone, hpet, pmtmr, tsc, pit. The cyclone and hpet timers aren't present in a VM. The pmtmr is present, but I think Linux decides not to use it if you turn off ACPI support.

In SuSE SLES9, there is a patch that changes the preference order to put tsc in front of pmtmr. So the effect is that on SLES9 in a VM, you get clock=tsc by default, while in a vanilla 2.6 kernel in a VM you get clock=pmtmr by default.

Between 2.4 and 2.6, some complex code has been added to the routines that are used if clock=tsc. 2.4 didn't have the clock= option, but the code that is now under clock=tsc is what was used on all machines with TSCs. In my testing on SLES9, clock=tsc makes the clock run too fast in a VM. I'm going to have to read the Linux code much more carefully to find out why and whether we can do something to tame it.

The good news is that my SLES9 VM seems to keep time okay with clock=pmtmr. Using clock=pit should work too (though I haven't tried it yet), but that causes gettimeofday in the VM to be a lot more coarse-grained, so it seems preferable to use pmtmr if it works.

The thing that worries me here is that a previous poster (sindu) reported that using clock=tsc worked better than clock=pmtmr on Debian Sarge with its 2.6.8-1 kernel; however, later he said that clock=pit was still better. I can understand clock=pit being better than clock=tsc, but not why clock=pmtmr was worse than clock=tsc -- that doesn't reproduce for me. Maybe it's because SLES9 uses a 2.6.5-based kernel? Ugh.

Re: GSX3 and linux kernel 2.6 guest findings

13. Nov 23, 2004 9:41 PM in response to: image
Click to view linux_blade_guy's profile Novice 16 posts since
Oct 8, 2004
I've had very good results with clock=pit. The clock drift with the other routines was too much for NTP to keep synchronized, while pit still seems to have some jitter it's almost insignificant comapred to the problems with other clock settings.

I've been running for a couple of weeks now and all my clocks are staying synchronized to within a milisecond - beauty!

Re: GSX3 and linux kernel 2.6 guest findings

14. Nov 24, 2004 11:18 AM in response to: TimMann
Click to view TimMann's profile Enthusiast VMware Employees 89 posts since
Oct 2, 2003
I've completed some additional testing and study of the Linux code. The bottom line is that I have to recommend clock=pit. You'll also need to be sure that VMware Tools are installed and tools time sync is on, or else your clock will lose time in this mode.

There is a bug in the Linux 2.6 clock=tsc code, and a similar but less severe bug in clock=pmtmr, that can cause the clock to gain time. I'm not sure how often the bug hits on real hardware, but it hits frequently in a VM. VMware Tools aren't prepared to correct for the guest clock running too fast, so that makes these modes unusable. For details on the bug, see my post to the Linux kernel mailing list, at http://gossamer-threads.com/lists/linux/kernel/494604.

The bug is in code that tries to make up for lost clock interrupts ("ticks"). The code gets some false positives when trying to detect lost ticks and ends up adding extra ticks to the clock when it shouldn't. With clock=pit, there is no attempt to detect lost ticks, so this doesn't happen. Unfortunately some ticks really are lost, which is why you need to run VMware Tools time sync to keep the clock from running slow.

VMware Beta Programs

Want to be Considered for Future Beta Programs?

Learn More

VMware Developer

Download SDKs, APIs, videos,
training, and more in the Developer community.

Learn More

Developer
Sample Code

Increase your developer productivity with VMware API sample code.

Learn More

VMworld
Sessions & Labs

Online access to the latest VMworld Sessions & Labs and online services.

Learn more

Purchase PSO Credits Online

Purchase credits to redeem training and consulting services online.

Buy Now

Community Hardware Software

View reported configurations or report your own.

Learn More

Only VMware ... Delivers Nexus 1000V

Ensure consistent, policy-based network capabilities to virtual machines across your data center.

Learn More

Communities