I'm having some issues with virtual machines, being slow to respond to the first request, after having beeing idle for some time.
A virtual Windows 2003 R2 server, fully patched, with 10GB HD, 768MB RAM, Normal priority, installed with Apache Tomcat 5.5, running on an ESX host 3.0.1 (32039).
If the machine is allowed to sit idle for a while (30 minutes and upwards), the first request for a web-page (have not tried with any other traffic) is extremely slow.
From 20 to 45 seconds, i have to sit and wait for the response.
Ping times are excellent during this period.
When the page finally appears, subsequent page requests have excellent response times.
I realize that there may be a small delay on the first page request, but i would rather not exceed a couple of seconds or so.
One thing i've noticed, is that "Guest Memory Usage" goes really low, when the machine is sitting idle.
I've seen it as low as 18MB.
After the first request, the Guest memory usage rises to 65-75MB.
This, kind of, all makes sense, except for the 45 seconds wait time.
The problem may seem small, as it's only the first request after idle time, but the customers sure doesn't think so.
I've tried to set the VM to "High Priority", and also tried to set a memory reservation to no avail.
Have also tried re-creating the VM, using the same disks.
Have tried another ESX host running 3.0.0 (27701).
Also, it seems that just logging on to the server remotely (we use Netop Remote Control), making some kind of activity, makes the web-request respond as it's supposed to.
Have any of you guys experienced anything like that?
Can't seem to figure out whether it's a VMWare issue, or a Windows/Tomcat issue.
What's the memory and CPU load on the host look like? Is there a swap i/o rate?
You need to power a system off for the memory reservation to take effect.
Have already tried the memory reservation to no avail.
Tried to reserve 100MB, as it rarely goes above that, at least for these "first response" tests i'm doing. Dunno if that's sufficient.
Have had it powered down since the reservation was made (i'm assuming we're talking about the VM here, right!?).
Anything else you can think of?
Not quite sure how to check for the swap i/o.
The host is using 8 out of 12GB ram, so i don't think that we overcommit on mem.
Any input appreciated.
It could be the balloon driver in the VMware tools that takes your memory. Check the Ballon drive in the VMware tools.
Really sounds like an OS/Apache issue. Do you have any power saving features turned on inside the VM? If your ping times are fine then I would start with tuning Apache & Tomcat.18MB of guest memory usage is way too low. ESX cannot use it's memory page sharing features on JVM's so your guest mem usage should be alot higher. If you look at Task Manager how large is your java.exe? WHat do you have your minumum/maximum JVM sizes set to in Tomcat? You also might try temporarily disabling the ESX memory saving features by setting a memory reservation equal to the amount of RAM you assigned to the VM.
Well you are not the only one to see those symptoms. I see it with our WSS 2.0 installation as well and other installations. After a danish VMUG where one of the vmware kernel developers had a presentation, one of my coworkers asked him about it and he would look into it (no news yet).
As you say it looks like the memory is swapped out even though no ballooning or memory pressure occurs and the vm has all memory reserved. It's a problem you need to report to vmware support. It still does it on 3.5. You should try to remove the baloon and shared folder option from the vmware tools (tip from the vmware kernel developer).