I have been trying to migrate some services from one VM to another, where the old VM is SLES 9 (2.6.5-7.276-smp) and the new VM is CentOS 5 (2.6.18-53.1.14.el5). The VMs both have 768MB of memory and 2 CPUs. VMware environment is ESX 3.0.2.
Whenever even light to moderate I/O is taking place, the machine because almost unresponsive. Some examples:
- Making a new ext3 filesystem on a partition. It starts writing the inode tables out, and the first few hundred go fine, but then it starts -crawling-, like 1 inode table per 20 seconds. A "vmstat 3" or a "sar -B 3 0" takes about 20-30 seconds for each new line of output. The output itself will show the system at like 20-30% iowait. No major page faults reported by sar.
- Rsyncs that I have seen take 20 minutes on older kernels will take several hours on this system. And during the rsync, attempting to do anything else on the system (ssh to it, run 'ls', etc...) pauses for many seconds before returning.
- Any attempt to run a yum update/install/search results in major pauses and hangs on the order of minutes. It often is 1-2 minutes before yum produces any output at all.
- I have noticed very similar behavior out of a Fedora Core 7 VM on the same ESX host.
During these episodes, system load numbers stay low. The machine remains pingable. Just the rest of the system becomes unresponsive. In the case of making ext3 filesystems, the slowness lasted about 5 minutes, then it picked up speed again. In general, it lasts anywhere from seconds to minutes.
Things I have tried already:
- Updated all of the packages to latest from CentOS, including kernel.
- Went down to 1 CPU for the VM just to see if that helped (it did not).
- Run vmstat and sar and atop during episodes to look for suspect numbers. Noting really obvious yet, everything sort of seems within "normal" params but I am not an expert with these tools. The only thing that seems to increase is iowait time.
I have read similar-sounding threads online that seem to point a finger at the kernel tick system or the tickless kernels. I have seen a lot of people trying various kernel flags to alter the behaviour of the ticks but they tend to report back that it had no effect.
Questions:
- Does this sound familiar to anyone? Is this a known issue and if so, is there a fix?
- Is there a way for me to debug this from the ESX host side, to see if it is seeing lots of io requests from the VMs? Virtual Center performance graphs show very little activity.
Happy to provide more debugging data, just didn' t know what would be useful to include here yet.
Thanks!
