elszben
Contributor
Contributor

Mysterious high load in Linux

Hi,

We are running a cluster in vmware esx 3.5i and we are experiencing wierd load values on the linux virtual machines.

The

cluster contains 4 Linux VMs and 4 VMs with other OS. The whole cluster

is in idle state, none of the VMs are doing much.

The problem is that on some of the linux machines the load is high even if it is not doing anything.

According to the top command the cpu is heavily (~50%) used but none of the processes are doing anything.

There are only ~5 processes running each uses ~1% CPU time.

Output of top:

15:14:47 up 2:49, 1 user, load average: 0.53, 0.89, 1.52

Tasks: 187 total, 2 running, 185 sleeping, 0 stopped, 0 zombie

Cpu(s): 29.7%us, 15.4%sy, 0.3%ni, 54.2%id, 0.0%wa, 0.0%hi, 0.3%si, 0.0%st

Mem: 2034688k total, 688784k used, 1345904k free, 0k buffers

Swap: 0k total, 0k used, 0k free, 508088k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND

20245 xxxxx 16 0 2316 1088 764 R 1.3 0.1 0:01.18 top

29731 root 25 0 2524 1280 984 S 1.3 0.1 0:00.04 xxxxxxx

22811 xxxxxx 16 0 9076 1544 1060 S 1.0 0.1 0:01.60 sshd

5581 root 16 0 235m 24m 9140 S 0.7 1.2 0:27.83 java.exe

29775 root 25 0 2532 1260 956 S 0.7 0.1 0:00.02 xxxxxxxxx

2087 root 16 0 103m 1008 744 S 0.3 0.0 0:45.19 nscd

2937 root 16 0 1030m 1120 796 S 0.3 0.1 0:12.71 xxxxxxx

2990 root 17 0 1030m 1812 1360 S 0.3 0.1 0:18.83 xxxxxxxx

6362 root 16 0 239m 27m 9800 S 0.3 1.4 0:38.30 java.exe

1 root 16 0 720 272 236 S 0.0 0.0 0:03.45 init

2 root 34 19 0 0 0 S 0.0 0.0 0:00.21 ksoftirqd/0

3 root 10 -5 0 0 0 S 0.0 0.0 0:00.03 events/0

4 root 10 -5 0 0 0 S 0.0 0.0 0:00.00 khelper

5 root 10 -5 0 0 0 S 0.0 0.0 0:00.00 kthread

7 root 20 -5 0 0 0 S 0.0 0.0 0:00.00 kblockd/0

10 root 20 -5 0 0 0 S 0.0 0.0 0:00.00 khubd

87 root 20 0 0 0 0 S 0.0 0.0 0:00.00 pdflush

88 root 15 0 0 0 0 S 0.0 0.0 0:00.00 pdflush

90 root 20 -5 0 0 0 S 0.0 0.0 0:00.00 aio/0

89 root 25 0 0 0 0 S 0.0 0.0 0:00.00 kswapd0

163 root 10 -5 0 0 0 S 0.0 0.0 0:00.00 kseriod

208 root 16 -5 0 0 0 S 0.0 0.0 0:00.00 ata/0

2096 root 16 0 4956 1096 768 S 0.0 0.1 0:00.01 sshd

2101 root 16 0 4116 4116 3132 S 0.0 0.2 0:01.06 ntpd

2104 nobody 15 0 1556 416 328 S 0.0 0.0 0:00.00 portmap

2107 root 10 -5 0 0 0 S 0.0 0.0 0:01.51 rpciod/0

2108 root 19 0 0 0 0 S 0.0 0.0 0:00.00 lockd

2128 root 16 0 1520 496 424 S 0.0 0.0 0:00.01 agetty

2130 root 16 0 1520 492 424 S 0.0 0.0 0:00.00 agetty

2132 root 16 0 1520 496 424 S 0.0 0.0 0:00.00 agetty

2134 root 16 0 1520 496 424 S 0.0 0.0 0:00.00 agetty

2138 root 16 0 1520 492 424 S 0.0 0.0 0:00.00 agetty

2140 root 16 0 1516 492 424 S 0.0 0.0 0:00.00 agetty

2142 root 16 0 1516 492 424 S 0.0 0.0 0:00.00 agetty

2182 root 15 0 2460 928 824 S 0.0 0.0 0:02.07 xxxxxxxxx

2184 root 21 0 2524 1132 932 S 0.0 0.1 0:00.01 xxxxxxxxx

2185 root 24 0 2524 1192 948 S 0.0 0.1 0:00.08 xxxxxxxxx

2205 root 16 0 19248 1072 924 S 0.0 0.1 0:02.99 xxxxxxxx

2405 root 16 0 1031m 3472 2900 S 0.0 0.2 0:01.59 xxxxxxxx

2407 root 15 0 1030m 1340 948 S 0.0 0.1 0:00.06 xxxxxxxx

2410 root 16 0 1030m 1048 736 S 0.0 0.1 0:00.00 xxxxxx

The output of vmstat:

procs -


memory----


-swap -


io---- -system-- -


cpu------

r b swpd free buff cache si so bi bo in cs us sy id wa st

3 0 0 1346028 0 508084 0 0 0 0 420 634 39 26 34 0 0

Has anyone ever experienced something similar ? Any idea how to fix it?

For testing we use other simulated environments and real hardwares as well and this fault does not happen in any of the other environments.

I guess this is a resource management problem somewhere with ESX but I don't really know what to try.

Thanks in advance.

Best regards,

Laszlo Benedek

0 Kudos
5 Replies
AWo
Immortal
Immortal

Have you installed the VMware Tools in the guest?

What does "esxtop" on the ESX console tell you about the load of the guest?

How many vCPU's are assigned to the guest?

Depending on which timer source used in the Linux guest there can be some amount of "basic" load.


If you found this information useful, please consider awarding points for "Correct" or "Helpful" replies. Thanks!!


AWo

VCP / vEXPERT 2009

vExpert 2009/10/11 [:o]===[o:] [: ]o=o[ :] = Save forests! rent firewood! =
elszben
Contributor
Contributor

Thanks for the quick reply!

What difference should the vmware tools make ? Isn't that for time synchronization ? Does it have a load related part ? These linux distributions are very special, it is not trivial to install anything on them.

I couldn't check esxtop, the test clusters are busy at the moment. I'll check it as soon as they are idle and I have the opportunity to do it.

Each virtual machine has 1 cpu assigned, they don't have SMP kernels either.

The load we are experiencing is quite high, I think this is much higher than the 'basic' load. Do you think 50% load can be considered as normal?

Best regards,

Laszlo Benedek

0 Kudos
Lightbulb
Virtuoso
Virtuoso

Well vmware tools do a lot more than time sych. It may not help your issue but it easy to do and there is not much in the way of a downside.

0 Kudos
AWo
Immortal
Immortal

In general you have to be careful with the load measured within a guest, as this always includes time measurements also from within the guest. You better use tools which looks from the outside at the load of the guest (like esxtop).

VMware Tools keep care about:

- detecting idle times

- memory management

- time synchronization

- script interafce into the guest

- best fitting drivers (speed, stability)

- video driver (if you use X)

They also help other tools from the outside to report the load and memory usage correctly.


If you found this information useful, please consider awarding points for "Correct" or "Helpful" replies. Thanks!!


AWo

VCP / vEXPERT 2009

vExpert 2009/10/11 [:o]===[o:] [: ]o=o[ :] = Save forests! rent firewood! =
Texiwill
Leadership
Leadership

Hello,

Moved to Virtual Machine and Guest OS.


Best regards,
Edward L. Haletky
VMware Communities User Moderator, VMware vExpert 2009
====
Author of the book 'VMWare ESX Server in the Enterprise: Planning and Securing Virtualization Servers', Copyright 2008 Pearson Education.
Blue Gears and SearchVMware Pro Blogs -- Top Virtualization Security Links -- Virtualization Security Round Table Podcast

--
Edward L. Haletky
vExpert XIV: 2009-2022,
VMTN Community Moderator
vSphere Upgrade Saga: https://www.astroarch.com/blogs
GitHub Repo: https://github.com/Texiwill
0 Kudos