Performence issue, and high CPU Ready

[Qwerty256]

Hello,

On one (the most used) VM i have performence issue and very big load average, but there is nothing worng about the cpu from ESX side, just one thing i spot is READY% is pretty big, and the problems usually are when READY is close to USAGE, but the machine has a lot left CPU, so could anybody tell me what can be wrong with my configuration ?

[Sreec]

Hi,

    Overprovisioning of vCPUs will cause high CPU ready time.Please follow the steps in KB:http://kb.vmware.com/kb/1005362and do let me know the result.

[Qwerty256]

%CSTP
  0.00
  1.37
  1.33
  0.39
  0.09
  0.40
  1.38
  0.38
  0.00
  0.00

No issue

[Sreec]

Hi,

    When you captured this O/P did you had performance issue?You have to check the esxtop exactly when the issue is happening.

[Qwerty256]

Yes i didnt have big issue then, the performence is slower anyway, but anyway will check when the issu with server occures, thanks

[rickardnobel]

Could you also check %MLMTD while having high %RDY?

[Qwerty256]

%MLMTD  is 0 all the time, i will do printscreen when will have issue with performence

[Qwerty256]

extop when issue

[Qwerty256]

Also this is problem and i don't know why,

I have plenty of free resources, there is no machine which have reservation, and i cannot give 20 000 Mhz to any VM,

10 000mhz is possible. IMHO should be possible at least 150 000 Mhz ... Am i doing anything wrong ?

[rickardnobel]

Could you post a screenshot of the m (memory) screen? Use "V" to clear all other instances than the VMs.

Could you also post another screenshot of the memory, but with this customized:

f - to select fields,

remove: B, K, L and O.

add: G = NUMA STATS

Make sure the N%L is visible and then post the screenshot.

[Qwerty256]

In attachment

[Sreec]

Hi ,

      Looking at the attached esxtop screen i can see rdy% value has exceeded the threshold(Threshold value is 10).Please follow steps in KB::http://kb.vmware.com/kb/1005362

[rickardnobel]

From what I see most of your VMs are 8 vCPU and 32 GB vRAM, which is of course quite large. However, your NUMA nodes are 10 cores and 64 GB so they should fit good, and the only VM with somewhat less good NUMA locality is the CentOS 5.5-LB, but it should not really explain why all VMs experience high RDY time.

[Qwerty256]

http://kb.vmware.com/kb/1005362
On the CPU screen, check the %CSTP  value. If this number is higher than 100, the performance issues may be  caused by the vCPU count. Try lowering the vCPU count of the virtual  machine by 1.

it's not higher than 100

[Qwerty256]

I found in that thread:

http://communities.vmware.com/thread/391284

I had the same problem. A temporary fix is to set "Numa.RebalanceEnable" to 0

esxi-host configuration, Software, Advanced Settings, Numa, Numa.RebalanceEnable = 0

Supermicro Opteron 61xx and 62xx hosts.

When I did this the cpu load balanced evenly between the cores and the ready times dropped to 0.XX as was expected.

did the same, result in screenshot, pretty better isnt it ?

[rickardnobel · Accepted solution]

Qwerty256 wrote:

did the same, result in screenshot, pretty better isnt it ?

So it was indeed the NUMA that caused it. Nice that you got your RDY back to normal!