Host CPU usage - core "0" is allways near 100%

[Najtsob]

Hi,

we have single virtual center and two clusters. On all servers in both clusters I noticed something that bothers me. On every single server if you check CPU usage you can se that core "0" usage is near 100%.

I check on my whitebox server and some older servers that we have for test/dev and this is not happening.

When I move all the VMs off the server and reboot it, the CPU usage of the core "0" stays near 100% although server is empty and under Summary tab in vsphere client the reported CPU usage in Mhz is around 2000 Mhz. Again on my whitbox server and other test/dev servers after reboot and with no running VMs the CPU usage in Mhz iz something around 400Mhz.

This is my server under load:

And this is server after reboot and with no running VMs:

Any ideas why is this happening ? Is this normal/expected (it's not in my opinion) ?

And most importantly, how to troubleshot it ?

Thank you for help,

best regards

[Najtsob]

Esxi on the servers is 5.5U1, vCenter is windows Server based and also 5.5U1. Servers in the first cluster are Cisco B200 m3 and in second cluster are B230 m2.

I don't have information if there was this issue before upgrade, it is more than a month since we upgraded and I didn't work on this infrastructure before.

[vervoortjurgen]

did you use vmware esxi image or vendor custom image

i would think firmware/driver problem

i would upgrade the servers to latest firmware and use the vendor image of cisco

[Najtsob]

I tried with following images:

- vmware esxi 5.5 u1 image

- vmware esxi 5.5 u1 driver rollup 2 image

- cisco custom esxi 5.5 u1 image

It's same behavior with all of them.

Servers and everything is also upgraded to the latest firmware.

Is there a way to see exactly which process/driver is using all this CPU ?

[vervoortjurgen]

cli command esxtop should show something whats running if your esx host is empty

[Najtsob]

I posted screenshot of esxtop and vsphere client from when the host as empty in my original post.

You can see there that core "0" has average load 86% and the rest are idling. But I still can't see which process is causing this. I was watching esxtop for few minutes and nothing significant showed there. In %USED column nothing went past 10%.

Did you look at those screenshots and do you see anything odd ? Maybe I just don't know the right way how to look on this results and how to interpret them

[lvaibhavt]

can you check the monthly graph for the cpu core 0 to verify when did the issue start?

then check if any changes were made on that date

[Najtsob]

The graph which shows usage per core is not available for month back.

Besides that environment is pretty much static, even upgrades are quite rare.

[vervoortjurgen]

yes i saw the screen shot

i couldnt see anything weird either

any pluging still running on host? VASA or other storage plugins?

[lvaibhavt]

check Re: Host CPU usage - core "0" is allways near 100%

also can you do a clean install of esxi 5.5 U2 --- dont do a upgrade just a clean install

[Najtsob]

We don't have any plugins instlled.

Regarding esxtop screen shot. If you sum up values under %USED column you come to about 10%, so there must be something that esxtop doesn't show or there is something wrong with output ? Or am I just misinterpreting it ?

[Najtsob]

Will try that and let you know how it goes.

I will just ask once again,is this really an issue so that I don't just chase the ghosts

Well it bothers me, and I don't think that CPU usage should be that high on empty host. But still maybe just maybe (probably not) is this normal/expected CPU usage and I'm overreacting.

[lvaibhavt]

this is an issue >>> if I was at your place I would check to make sure why your cpu core 0 at at 100 percentage

let me know how it goes ?

Somehow the link that I mentioned earlier was pointing to this thread only

correct link is High host CPU usage after upgrade to ESXi 5,5

[Najtsob]

Update 2 resolves the issue, either we perform update or clean install.

So i'm guessing that there must be some driver incompatibility with update 1, which is strange since even Cisco custom image didn't work properly....

Also I'm still wondering why esxtop didn't show runaway process, is there any other way to check this ?