ESX Service Console - CPU Wait above 95 %

[Tidian]

Hello,

we are running 10 ESX 4.1 U1 server. All ESX servers are hosting nearly same count of VMs (15 - 20) and the load is nearly equal.

One of these ESX server have a heavy loaded CPU in service console. The CPU WAIT is above 95 %.

TOP shows:

Tasks: 120 total,   2 running, 118 sleeping,   0 stopped,   0 zombie
Cpu(s):  1.0%us,  0.0%sy,  0.0%ni,  0.0%id, 98.7%wa,  0.0%hi,  0.3%si,  0.0%st
Mem:    602016k total,   507484k used,    94532k free,    86724k buffers
Swap:  1228932k total,      316k used,  1228616k free,   250092k cached

Unfortunately, I didn't find a reason, why the load is so heavy. I have checked vmstat -d, but disk are used normally.

Any ideas or hints?

Thank you in advance

Tino

[idle-jam]

check this out: http://communities.vmware.com/servlet/JiveServlet/downloadBody/14905-102-2-17952/vsphere41-performan...

[Tidian]

I've checked out the document, but I couldn't find information about troubleshooting performance issues inside the service console. This behavior occurs only in the service console. The hosted VMs are running fine.

[Tidian]

Hello,

I've checked performance data (disk latency times) from involved ESX host. I've found a very high value at read kernel latency.

Please see attached picture.

Curious is, that disk named "naa.60a98000572d576f464a642d46414962) does not exist on ESX-Server. I've rescanned HBA, but no change.

Any ideas?

Tino