Hung VM - Running ESX4 - 3 - node cluster - Unusual performance spike...

[chadwickking]

Hello Community,

I could use helpful insight to an issue we experienced in our production enviroment on a 3-node cluster that had a VM running on it. The VM is used only for printing as we isolate applications from running together on similar servers. The technician who was working the issue reported a couple of things but I wasn't there so coudn't say what was done. The end result was he had to perform a "reset" of the VM because he couldn't connect via RDP and the console wouldn't accept crtlaltdelete. Vmware tools is up to date. He will be in later tonight so i will talk to him to get more details about the issue and what troubleshooting was done. The issue occured this morning around 1:30 to 2:30 am on the "C" host (host are 1a, 1b, 1c). Seeing that just two days ago I attained my VCP I thought I would look deeper into the problem. What I found was a bit intriging. There were no changes made in any way to the cluster recenetly other than adding some VM's. Around the time of the issue nothing was being performed that we could fine logged in vCenter. Also I connected to the console and found it EXTREMELY laggy. This is partly due to the fact that we use vCenter here in our Home Office and all the Distribution Centers talk back to the vCenter here so I am sure it is due to the WAN. Anyways, after finding that it was still unresponsive I just deciding to look at some performance counters and statistics in vCenter. What I found out was that between the time that the issue actually occured there is a rather large spike in ALL activity related to Disk, Memory, Network, and CPU. See attached images. I thought this was very interesting and anyone could lean a hand into possibly seeing what may of caused this type of spike in all host in this cluster that would be very helpful. All the vm's are provisioned with 1vcpu and only 4gb of memory. We are using HP Lefthand VSA Solutions for shared iSCSI storage. Appreciate any feedback!

[chadwickking]

Can anyone give a hand? Posted this over a week ago and could use some help.

[chadwickking]

Can anyone give a hand? Posted this over a week ago and could use some help.

[jcwuerfl]

Are there any backups going on around that time? or anybody copying large files?

[chadwickking]

Highly likely... yes.

[Phokay · Accepted solution]

Given, you have iSCSI storage attached to ESX servers, it's normal to have spikes on CPU, Disk and Network if you use normal NIC (not iSCSI HBA). When ESX access disks on iSCSI storage via normal NIC, it access the content over the network and uses CPU to process iSCSI packets. If you have iSCSI HBA, you shouldn't see CPU usage spikes like this.