ESX performance decrease after applying patches

[daniel345]

This Saturday we applied patches to all of our ESX servers using update manager with following vmware-tools upgrade on all hosts. After that, we see a tremendous increase in CPU usage. A few heavily used MSSql servers are using 100% CPU constantly and users complain about time-outs.

We are using HP BL 25p with a MSA SAN.

Since all ESX-servers are upgraded we cannot pinpoint vmware by migrating.

/Daniel

[bradley4681]

First thing is I'd open a case with support since it's inpacting production systems, then i'd look around the forums.

that being said, is the host cpu peg at 100% or just the guest cpus? were the guest vmware tools upgraded as well? what process on the guests are using all the cpu resources?

Cheers,

Bradley Sessions

If you found this or other information useful, please consider awarding points for "Correct" or "Helpful".

[daniel345]

A case is open but the answer delays. I think we pinpointed the problem to slow SAN communication. So disc intensive applications like databases were affected when they were pressured. When we downgraded one ESX to 3.5 (pre update1) the complete farm seems fine even under pressure.

Can't wait what the case answer will tell...

Cheers

[bradley4681]

Was there any HBA driver updates in the release notes?

Cheers,

Bradley Sessions

If you found this or other information useful, please consider awarding points for "Correct" or "Helpful".

[dominic7]

Did you use DRS/VMotion to migrate the VMs around when you upgraded?

Have you applied Update 1 to VirtualCenter 2.5?

There is a bug ( pre-update 1 in VC ) where a DRS enabled cluster can migrate a VM, and that VM will spin to 100% CPU usage and stay there.... forever. The only way to get the CPU usage is to power off the VM, and then power it back on.

Edit: That's not the only way to get the CPU usage back down, you could also upgrade to VC 2.5 Update 1 and VMotion your VMs around again.

[bradley4681]

good call!

Cheers,

Bradley Sessions

If you found this or other information useful, please consider awarding points for "Correct" or "Helpful".

[dominic7]

After reading some other threads, it looks like this problem was re-introduced with the patches that came out after ESX 3.5.0 U1. So, open a SR with VMware and yell at them.

[daniel345]

Yes, I used VMotion to move the guests. And no, VC update 1 was not applied. DRS is enabled. Looks like you are on to something....

[daniel345]

When we disabled DRS, right after downgrading one host, all farm seems to be ok again. Still waits for WMvare for the report from VMware.

/Daniel

[daniel345]

VC was not upgraded to update 1 and this caused the problem. When upgrading to ESX update 1. VC also must be upgraded!

/Daniel