ESX 3.5 Update 4 Guest/Host Performance Issues

[razablayde]

We recently upgraded to Update 4 and now about a week later I've noticed some drastic performance issues in our VMs. VMs (Win2k3 -64bit, 2K8-64bit, XP Pro) that used to take about a minute to boot now take 5 minutes. Operation inside of these machines is extremely slow as well; simply logging in takes another few minutes and every action performed takes a while. What's puzzling is that there are no visible errors within the logs (both VC and Windows) and the performance counters all look fine. Also, if I "Export" a VM from my ESX infrastructure to a ESXi "free" host that I use for testing, the VM performs fine.

We are currently running ESX3.5 w/ VirtualCenter (separate physical machine) on 3 - Quad Core 3GHz hosts with 24 GB Ram each connected to a FIbre Channel SAN.

Has anyone else experinced any degradation after upgrading to Update 4? Please help.

[shalshar]

Did you upgraded your Virtual Center to 2.5 Update4 and updated VMware tools on VM's?

Notes:

1. Not all combinations of VirtualCenter and ESX Server versions are supported and not all of these highlighted features are available unless you are using VirtualCenter 2.5 Update 4 with ESX Server 3.5 Update 4. See the ESX Server, VirtualCenter, and VMware Infrastructure Client Compatibility Matrixes for more information on compatibility.

2. This version of ESX Server requires a VMware Tools upgrade.

[Zahni]

Try to set Mem.ShareScanGHz = 0 at advandec config and reboot ESX:

Reference

[TiJa · Accepted solution]

I had similar issues in my lab setup. Turned out that the shares that were granted to the virtual machines and resource pools were completely messed up. As a temporary workaround, I simply removed all resource pools by disabling DRS on my test cluster.

Another issue I experienced after upgrading to U4 is that the virtual machine networks randomly started showing a very high latency from time to time (for example: from a physical workstation I ping a virtual machine that is on the same gigabit switch -- latencies vary between 10 ms and 2000 ms !!!). I have been unable to pinpoint the exact circumstances under which this happens, but it certainly has a devastating effect on the startup times of my machines. I think that messing around with the service console memory & CPU reservations + increasing the "system resources" somewhat solved the issue but I am not 100% sure that was the root cause (this was a common solution for VM network latencies in previous versions of ESX). In the meantime, I also upgraded my VMware Tools to the U4 version (which has a new vmxnet driver) and I did not notice the issue again since. Again, I am not sure that this is the root cause and solved the problem given its random occurence.

[razablayde]

Thank you very much for that suggestion, that fixed it for me.

I set the DRS from Fully Automated to Partially Automated and that solved it. Is Fully Automated a resource hog amongst the hosts?

[TiJa]

Thank you very much for that suggestion, that fixed it for me.

I set the DRS from Fully Automated to Partially Automated and that solved it. Is Fully Automated a resource hog amongst the hosts?

Not really, not more than partially automated (from a DRS point of view -- of course a possible vMotion that is executed as a DRS suggestion has some impact). I also don't fully understand what went wrong, if problems reoccur on my test machine, I'll be able to do some more research on it. I did notice that attempting to reenable DRS seriously congests the ESX host (very slow response in VI Client). Again, some more research is needed to see if it is really the hostd process going bananas :).

[razablayde]

Well it seems that the sluggish behavior within vCenter resumed. I changed DRS to Manual and have seen an improvement but it is still somewhat laggy compared to the ESXi install I have. Maybe the reponse time of vCenter will get included in the next round of updates, or we'll just have to switch to vSphere

[Cody_Page]

TiJa

We are having the exact same issues as you described. We upgraded our cluster during production hours as we've done in the past and ran into major problems on our guest VMs that didnt yet have VMware tools updated. Nearly all our VM's had to be rebooted for any kind of response.

Now we're left with VMs that have sluggish performance and high latency. Our hosts and w2k3 VMs are not showing any errors out of the ordinary. We've opened a ticket with VMware all we could come up with is 'Sluggish Performance of VMs after U4'. vCenter and Hosts we're previously U2.

I will report back if we hear anything from support. My .02: If you are looking into updating to U4, I would highly suggest you wait before upgrading.

[razablayde]

If you haven't already tried the DRS suggestions, I would. It dramatically increased performance, even with VMs without the latest Tools build.

[Cody_Page]

Disabling DRS seemed to have fixed our issue. Also, we currently dont have any resource pools setup.

[TiJa]

Can you please post more details about your setup to see if we're running similar systems?

I am running a cleanly installed ESX 3.5 U4 host in an HA enabled "cluster" (of that single host) which is managed by the vCenter 4.0 beta (which in my case might also be a potential issue since I am pretty sure it is not a very supported configuration ). Can you specify which vCenter version you are using? This might help to find similarities in our config and perhaps point other people with similar issues in the good direction.