vCenter 4.1 slow at navigating as a VM???

[erikjespersen]

My vCenter server has been slow for some time now. Even after a clean install of Windows Server 2008R2 and change the vCenter Server settings for database retention to 15 days. It still is running slow when navigating through. We have 18 ESX hosts and 558 VM’s.

The vCenter Specs are 4x 2.66GHz, 4GB of RAM, 40GB HDD. We are using 3.36GB of RAM. Maybe more RAM is needed? vCenter is on the same ESX hosts its managing.

The Database server is also a VM on the same Cluster as vCenter. Database server specs are 4x 2.66GHz, 4GB of RAM, 50GB OS HDD, 80GB Database, and 40GB Log HDD. Running SQL Server 2008 on Windows Server 2008R2. We are using almost all of the RAM but I think that’s just because of SQL takes anything it can get. Does not mean it’s really using it.

Please let me know what you think?

Thank You

Erik

[Walfordr]

erikjespersen wrote:

My vCenter server has been slow for some time now. Even after a clean install of Windows Server 2008R2 and change the vCenter Server settings for database retention to 15 days. It still is running slow when navigating through. We have 18 ESX hosts and 558 VM’s.

Erik,

I think I am facing the same issue but with a different setup.

My setup:

virtual vCenter.

VM specs: Server 2008 R2, SQL 2008 R2, 4GB of RAM, 2 processors.

Host is a DL380 G7 with 40GB. For now I only have the vCenter on this host. Have individual LUN for server.

3, datacenters. Total of 18host. 200+ VM.

Upgraded from virtual center 2.5 (I believe u6) to vCenter 4.1 u1. Used converted DB from the datamigration utility.

*I still have some 3.5 hosts and pointing to the old vCenter for licensing of those hosts.

Symptoms:

Logon takes 15-20 seconds (that is OK by my standards).

After logon, navigating the objects are slow.  Specifically noticed on Host and Cluster and Datastore view.  The objects has this hesitation where the object faded (with powered off icon for VM) and then takes a few seconds to display the proper icon.

I opened a case and the tech told me its normall.  First he was saying that If I upgraded from virtual center earlier to u6 I would run into DB issues, but could not prove which version I upgraded from. I also had a high amount of idle session and the tech told me its a bug and I have to enable the client side idle time out, under client settings...I took what he said with a grain of salt.

CPU usage on the host is not high. Memory is high due to the database and the way SQL takes all it can.

My client side is win7 x64.  Yes, I have applied the compatability view settings from the KB and it didn't help. (http://kb.vmware.com/kb/1027836).  I have the issue on the local vCenter server and XP aswell.

Hopefully we'll get some suggestions.

[nathanw]

That is not usual to be this slow, the VM itself is quite large.

How is the performance of the DBs erver in general?

do you have the vMA installed?

try running resxtop from the vMA and pointing it toward your Host running the VC and perhaps the DB server, look for  a high %RDY count, this would indicate CPU contention.

you could remove 2 vCPU's to reduce the time the VM has to wait for CPU's to be available if the %RDY count is high.

adding extra unnecessary vCPU's can hinder rather than enhance you VM's performance.

[erikjespersen]

Thanks you Nathanw, I’ve looked before at the ready time for both vm’s they have always been low. I took screen shots of them below if you want to look. Maybe I’m missing something?

Also the 4x vCPU is a lot but there are times that both vm’s need and use them. When our teams deploy a lot of VM’s at once the vCenter server will be hit hard during that time. I’m talking about like 30 –to- 120 vm’s at once. I’ve tried everything to have them load lance them but they do not do it. L

I also have set the vCenter and Database VM’s to high shares and still does not look like it has improved.

When I mean slow at navigating I mean it takes about on average 10 –to- 20 seconds when I click on the VM to bring up the summary page for example. I have a vCenter server with less ESX hosts and VM’s it load much faster, like within 1 –to- 2 seconds. Is this just a limitation of vCenter?

Thank You

Erik

[DrJohnEaton]

I will, thanks.

[Kahonu84]

Aloha -

Curious - did this start happening when you went to VCenter 4.1?? I ask because when I was at 3.5 U4, VCenter client was very snappy. After I upgraded to 4.1 U1 - I'm seeing weird lags like buttons not immediately appearing, things grayed out when client first starts up, flickering and general sluggishness. Both 3.5 and 4.1 VCenter's were/are physical boxes with the 4.1 box being much beefier. VCenter works but this type of behavior is a bit troubling.

Bill

[erikjespersen]

We have had this issue when we upgraded from vCenter 2.5 Update5 to vCenter 4.0. vCenter 2.5 was also a physical server so maybe that helped with performance? With vSphere 4.0 we went to a VM for vCenter Server. We asked multiple people at VMware if there are any issues with doing this. They said no it is much better and they recommended it.

So we moved vCenter to a VM. We started to see performance issues with vCenter about 3 months after we upgraded. We did a lot of troubleshooting with VMware on this and everyone came to the same conclusion let start over fresh and see if that will fix it…. Lol :smileylaugh:… New database server, new database, new vCenter server with a new installation. Everything new and started over from scratch. Thank GOD we did not have distributed virtual switch installed yet!?!

After about again 3 months we started to see performance issues with vCenter. The only thing we installed on the VM with vCenter is NETAPP system manager and storage console for vSphere.

I’m really starting to think that maybe it’s not a good idea to install vCenter as a VM? What happens if we are/get much bigger?

Thank You

Erik

[bilalhashmi]

Have you looked at the storage stats to see if the issue is there? Keep in mind the DB will be bound by several jobs at diff times to make sure vCenter has all that it needs... it could become a high IO DB... take a look at that... I tend to leave my vCenter as a VM and my DB on a pre-existing SQL cluster thats usually physical..

Follow me @ Cloud-Buddy.com

[Walfordr]

I defraged the performance indexes on my database and it seem to have helped a little.  I still notice the lag navigating on the the tree under Host and Clusters.  The tabs for me refreshes very quick (has always been), much faster than when I was on virtual center 2.5.

I used the steps from here after hours and did not have to restart any of the services: http://virtualserverguy.com/2010/10/vcenter-database-optimization-thoughts/

Here are the official steps from VMware: Defragmenting VirtualCenter performance data indexes on a Microsoft SQL database

Given that you rebuilt from scratch all your performance data should be new.

[erikjespersen]

Thank You bilalhashmi, I have looked at the database server and the IOPS is low from what i can see. We are running it on NETAPP with NFS. so i can’t get much performance statistics from vCenter, but when i run esxtop as you can see below its really not that high IOPS. I wish I can get historical data L

From what I can see at our highest peak of the day we hit 37IOPS writes. On average we are about 10IOPS writes. This to me this does not seem high?

Maybe defrag the database would help? I'm not a DBA so this is really out of my area of expertise.

Thank You

Erik

[Cyberfed27]

I am seeing the same issues, we just recently added our 4th and 5th host online into our cluster.

I have noticed big lag in navigating the menu objects within vCenter.

I'm thinking it may be because we used the bundled SQL Express install for vCenter.

I read a VMware PDF that says 5 physical hosts is the limit for SQL express.

The server running vCenter and SQL Express is a power house box, dual quad core CPU's with 32GB of RAM. So its not hurting for horsepower.

Just curoius if anyone had found a reasonable fix that can be done within vCenter or perhaps in the database. We are thinking about migrating the vCenter DB to our high end SQL cluster to see if that helps.

Thoughts?

[erikjespersen]

We have recently upgraded to vSphere 5.0 and have seen an improvement in speed on vCenter as a VM. I don’t know how long this will last as I have seen this before moving from physical vCenter 4.0 to a VM. Just wanted to give everyone an update. Maybe vSphere 5.0 performs better as a VM now?

Currently we have 2 vCenter Server in link mode with 4 Clusters and 76 ESX Hosts. With a total of 3,460 VM’s. We have grown significantly in the last year.

Thank You

Erik Jespersen