VCO Performance Issue?

[adamflaig1]

We have upgraded a VCO standalone server to version 5.5.  Expanding the datacenter objects and host folder objects when using the vCenter 5.5 plugin can take up to 10 minutes.

Upgrade Process

1. exported packages and configuration

2.  stopped services and uninstalled the application

3.  Took a backup of the MS-SQL DB and and stood up a new database in its place

4.  installed the new application version 5.5

5.  imported the configuration and packages into the application.

Has anyone seen this behavior? or better yet have a fix for it?  5.5 seems to be significantly slower than the 5.1  version we were running before.

System Specs  Running Virtually....

Win 2008 R2

2 CPU i7

12 GB RAM

Java Heap size has been increased to ...

maxHttpHeaderSize="163840"

[iiliev]

Are there any errors/exceptions in vCO server log that look like vCO <-> VC communication issues?

[adamflaig1]

No communication issues logged in

/installdirectory/app-server/logs/*

Its not timing out just taking a very long time to return the list of objects.

I am in a very large environment.  I am used to a performance issue with the plugin BUT seems to have gotten worse with the newer plugin version.

I am not using SSO? Using LDAP through AD as my authentication method.  I do not have any issues logging into the application.  That seemed to improve with the new version.

[QLD]

Last week I installed the 5.5 appliance, 2008 SQL DB / LDAP.  Browsing our vcenter 4.1 environments are painfully slow even enumerating a blue folder with only four sub folders takes about 20 seconds.  If I then expand a folder with 50 vm's of more, it can time out and the folder will not display the pointer next to it removing any option to try and expand it and I need to restart the vco client - seems to time out.  Once this has happened - no folders expand and the client needs a restart.

Whats interesting is we have a new vcenter 5.1 environment which browses a little laggy but doesn't experience the same time out issue.

Server log:.

2013-11-22 08:31:47.249+1000 [http-bio-172.16.251.157-8281-exec-2] ERROR {} [VimVmFolder] getPropertyContent() [aa_vco_svc@https://vcenterurl:443/sdk/Main#6f374dd, Folder<group-v871>] --> Unable to find property 'childEntity' before the UpdateThread timeout '20000' millis

2013-11-22 08:31:47.255+1000 [http-bio-172.16.251.157-8281-exec-2] ERROR {} [VimVmFolder] getVm() --> Timeout, unable to get property 'childEntity'

com.vmware.vmo.plugin.vi4.model.TimeoutException

Really need some help here.  The only additional thing is I have installed the Active Directory plugin 1.0.3 which is currently disabled and configured a Powershell Host.

Pulling at straws here.

[mcfadyenj]

Hi Adam,

We are experiencing similar in 5.1, can you post details of your java heap size. We identified a similar issue with IIS within our environment has proven to solve the issue. This fix you have applied looks remarkably similar and might be what I am looking for to solve our issue.

regards,

John

[adamflaig1]

Here is the VMware KB to adjusting the java Heap size in VCO.  I hope this helps your issue.

http://kb.vmware.com/kb/2053223

[mcfadyenj]

thank you very much.

[stevenbright]

I experienced the same issue after upgrading to the 5.5 vCO appliance. After spending time on the line with the VMware Support, they notified that this is a known issue with the current release of vCO. They didn't say whether or not it was specific to the appliance or the Windows version. In our case, we had to downgrade back to 5.1 as I couldn't even complete a simple search for a VM to execute a workflow on. Unfortunately I don't have any official documentation that references this. I would suggest that you contact VMware Support to obtain further information regarding vCenter Inventory performance with the vCO 5.5. appliance.

[adamflaig1]

I have a case opened with Business Critical Support.  Today I duplicated the the issue over WebEx, compared the normal behavior on another VCO server running version 5.1.  Support is taking the details and logs and sending them over to the product engineers.  I will keep this thread up to date on the results of those conversations.

In the mean time as a work around I have created some custom actions and configuration elements to get around this issue.  If a solution cannot be found through support or their internal engineers I will attempt to post a generic version of that work around to this thread.

[mcfadyenj]

I applied this patch and no different on our side.

Just for reference sake here is an interesting scenario on our site you may want to check. We narrowed our slowness down to Active Directory.

As it turns out having a large number of user groups causes performance to nose dive significantly. We only tested this due to the fact we have the same issue on other applications.

Created a new user with no groups other than the orchestrator admin group and boom performance was normal, this of course doesn't fix the problem it just worked around it. might be worth a crack on your site as a test. 

[adamflaig1]

Awesome thank you for the input.  I will mention it to support when they are done with their anaylsis.

[QLD]

FYI:  Appliance 5.5.  AFter noticing the problem, I went back and tested this 'out of the box' with a default config using postgres or sql and the latency was present.  (To rule out the additional plugins I'm using).

When browsing vCenter 4.1 subfolders, (Host, VM, Datastore, Network) when opening the fourth folder, it will 'always' hang and both the client and vco requires a restart.

It doesn't matter in which order I try to expand them either or on which of our three 4.1 vcenters.  Connecting to vcenter 5.1 seems to be fine but then that has 90% less objects.

I'm pretty sure my latency / hanging is not caused by AD groups as vCenter 5.1 behaves much better within the same AD Domain as our vCenter 4.1. (but worth noting - we do have 5700 groups !!)

Hoping for a miracle cure.

[QLD]

I would like to try the javaheap fix but the location for jbossweb-tomcat55.sar doesn't exist in 5.5 appliance?  Would anyone know where I could locate this file?

If you deployed the Orchestrator Appliance, navigate to /opt/vmo/app-server/server/vmo/deploy/jboss-deploy-tomcat/jbossweb-tomcat55.sar/

• Open the file in a text editor and replace all occurrences of
  HttpHeaderSize="16384"
  with
  maxHttpHeaderSize="163840"

[richardjamesloc]

I'm also having major performance issues with the vCO 5.5 appliance.  Parameter validation is hanging in the Orchestrator Client for long periods and it is generally slow to open workflows when running them.

It looks like the appliance is now running a later version of Tomcat, but I've not found the equivalent configuration file yet.  Is there any other way to resolve performance issues with vCO 5.5?

Thanks, Richard

[igaydajiev · Accepted solution]

There is a new technical preview version of vCenter plugin available here.  Technical Preview Version of VMware vCenter Orchestrator Plug-In for VMware vSphere 5.5.1

This version resolves some performance issues related to vCenter plugin inventory and starting workflows aceptinag as parameter vCenter objects..

Hope it helps.

[richardjamesloc]

Thanks for your help.  I've upgraded to that preview version and the performance is much better when running a workflow and providing input parameters.

Kind regards, Richard

[adamflaig1]

This is the slution the support engineers came back with.  After uninstalling the previous plugin and installing the new one the

UNINSTALL Instructions
http://kb.vmware.com/selfservice/microsites/search.do?cmd=displayKC&docType=kc&externalId=2064575

The browsing times went from

15 minutes to drill down to the host objects

to

seconds to drill down to the host objects.

Thank you eveyone for all your input

[QLD]

Yep - much better - thanks all

[mcfadyenj]

which plugin was the cause?

[richardjamesloc]

Hi mcfadyenj,

It was the vCenter plugin.  Using the latest preview version brought the performance back to the same as with the vCO 5.1 appliance.

Richard