Exception thrown by SsoUtil: SSO admin service failure

[mikemayors]

I'm diagnosing an issue in an Enhanced Linked Mode environment running 6.0 update 1 (2 vCenters, same domain) where I'm getting the following exception when calling UserSessionService.getUserSession():

[ERROR] data-service-pool-2704       70000426 100010 200005 c.vmware.vsphere.client.usersession.impl.UserSessionServiceImpl   There was an issue while extracting the list of system domains com.vmware.vise.vim.security.sso.exception.SsoServiceException: SSO admin service failure

  at com.vmware.vise.vim.security.sso.SsoUtil.getAdminService(SsoUtil.java:256)

  at com.vmware.vsphere.client.usersession.impl.UserSessionServiceImpl.extractSystemDomains(UserSessionServiceImpl.java:179)

  at com.vmware.vsphere.client.usersession.impl.UserSessionServiceImpl.getUserSession(UserSessionServiceImpl.java:156)

  at sun.reflect.GeneratedMethodAccessor496.invoke(Unknown Source)

  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

  at java.lang.reflect.Method.invoke(Method.java:606)

  at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:319)

  at org.eclipse.gemini.blueprint.service.importer.support.internal.aop.ServiceInvoker.doInvoke(ServiceInvoker.java:56)

  at org.eclipse.gemini.blueprint.service.importer.support.internal.aop.ServiceInvoker.invoke(ServiceInvoker.java:60)

  at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)

  at org.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:131)

  at org.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:119)

  at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)

  at org.eclipse.gemini.blueprint.service.importer.support.LocalBundleContextAdvice.invoke(LocalBundleContextAdvice.java:57)

  at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)

  at org.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:131)

  at org.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:119)

  at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)

  at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202)

  at com.sun.proxy.$Proxy503.getUserSession(Unknown Source)

Does anyone have any pointers on diagnosing this?  I could not find much information about this error in the SDK docs.  Thank you

- Mike

[mikemayors]

*bump*

Has nobody ever seen this in enhanced linked mode? 

[ryantodd]

Seeing this in a couple of 6.0 test systems *not* in linked mode. The issue we have is also intermittent. I.e. you cannot log in for some time, then all of the sudden, login works, then log out, and try to log in again, only for it to fail again. No idea yet what the actual cause is though...

[ryantodd]

Btw, this seems to be an issue for us only in VCSA 6.0 U1B. VCSA 6.0 U1 seems to be fine in this regard.

[virat1234]

I am seeing this on the vcenter appliance 6u1. No idea how to fix it though.

[ryantodd]

After a few days of attempting to work around this issue by making multiple changes to our code, attempting to absolutely minimize any calls to UserSessionService.getUserSession, we've pretty much exhausted everything we can try. At this point I believe that all we can do is recommend to any users of our product plugin to not upgrade to 6.0 U1B.

Can any vmware folks weigh in on this issue, any recommendations?

[ryantodd]

Virat1234 -- are you seeing the reference to extractSystemDomains in your stacktrace, or something else entirely?

[virat1234]

Virat1234 -- are you seeing the reference to extractSystemDomains in your stacktrace, or something else entirely?

That is exactly what I am seeing

[EJIE]

Finally, could you solve this issue?

Thanks

[mikemayors]

Has anyone had any luck in resolving this issue?  Or perhaps any pointers on where to look? I checked that both forward and reverse DNS is configured for all vCenters and the PSC and also checked that all system clocks are synced.  

[_vladi_]

This problem is currently under investigation by the vSphere Web Client team.

A similar thread is: "SSO admin service failure" exception in vSphere 6.0.2

Cheers,

Vladi

[vesuvius_prime]

Can you, please, post the full stack trace, including the 'Caused By' sections? I wish to see if it's the same issue as in https://communities.vmware.com/message/2602308.

[vesuvius_prime]

Would you like us to create a fixed class file for release 6.0 Update 1 just like we did for release 6.0 Update 2 at "SSO admin service failure" exception in vSphere 6.0.2?

[mikemayors]

[2016-02-17T08:45:41.702-05:00] [ERROR] data-service-pool-283711     70033948 100252 200143 c.vmware.vsphere.client.usersession.impl.UserSessionServiceImpl   There was an issue while extracting the list of system domains com.vmware.vise.vim.security.sso.exception.SsoServiceException: SSO admin service failure

  at com.vmware.vise.vim.security.sso.SsoUtil.getAdminService(SsoUtil.java:256)

  at com.vmware.vsphere.client.usersession.impl.UserSessionServiceImpl.extractSystemDomains(UserSessionServiceImpl.java:179)

  at com.vmware.vsphere.client.usersession.impl.UserSessionServiceImpl.getUserSession(UserSessionServiceImpl.java:156)

  at sun.reflect.GeneratedMethodAccessor688.invoke(Unknown Source)

  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

  at java.lang.reflect.Method.invoke(Method.java:606)

  at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:319)

  at org.eclipse.gemini.blueprint.service.importer.support.internal.aop.ServiceInvoker.doInvoke(ServiceInvoker.java:56)

  at org.eclipse.gemini.blueprint.service.importer.support.internal.aop.ServiceInvoker.invoke(ServiceInvoker.java:60)

  at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)

  at org.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:131)

  at org.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:119)

  at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)

  at org.eclipse.gemini.blueprint.service.importer.support.LocalBundleContextAdvice.invoke(LocalBundleContextAdvice.java:57)

  at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)

  at org.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:131)

  at org.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:119)

  at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)

  at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202)

  at com.sun.proxy.$Proxy587.getUserSession(Unknown Source)

  [ sensitive packages removed ]

  at sun.reflect.GeneratedMethodAccessor1042.invoke(Unknown Source)

  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

  at java.lang.reflect.Method.invoke(Method.java:606)

  at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:319)

  at org.eclipse.gemini.blueprint.service.importer.support.internal.aop.ServiceInvoker.doInvoke(ServiceInvoker.java:56)

  at org.eclipse.gemini.blueprint.service.importer.support.internal.aop.ServiceInvoker.invoke(ServiceInvoker.java:60)

  at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)

  at org.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:131)

  at org.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:119)

  at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)

  at org.eclipse.gemini.blueprint.service.importer.support.LocalBundleContextAdvice.invoke(LocalBundleContextAdvice.java:57)

  at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)

  at org.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:131)

  at org.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:119)

  at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)

  at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202)

  at com.sun.proxy.$Proxy661.getInfo(Unknown Source)

  [ sensitive packages removed (my implementation of property provider adapter) ]

  at com.vmware.vise.data.query.impl.DataManager.getDataFromPropertyProvider(DataManager.java:1312)

  at com.vmware.vise.data.query.impl.DataManager.getResultFromPropertyProvider(DataManager.java:1283)

  at com.vmware.vise.data.query.impl.DataManager.access$000(DataManager.java:67)

  at com.vmware.vise.data.query.impl.DataManager$1.run(DataManager.java:858)

  at com.vmware.vise.util.concurrent.ExecutorUtil$2.run(ExecutorUtil.java:187)

  at com.vmware.vise.util.concurrent.ExecutorUtil$ThreadContextPropagatingRunnable.run(ExecutorUtil.java:584)

  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)

  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

  at java.lang.Thread.run(Thread.java:745)

Caused by: java.util.concurrent.ExecutionException: com.vmware.vim.binding.vmodl.RuntimeFault: Unable to dispatch request: Failed to create session

  at java.util.concurrent.FutureTask.report(FutureTask.java:122)

  at java.util.concurrent.FutureTask.get(FutureTask.java:188)

  at com.vmware.vise.util.concurrent.client.ClientMonitorImpl.authenticate(ClientMonitorImpl.java:89)

  at com.vmware.vise.vim.security.sso.impl.SsoAdminServiceImpl.getClientMonitor(SsoAdminServiceImpl.java:134)

  at com.vmware.vise.vim.security.sso.impl.SsoAdminServiceImpl.<init>(SsoAdminServiceImpl.java:110)

  at com.vmware.vise.vim.security.sso.SsoUtil.getAdminService(SsoUtil.java:252)

  ... 68 common frames omitted

Caused by: com.vmware.vim.binding.vmodl.RuntimeFault: Unable to dispatch request: Failed to create session

  at sun.reflect.GeneratedConstructorAccessor461.newInstance(Unknown Source)

  at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)

  at java.lang.reflect.Constructor.newInstance(Constructor.java:526)

  at java.lang.Class.newInstance(Class.java:383)

  at com.vmware.vim.vmomi.core.types.impl.ComplexTypeImpl.newInstance(ComplexTypeImpl.java:173)

  at com.vmware.vim.vmomi.core.types.impl.DefaultDataObjectFactory.newDataObject(DefaultDataObjectFactory.java:26)

  at com.vmware.vim.vmomi.core.soap.impl.unmarshaller.ComplexStackContext.<init>(ComplexStackContext.java:31)

  at com.vmware.vim.vmomi.core.soap.impl.unmarshaller.UnmarshallerImpl$UnmarshallSoapFaultContext.parse(UnmarshallerImpl.java:141)

  at com.vmware.vim.vmomi.core.soap.impl.unmarshaller.UnmarshallerImpl$UnmarshallSoapFaultContext.unmarshall(UnmarshallerImpl.java:102)

  at com.vmware.vim.vmomi.core.soap.impl.unmarshaller.UnmarshallerImpl.unmarshalSoapFault(UnmarshallerImpl.java:89)

  at com.vmware.vim.vmomi.core.soap.impl.unmarshaller.UnmarshallerImpl.unmarshalSoapFault(UnmarshallerImpl.java:84)

  at com.vmware.vim.vmomi.client.common.impl.SoapFaultStackContext.setValue(SoapFaultStackContext.java:41)

  at com.vmware.vim.vmomi.client.common.impl.ResponseUnmarshaller.unmarshal(ResponseUnmarshaller.java:112)

  at com.vmware.vim.vmomi.client.common.impl.ResponseImpl.unmarshalResponse(ResponseImpl.java:273)

  at com.vmware.vim.vmomi.client.common.impl.ResponseImpl.setResponse(ResponseImpl.java:230)

  at com.vmware.vim.vmomi.client.http.impl.HttpExchangeBase.parseResponse(HttpExchangeBase.java:144)

  at com.vmware.vim.vmomi.client.http.impl.HttpExchange.run(HttpExchange.java:51)

  at com.vmware.vim.vmomi.client.http.impl.HttpProtocolBindingBase.executeRunnable(HttpProtocolBindingBase.java:186)

  at com.vmware.vim.vmomi.client.http.impl.HttpProtocolBindingImpl.send(HttpProtocolBindingImpl.java:115)

  at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl$CallExecutor.sendCall(MethodInvocationHandlerImpl.java:581)

  at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl$CallExecutor.executeCall(MethodInvocationHandlerImpl.java:562)

  at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl.completeCall(MethodInvocationHandlerImpl.java:348)

  at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl.invokeOperation(MethodInvocationHandlerImpl.java:308)

  at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl.invoke(MethodInvocationHandlerImpl.java:182)

  at com.sun.proxy.$Proxy602.retrieveServiceContent(Unknown Source)

  at com.vmware.vise.vim.security.sso.impl.SsoUtilInternal.getSsoAdminServiceContent(SsoUtilInternal.java:253)

  at com.vmware.vise.vim.security.sso.impl.SsoAdminServiceImpl.processLogin(SsoAdminServiceImpl.java:143)

  at com.vmware.vise.vim.security.sso.impl.SsoAdminServiceImpl.access$300(SsoAdminServiceImpl.java:57)

  at com.vmware.vise.vim.security.sso.impl.SsoAdminServiceImpl$SolutionUserAuthenticator.authenticate(SsoAdminServiceImpl.java:497)

  at com.vmware.vise.vim.security.sso.impl.SsoAdminServiceImpl$SolutionUserAuthenticator.authenticate(SsoAdminServiceImpl.java:481)

  at com.vmware.vise.util.concurrent.client.ClientMonitorImpl$1.call(ClientMonitorImpl.java:209)

  at com.vmware.vise.util.concurrent.client.ClientMonitorImpl$1.call(ClientMonitorImpl.java:206)

  at java.util.concurrent.FutureTask.run(FutureTask.java:262)

  at com.vmware.vise.util.concurrent.client.ClientMonitorImpl.authenticate(ClientMonitorImpl.java:74)

  ... 71 common frames omitted

As you can see, the "Caused by" is : Caused by: com.vmware.vim.binding.vmodl.RuntimeFault: Unable to dispatch request: Failed to create session

vesuvius_prime you said in the other thread 'perhaps the users create too many sessions but don't log out of them explicitly afterwards. And it takes a while for the sessions to expire by themselves. So, probably sometimes you (or your customer) hit the limit and you get the error.'

Do you know if it's possible for an extension to flood the PSC with too many sessions? I thought an extension can only get the current active session from the VIM API, not create new sessions.  Just trying to rule out if this is our extension, the customer, or a VMware issue.

Thanks again for your responsivness, you've been a big help.

[vesuvius_prime]

Hmmm, I checked the source. There's a problem with method UserSessionServiceImpl.extractSystemDomains() which is invoked by method UserSessionServiceImpl.getUserSession() and is in the stack trace that you posted. In short, an authentication to the SSO Admin Service is performed but after the work is done, there's no invocation of logout(), so the authenticated session remains. These sessions will eventually expire, but if people do lots of invocations of UserSessionServiceImpl.getUserSession() before the previous authenticated sessions to the SSO Admin Service can expire, then these session will pile up and will exceed the limit and then the errors will start to occur.

I can prepare yet another hack for you. I can fix method UserSessionServiceImpl.extractSystemDomains() and I can invoke logout() explicitly and this will likely solve your problem, but I don't know if you (or your customer) will be okay with applying such patches. Also, I need to say the following:

1. Before you put this fix in production, please, try it out on some non-production environment, just in case. I will test it too, but I don't have an extensive set of plug-ins with which to test, so my testing will be somewhat limited. Furthermore, this is a hack, not an official patch, so I cannot get our QAs deeply involved in all this. You have a real-life environment which can be invaluable in testing this. For example, you can let your plug-in invoke UserSessionService.getUserSession() several hundred times and you can see if you'll hit the problem.

2. Please, either you or your customer should log an SR which will probably force an early hot patch for this issue. I'm not in control of releases, especially patches for past releases, but as I was already told, releasing a hot patch requires an SR from a customer. Of course, I suppose that an SR doesn't automatically mean that a hot patch will be released. The SR will be evaluated, different approaches/fixes/workarounds will be discussed and a decision will be made. But the SR is an important first step.

[mikemayors]

Nice analysis.  Few questions:

1. What is "work" when you say  "an authentication to the SSO Admin Service is performed but after the work is done, there's no invocation of logout()" - do you mean logout() is supposed to be called after the extension is finished with the session that is returned to it?

2. Will an authenticated session to the admin service show up when you browse active vCenter sessions? Or is this something internal? ("These sessions will eventually expire...)

3. Do you know the expiration time when a session is created with the admin service?

4. Would calling UserSessionService.getUserSession() in a loop be a good attempt at reproducing this?

I'll be more than glad to test this in house and let you know how it goes.

[vesuvius_prime]

Answers:

1. What is "work" when you say  "an authentication to the SSO Admin Service is performed but after the work is done, there's no invocation of logout()" - do you mean logout() is supposed to be called after the extension is finished with the session that is returned to it?

I mean that method UserSessionServiceImpl.extractSystemDomains() should invoke logout() when it's done. This should be transparent to you.

By the way, the code in the upcoming release 6.5 is quite different and this is not an issue there. The actual problem appeared in release 6.0 Patch 2 and is also present in Update 1 and Update 2.

2. Will an authenticated session to the admin service show up when you browse active vCenter sessions? Or is this something internal? ("These sessions will eventually expire...)

No. These are not vCenter sessions, but sessions to the SSO Admin Service.

3. Do you know the expiration time when a session is created with the admin service?

I guess it's 30 minutes. But I sent an email to some of the SSO guys asking them to tell me what the expiration timeout is.

4. Would calling UserSessionService.getUserSession() in a loop be a good attempt at reproducing this?

Yes, it should be good. Please, first try doing the loop without a patch. If it ends in error, then great -- we have reproducibility. Then, after applying the patch, try the loop again. If it works without errors, great again -- the problem is fixed. I will post a patch very soon.

[vesuvius_prime]

Okay, attached the patch. I assumed that your vSphere Web Client build number is 3271482 -- at least that's what we determined in Re: "SSO admin service failure" exception in vSphere 6.0.2.

[mikemayors]

Thanks for the quick turnaround. The vCenter we're using for development is running build 2559277 (Version 6.0.0 Build 2559277).  Can you make a patch for this build or does this build not contain the change that introduced this issue? 

Let me know. If necessary I'll upgrade to Update 1 (or Update 2, whichever corresponds to 3271482) and try flooding the vCenter with getSession() calls.

[mikemayors]

Actually, let me update our vCenter to what the customer is running.  Do you know which update 3271482 corresponds to? (i.e. 1B or 1A)  I'll just upgrade to that. Probably best I have close to the same environment as possible.