Solved: Re: Exception thrown by SsoUtil: SSO admin service... - Page 2

mikemayors · ‎02-22-2016

I'm diagnosing an issue in an Enhanced Linked Mode environment running 6.0 update 1 (2 vCenters, same domain) where I'm getting the following exception when calling UserSessionService.getUserSession():

[ERROR] data-service-pool-2704 70000426 100010 200005 c.vmware.vsphere.client.usersession.impl.UserSessionServiceImpl There was an issue while extracting the list of system domains com.vmware.vise.vim.security.sso.exception.SsoServiceException: SSO admin service failure

at com.vmware.vise.vim.security.sso.SsoUtil.getAdminService(SsoUtil.java:256)

at com.vmware.vsphere.client.usersession.impl.UserSessionServiceImpl.extractSystemDomains(UserSessionServiceImpl.java:179)

at com.vmware.vsphere.client.usersession.impl.UserSessionServiceImpl.getUserSession(UserSessionServiceImpl.java:156)

at sun.reflect.GeneratedMethodAccessor496.invoke(Unknown Source)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:606)

at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:319)

at org.eclipse.gemini.blueprint.service.importer.support.internal.aop.ServiceInvoker.doInvoke(ServiceInvoker.java:56)

at org.eclipse.gemini.blueprint.service.importer.support.internal.aop.ServiceInvoker.invoke(ServiceInvoker.java:60)

at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)

at org.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:131)

at org.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:119)

at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)

at org.eclipse.gemini.blueprint.service.importer.support.LocalBundleContextAdvice.invoke(LocalBundleContextAdvice.java:57)

at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)

at org.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:131)

at org.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:119)

at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)

at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202)

at com.sun.proxy.$Proxy503.getUserSession(Unknown Source)

Does anyone have any pointers on diagnosing this? I could not find much information about this error in the SDK docs. Thank you

- Mike

vesuvius_prime · ‎06-21-2016

Build 2559277 is vanilla 6.0 release. It doesn't have the issue. The problem starts with Patch 02 which your customer has.

vesuvius_prime · ‎06-21-2016

Build 3271482 corresponds to release 6.0 Patch 02.

mikemayors · ‎06-24-2016

Hey,

I've been trying to reproduce for about a day now. I'm calling UserSessionService.getSession() in a loop every second. I also tried every half a second. Yesterday I specified no delay and just called getSession() in an infinite loop roughly 10,000 times in 10 minutes. I'll continue with this approach this afternoon but if you have other ideas I'm all ears. The vCenter I'm using now is running update 2, which I hope still has this issue because that's all my IT department has for me

(Version 6.0.0 Build 2997665)

Thanks again

vesuvius_prime · ‎06-24-2016

You mean, you are trying to reproduce the problem without having applied the patch and you can't reproduce it? In other words, the same exception as before doesn't appear in the log? That's strange, because I'm reproducing it consistently exactly with a loop (a straight loop, without limiting it to once per second or half-second). I can see the exact same error in the log. However, when I apply the patch, the error no longer appears in the log. I deployed build 3271482 (release 6.0 Patch 02) and I'm testing on it, but you should get the same result in release 6.0 Update 2 because the code is the same there.

Can you, please, describe the original problem? Were you not able to get the UserSession session at all? Or were you getting it but you were also seeing the exception in the log? I examined the code, and I can see that when the following method gets executed:

at com.vmware.vsphere.client.usersession.impl.UserSessionServiceImpl.extractSystemDomains(UserSessionServiceImpl.java:179)

(as per the stack trace that you provided), the exception is caught and in this case an empty list of System Domains is returned, but this doesn't prevent the UserSession from being returned to the caller -- its 'systemDomains' field will be empty, but the rest should be okay. However, without the patch, the method certainly does leave authenticated sessions behind and this must be fixed because eventually the SSO server reaches the maximum number of sessions and it refuses to authenticate any further, which may cause problems in completely unrelated scenarios because the SSO server gets contacted in other places too and they may fail because of the exhaustion of authenticated sessions.

So, again, can you describe the exact problem you're experiencing? What are the symptoms?

mikemayors · ‎06-24-2016

The original problem was that UserSessionService.getUserSession() throws an SsoServiceException. This prevented a user session from being returned to the caller (as you can see in the stack trace).

To reproduce I have created a Java service where UserSessionService is passed in via OSGI (same as you do in the sample apps). In the Flex client I have a toggle button which starts off an infinite loop. In the loop I call a method on my Java service which then calls UserSessionService.getUserSession(). If I get an exception I log it and throw it back to the Flex client. I've run this loop about 20,000 times and still have not hit an exception when calling UserSessionService.getUserSession(). The only difference I can see is that my vCenter is running 6.0 update 2 and our customer is running 6.0 patch 2.

Let me know if I misunderstood anything.

EDIT: Correction - calling UserSessionService.getUserSession()

Mike

vesuvius_prime · ‎06-24-2016

Okay, I'm looking at the code of method UserSessionService.getUserSession() and it basically does this:

try {
   ...
} catch (Exception e) {
   _logger.error("Failed to get the user session", e);
   return null;
}

Furthermore, method UserSessionServiceImpl.extractSystemDomains(), which is part of the stack trace you quoted earlier, does:

try {
   ...
} catch (Exception e) {
   _logger.error("There was an issue while extracting the list of system domains", e);
}

Neither of these can throw an exception that can propagate to the caller. (They can throw an Error or a plain Throwable, but we're not dealing with such here.) What you see in the log is just a message that there was an error, but in reality you will either get a null UserSession, or a UserSession in which field 'systemDomains' is empty.

By the way, I reproduced the problem by directly placing a loop around the try-catch block in method extractSystemDomains() -- the block which I quoted above. I ran a loop of 1,000 iterations but the actual error started to appear after much fewer iterations. I didn't bother to record how many, but maybe after 200-300 iterations. Don't quote me on that, though -- I might be wrong.

I'm quite surprised you can't reproduce the problem by doing remote invocations of UserSessionService.getUserSession(). And I'm even more surprised that you manage to get SsoServiceException thrown out of method getUserSession(). In fact, it's a checked exception and the method doesn't have any throws clause, so this makes the matter even more puzzling.

mikemayors · ‎06-24-2016

Aha! So it looks like we call getUserSession() which causes an exception to be thrown, is caught by UserSessionServiceImpl which then returns null. I just checked the logs and I'm seeing the exception. Gonna try your patch now.

[2016-06-24T16:00:45.107-04:00] [ERROR] http-bio-9443-exec-5 c.vmware.vsphere.client.usersession.impl.UserSessionServiceImpl There was an issue while extracting the list of system domains com.vmware.vise.vim.security.sso.exception.SsoServiceException: SSO admin service failure

at com.vmware.vise.vim.security.sso.SsoUtil.getAdminService(SsoUtil.java:256)

at com.vmware.vsphere.client.usersession.impl.UserSessionServiceImpl.extractSystemDomains(UserSessionServiceImpl.java:176)

at com.vmware.vsphere.client.usersession.impl.UserSessionServiceImpl.getUserSession(UserSessionServiceImpl.java:153)

at sun.reflect.GeneratedMethodAccessor495.invoke(Unknown Source)