VMware Cloud Community
CloudAutoCraft
Enthusiast
Enthusiast
Jump to solution

vRA 6.2.3 environment mysteriously stopped working

Hi all,

Distributed (HA) vRA 6.2.3 environment just stopped working.  It's been running solid for a couple years; I was logged into it just yesterday, no issues.  Initial symptom was when I attempted to login today and got the awesome error message: "Login failed. Please contact your System Administrator and report error code Q2n6XUV7"    (BTW -- if anyone can tell me the relevance of that error code and where I can find it in vRA logs, that would be awesome.  Back on point...)

I noticed Authentication Failures in catalina.out on the Cafe VAs, so I then started looking through logs on the Identity Appliance.  Lots of authentication failures there too.  I shutdown the Cafe VAs and bounced the Identity Appliance.  Once the Identity Appliance VAMI showed that SSO was initialized, I started the Cafe VAs again and watched the services light up in the VAMI.  The eventlog-service was the first one to fail.  The next time I refreshed, authentication, authorization, and identity had also failed.  At the end of the startup cycle, this is what my services looked like:

Long story short, it's still not working.  Below are some log entries that I believe are pertinent -- I have red-highlighted entries that I believe are important clues to this mystery.  If anyone can offer guidance on what they mean or how to troubleshoot/resolve, PLEASE HELP!!

cafe-services.png

Identity Appliance:  /var/log/vmware/sso/catalina.out:

This log has been suspiciously quiet for the last six months.  There's no timestamp on the error highlighted below, but the next timestamp is from this afternoon (Oct 6).  I'm guessing that error occurred today, but it's not clear if this is a symptom or cause of the issue.

Feb 12, 2016 11:13:24 AM org.apache.catalina.startup.Catalina start

INFO: Server startup in 21510 ms

Apr 12, 2016 1:55:31 PM org.apache.catalina.startup.Catalina load

INFO: Initialization processed in 3669 ms

Apr 12, 2016 1:55:52 PM org.apache.catalina.startup.Catalina start

INFO: Server startup in 20533 ms

Apr 12, 2016 3:25:38 PM org.apache.catalina.startup.Catalina load

INFO: Initialization processed in 4159 ms

Apr 12, 2016 3:26:02 PM org.apache.catalina.startup.Catalina start

INFO: Server startup in 23382 ms

Exception in thread "Thread-4" java.util.ConcurrentModificationException

        at java.util.HashMap$HashIterator.nextEntry(HashMap.java:922)

        at java.util.HashMap$ValueIterator.next(HashMap.java:950)

        at java.util.Collections$UnmodifiableCollection$1.next(Collections.java:1067)

        at com.vmware.identity.session.SessionCleanupWrapper.run(SessionCleanupWrapper.java:52)

        at java.lang.Thread.run(Thread.java:745)

Oct 06, 2016 2:02:12 PM org.apache.catalina.startup.Catalina load

INFO: Initialization processed in 3490 ms

Oct 06, 2016 2:02:34 PM org.apache.catalina.startup.Catalina start

INFO: Server startup in 21409 ms

Oct 06, 2016 3:40:29 PM org.apache.catalina.startup.Catalina load

INFO: Initialization processed in 2513 ms

Oct 06, 2016 3:40:53 PM org.apache.catalina.startup.Catalina load

INFO: Initialization processed in 2591 ms

Oct 06, 2016 3:41:13 PM org.apache.catalina.startup.Catalina start

INFO: Server startup in 19472 ms

Identity Appliance:  /var/log/vmware/sso/vmware-identity-sts.log:  Failed to authenticate by BST??!?  WTF is BST?!?

[2016-10-06 19:19:04,141Z tomcat-http--1 vsphere.local        845db8e1-e193-4feb-af50-3ebf8768f782 TRACE com.vmware.identity.sts.ws.handlers.SOAPHeadersExtractor] Encountering node {http://www.w3.org/2000/09/xmldsig#}Signature searching for {http://www.w3.org/2000/09/xmldsig#}Signature

[2016-10-06 19:19:04,141Z tomcat-http--1 vsphere.local        845db8e1-e193-4feb-af50-3ebf8768f782 DEBUG com.vmware.identity.sts.ws.handlers.SOAPHeadersExtractor] Searching for Action in http://www.w3.org/2005/08/addressing namespace.

[2016-10-06 19:19:04,142Z tomcat-http--1 vsphere.local        845db8e1-e193-4feb-af50-3ebf8768f782 DEBUG com.vmware.identity.sts.ws.handlers.XMLSignatureValidator] Inside XMLSignatureValidator

[2016-10-06 19:19:04,142Z tomcat-http--1 vsphere.local        845db8e1-e193-4feb-af50-3ebf8768f782 INFO  com.vmware.identity.sts.ws.handlers.XMLSignatureValidator] Found signature _d9f0ffd0-9d2c-4987-968b-db29da0ea7e7

[2016-10-06 19:19:04,142Z tomcat-http--1 vsphere.local        845db8e1-e193-4feb-af50-3ebf8768f782 DEBUG com.vmware.identity.sts.ws.SignatureValidator] Found KeyInfo

[2016-10-06 19:19:04,142Z tomcat-http--1 vsphere.local        845db8e1-e193-4feb-af50-3ebf8768f782 DEBUG com.vmware.identity.sts.ws.SignatureValidator] Found SecurityTokenReference

[2016-10-06 19:19:04,142Z tomcat-http--1 vsphere.local        845db8e1-e193-4feb-af50-3ebf8768f782 DEBUG com.vmware.identity.sts.ws.SignatureValidator] Found reference with URI: #_68cfd0c6-dead-4bbc-ab7e-448d12095a3d

[2016-10-06 19:19:04,142Z tomcat-http--1 vsphere.local        845db8e1-e193-4feb-af50-3ebf8768f782 INFO  com.vmware.identity.sts.ws.SignatureValidator] Got signing certificate

[2016-10-06 19:19:04,144Z tomcat-http--1 vsphere.local        845db8e1-e193-4feb-af50-3ebf8768f782 INFO  com.vmware.identity.sts.ws.handlers.XMLSignatureValidator] Signature _d9f0ffd0-9d2c-4987-968b-db29da0ea7e7 is valid

[2016-10-06 19:19:04,146Z tomcat-http--1 vsphere.local        845db8e1-e193-4feb-af50-3ebf8768f782 DEBUG com.vmware.identity.sts.ws.StsServiceImpl] Start handling 'issue' request

[2016-10-06 19:19:04,146Z tomcat-http--1 vsphere.local        845db8e1-e193-4feb-af50-3ebf8768f782 DEBUG com.vmware.identity.sts.ws.StsServiceImpl] Extracted tenantName from request: vsphere.local

[2016-10-06 19:19:04,146Z tomcat-http--1 vsphere.local        845db8e1-e193-4feb-af50-3ebf8768f782 DEBUG com.vmware.identity.sts.ws.StsServiceImpl] Extracted saml token:[(NULL)]

[2016-10-06 19:19:04,146Z tomcat-http--1 vsphere.local        845db8e1-e193-4feb-af50-3ebf8768f782 TRACE com.vmware.identity.sts.ws.StsServiceImpl] Found JAXB ActAs entity - null

[2016-10-06 19:19:04,146Z tomcat-http--1 vsphere.local        845db8e1-e193-4feb-af50-3ebf8768f782 DEBUG com.vmware.identity.sts.impl.STSFactoryImpl] Get STS for tenant: vsphere.local

[2016-10-06 19:19:04,146Z tomcat-http--1 vsphere.local        845db8e1-e193-4feb-af50-3ebf8768f782 TRACE com.vmware.identity.sts.impl.STSImpl] issue() token...

[2016-10-06 19:19:04,146Z tomcat-http--1 vsphere.local        845db8e1-e193-4feb-af50-3ebf8768f782 DEBUG com.vmware.identity.sts.impl.STSImpl] Validation of request

[2016-10-06 19:19:04,146Z tomcat-http--1 vsphere.local        845db8e1-e193-4feb-af50-3ebf8768f782 DEBUG com.vmware.identity.sts.impl.STSImpl] The request received is valid from 2016-10-06T19:19:03.916Z till 2016-10-06T19:29:03.916Z and now is: Thu Oct 06 14:19:04 CDT 2016

[2016-10-06 19:19:04,149Z tomcat-http--1 vsphere.local        845db8e1-e193-4feb-af50-3ebf8768f782 TRACE com.vmware.identity.perf] 'idm.getClockTolerance' took 3 ms.

[2016-10-06 19:19:04,149Z tomcat-http--1 vsphere.local        845db8e1-e193-4feb-af50-3ebf8768f782 TRACE com.vmware.identity.sts.impl.LifetimeConvertor] Gets date object from request for datetime: 2016-10-06T19:19:03.916Z

[2016-10-06 19:19:04,149Z tomcat-http--1 vsphere.local        845db8e1-e193-4feb-af50-3ebf8768f782 TRACE com.vmware.identity.sts.impl.LifetimeConvertor] Gets date object from request for datetime: 2016-10-06T19:29:03.916Z

[2016-10-06 19:19:04,149Z tomcat-http--1 vsphere.local        845db8e1-e193-4feb-af50-3ebf8768f782 TRACE com.vmware.identity.sts.auth.impl.UNTAuthenticator] Authenticating by UNT...

[2016-10-06 19:19:04,149Z tomcat-http--1 vsphere.local        845db8e1-e193-4feb-af50-3ebf8768f782 DEBUG com.vmware.identity.sts.auth.impl.UNTAuthenticator] No UNT found!

[2016-10-06 19:19:04,149Z tomcat-http--1 vsphere.local        845db8e1-e193-4feb-af50-3ebf8768f782 TRACE com.vmware.identity.sts.auth.impl.BSTAuthenticator] Authenticating by BST...

[2016-10-06 19:19:04,149Z tomcat-http--1 vsphere.local        845db8e1-e193-4feb-af50-3ebf8768f782 TRACE com.vmware.identity.sts.auth.impl.BSTAuthenticator] Looking up an user by subjectDN CN=cafe-8bd63676-fab1-4e6f-b882-6abab5299982 ...

[2016-10-06 19:19:04,154Z tomcat-http--1 vsphere.local        845db8e1-e193-4feb-af50-3ebf8768f782 TRACE com.vmware.identity.perf] 'idm.findSolutionUserByCertDn' took 5 ms.

[2016-10-06 19:19:04,154Z tomcat-http--1 vsphere.local        845db8e1-e193-4feb-af50-3ebf8768f782 DEBUG com.vmware.identity.sts.InvalidCredentialsException] About to censor authentication failure

com.vmware.identity.sts.InvalidCredentialsException: Solution user's certificate does not match the one in BST!

        at com.vmware.identity.sts.auth.impl.BSTAuthenticator.checkMatchingCertificate(BSTAuthenticator.java:147)

        at com.vmware.identity.sts.auth.impl.BSTAuthenticator.doAuthenticate(BSTAuthenticator.java:104)

        at com.vmware.identity.sts.auth.impl.BSTAuthenticator.authenticate(BSTAuthenticator.java:71)

        at com.vmware.identity.sts.auth.impl.CompositeAuthenticator.authenticate(CompositeAuthenticator.java:44)

        at com.vmware.identity.sts.auth.impl.CompositeAuthenticatorPerformanceDecorator$1.call(CompositeAuthenticatorPerformanceDecorator.java:54)

        at com.vmware.identity.sts.auth.impl.CompositeAuthenticatorPerformanceDecorator$1.call(CompositeAuthenticatorPerformanceDecorator.java:51)

        at com.vmware.identity.performanceSupport.PerformanceDecorator.exec(PerformanceDecorator.java:36)

        at com.vmware.identity.sts.auth.impl.CompositeAuthenticatorPerformanceDecorator.authenticate(CompositeAuthenticatorPerformanceDecorator.java:51)

        at com.vmware.identity.sts.impl.STSImpl.issue(STSImpl.java:127)

        at com.vmware.identity.sts.impl.MultiTenantSTSImpl.issue(MultiTenantSTSImpl.java:50)

        at com.vmware.identity.sts.impl.MultiTenantSTSImplPerformanceDecorator$2.call(MultiTenantSTSImplPerformanceDecorator.java:89)

        at com.vmware.identity.sts.impl.MultiTenantSTSImplPerformanceDecorator$2.call(MultiTenantSTSImplPerformanceDecorator.java:86)

        at com.vmware.identity.performanceSupport.PerformanceDecorator.exec(PerformanceDecorator.java:36)

        at com.vmware.identity.sts.impl.MultiTenantSTSImplPerformanceDecorator.issue(MultiTenantSTSImplPerformanceDecorator.java:86)

        at com.vmware.identity.sts.ws.StsServiceImpl.issue(StsServiceImpl.java:159)

        at sun.reflect.GeneratedMethodAccessor52.invoke(Unknown Source)

        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

        at java.lang.reflect.Method.invoke(Method.java:606)

        at com.sun.xml.ws.api.server.InstanceResolver$1.invoke(InstanceResolver.java:250)

        at com.sun.xml.ws.server.InvokerTube$2.invoke(InvokerTube.java:150)

        at com.sun.xml.ws.server.sei.EndpointMethodHandler.invoke(EndpointMethodHandler.java:261)

        at com.sun.xml.ws.server.sei.SEIInvokerTube.processRequest(SEIInvokerTube.java:100)

        at com.sun.xml.ws.api.pipe.Fiber.__doRun(Fiber.java:641)

        at com.sun.xml.ws.api.pipe.Fiber._doRun(Fiber.java:600)

        at com.sun.xml.ws.api.pipe.Fiber.doRun(Fiber.java:585)

        at com.sun.xml.ws.api.pipe.Fiber.runSync(Fiber.java:482)

        at com.sun.xml.ws.server.WSEndpointImpl$2.process(WSEndpointImpl.java:314)

        at com.sun.xml.ws.transport.http.HttpAdapter$HttpToolkit.handle(HttpAdapter.java:608)

        at com.sun.xml.ws.transport.http.HttpAdapter.handle(HttpAdapter.java:259)

        at com.sun.xml.ws.transport.http.servlet.ServletAdapter.invokeAsync(ServletAdapter.java:213)

        at com.sun.xml.ws.transport.http.servlet.WSServletDelegate.doGet(WSServletDelegate.java:159)

        at com.sun.xml.ws.transport.http.servlet.WSServletDelegate.doPost(WSServletDelegate.java:194)

        at com.sun.xml.ws.transport.http.servlet.WSServlet.doPost(WSServlet.java:80)

        at javax.servlet.http.HttpServlet.service(HttpServlet.java:646)

        at javax.servlet.http.HttpServlet.service(HttpServlet.java:727)

        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303)

        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)

        at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)

        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)

        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)

        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:220)

        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:122)

        at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:504)

        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:170)

        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)

        at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:950)

        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)

        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:421)

        at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1074)

        at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:611)

        at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:314)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

        at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)

        at java.lang.Thread.run(Thread.java:745)

[2016-10-06 19:19:04,154Z tomcat-http--1 vsphere.local        845db8e1-e193-4feb-af50-3ebf8768f782 DEBUG com.vmware.identity.sts.ws.StsServiceImpl] com.vmware.identity.sts.InvalidCredentialsException: Invalid credentials

        at com.vmware.identity.sts.InvalidCredentialsException.buildPublic(InvalidCredentialsException.java:45)

        at com.vmware.identity.sts.ws.StsServiceImpl.issue(StsServiceImpl.java:163)

        at sun.reflect.GeneratedMethodAccessor52.invoke(Unknown Source)

        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

        at java.lang.reflect.Method.invoke(Method.java:606)

        at com.sun.xml.ws.api.server.InstanceResolver$1.invoke(InstanceResolver.java:250)

        at com.sun.xml.ws.server.InvokerTube$2.invoke(InvokerTube.java:150)

        at com.sun.xml.ws.server.sei.EndpointMethodHandler.invoke(EndpointMethodHandler.java:261)

        at com.sun.xml.ws.server.sei.SEIInvokerTube.processRequest(SEIInvokerTube.java:100)

        at com.sun.xml.ws.api.pipe.Fiber.__doRun(Fiber.java:641)

        at com.sun.xml.ws.api.pipe.Fiber._doRun(Fiber.java:600)

        at com.sun.xml.ws.api.pipe.Fiber.doRun(Fiber.java:585)

        at com.sun.xml.ws.api.pipe.Fiber.runSync(Fiber.java:482)

        at com.sun.xml.ws.server.WSEndpointImpl$2.process(WSEndpointImpl.java:314)

        at com.sun.xml.ws.transport.http.HttpAdapter$HttpToolkit.handle(HttpAdapter.java:608)

        at com.sun.xml.ws.transport.http.HttpAdapter.handle(HttpAdapter.java:259)

        at com.sun.xml.ws.transport.http.servlet.ServletAdapter.invokeAsync(ServletAdapter.java:213)

        at com.sun.xml.ws.transport.http.servlet.WSServletDelegate.doGet(WSServletDelegate.java:159)

        at com.sun.xml.ws.transport.http.servlet.WSServletDelegate.doPost(WSServletDelegate.java:194)

        at com.sun.xml.ws.transport.http.servlet.WSServlet.doPost(WSServlet.java:80)

        at javax.servlet.http.HttpServlet.service(HttpServlet.java:646)

        at javax.servlet.http.HttpServlet.service(HttpServlet.java:727)

        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303)

        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)

        at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)

        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)

        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)

        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:220)

        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:122)

        at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:504)

        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:170)

        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)

        at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:950)

        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)

        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:421)

        at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1074)

        at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:611)

        at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:314)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

        at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)

        at java.lang.Thread.run(Thread.java:745)

[2016-10-06 19:19:04,154Z tomcat-http--1 vsphere.local        845db8e1-e193-4feb-af50-3ebf8768f782 DEBUG com.vmware.identity.sts.ws.SOAPFaultHandler] Ws Fault:

com.vmware.identity.sts.ws.WSFaultException: com.vmware.identity.sts.InvalidCredentialsException: Invalid credentials

        at com.vmware.identity.sts.ws.StsServiceImpl.throwFault(StsServiceImpl.java:440)

        at com.vmware.identity.sts.ws.StsServiceImpl.issue(StsServiceImpl.java:163)

        at sun.reflect.GeneratedMethodAccessor52.invoke(Unknown Source)

        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

        at java.lang.reflect.Method.invoke(Method.java:606)

        at com.sun.xml.ws.api.server.InstanceResolver$1.invoke(InstanceResolver.java:250)

        at com.sun.xml.ws.server.InvokerTube$2.invoke(InvokerTube.java:150)

        at com.sun.xml.ws.server.sei.EndpointMethodHandler.invoke(EndpointMethodHandler.java:261)

        at com.sun.xml.ws.server.sei.SEIInvokerTube.processRequest(SEIInvokerTube.java:100)

        at com.sun.xml.ws.api.pipe.Fiber.__doRun(Fiber.java:641)

        at com.sun.xml.ws.api.pipe.Fiber._doRun(Fiber.java:600)

        at com.sun.xml.ws.api.pipe.Fiber.doRun(Fiber.java:585)

        at com.sun.xml.ws.api.pipe.Fiber.runSync(Fiber.java:482)

        at com.sun.xml.ws.server.WSEndpointImpl$2.process(WSEndpointImpl.java:314)

        at com.sun.xml.ws.transport.http.HttpAdapter$HttpToolkit.handle(HttpAdapter.java:608)

        at com.sun.xml.ws.transport.http.HttpAdapter.handle(HttpAdapter.java:259)

        at com.sun.xml.ws.transport.http.servlet.ServletAdapter.invokeAsync(ServletAdapter.java:213)

        at com.sun.xml.ws.transport.http.servlet.WSServletDelegate.doGet(WSServletDelegate.java:159)

        at com.sun.xml.ws.transport.http.servlet.WSServletDelegate.doPost(WSServletDelegate.java:194)

        at com.sun.xml.ws.transport.http.servlet.WSServlet.doPost(WSServlet.java:80)

        at javax.servlet.http.HttpServlet.service(HttpServlet.java:646)

        at javax.servlet.http.HttpServlet.service(HttpServlet.java:727)

        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303)

        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)

        at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)

        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)

        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)

        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:220)

        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:122)

        at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:504)

        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:170)

        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)

        at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:950)

        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)

        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:421)

        at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1074)

        at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:611)

        at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:314)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

        at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)

        at java.lang.Thread.run(Thread.java:745)

Caused by: com.vmware.identity.sts.InvalidCredentialsException: Invalid credentials

        at com.vmware.identity.sts.InvalidCredentialsException.buildPublic(InvalidCredentialsException.java:45)

        ... 41 more

[2016-10-06 19:19:04,155Z tomcat-http--1 vsphere.local        845db8e1-e193-4feb-af50-3ebf8768f782 INFO  com.vmware.identity.sts.ws.SOAPFaultHandler] Returning a SOAP Fault with code: ns0:FailedAuthentication and description: Invalid credentials

[2016-10-06 19:19:07,077Z tomcat-http--19                                                           DEBUG com.vmware.identity.sts.ws.handlers.LogContextHandler] unable to extract correlation id from request. generated new correllation id [cfde93bc-f894-43ac-9f68-ac379c5eda3d]

[2016-10-06 19:19:07,078Z tomcat-http--19                                                           DEBUG com.vmware.identity.sts.ws.handlers.LogContextHandler] extracted tenant [vsphere.local] from the request

[2016-10-06 19:19:07,078Z tomcat-http--19                                                           DEBUG com.vmware.identity.sts.ws.handlers.LogContextHandler] extracted tenant name [vsphere.local] from the request

These BST authentication events are happening roughly 20 times per minute.

<identityVA>:/var/log/vmware/sso # grep 'Authenticating by BST' vmware-identity-sts.log

[2016-10-06 19:19:04,149Z tomcat-http--1 vsphere.local        845db8e1-e193-4feb-af50-3ebf8768f782 TRACE com.vmware.identity.sts.auth.impl.BSTAuthenticator] Authenticating by BST...

[2016-10-06 19:19:07,086Z tomcat-http--19 vsphere.local        cfde93bc-f894-43ac-9f68-ac379c5eda3d TRACE com.vmware.identity.sts.auth.impl.BSTAuthenticator] Authenticating by BST...

[2016-10-06 19:19:09,915Z tomcat-http--5 vsphere.local        61e4f1d8-9ce4-41c5-9ec1-aa9a0ef5b4c7 TRACE com.vmware.identity.sts.auth.impl.BSTAuthenticator] Authenticating by BST...

[2016-10-06 19:19:13,418Z tomcat-http--23 vsphere.local        e934216b-31b2-470f-80bd-067a2313003a TRACE com.vmware.identity.sts.auth.impl.BSTAuthenticator] Authenticating by BST...

[2016-10-06 19:19:16,318Z tomcat-http--16 vsphere.local        43a2b4e2-5d1b-4f0c-907b-9eb42be17a54 TRACE com.vmware.identity.sts.auth.impl.BSTAuthenticator] Authenticating by BST...

[2016-10-06 19:19:17,023Z tomcat-http--18 vsphere.local        227c2a84-1442-4519-a7dd-cbfda66b1c45 TRACE com.vmware.identity.sts.auth.impl.BSTAuthenticator] Authenticating by BST...

[2016-10-06 19:19:24,232Z tomcat-http--11 vsphere.local        21136f00-a064-48fd-9917-a03954c49b1e TRACE com.vmware.identity.sts.auth.impl.BSTAuthenticator] Authenticating by BST...

[2016-10-06 19:19:30,000Z tomcat-http--10 vsphere.local        f7c050f8-0740-45a9-a2b3-bbaba0904f6e TRACE com.vmware.identity.sts.auth.impl.BSTAuthenticator] Authenticating by BST...

[2016-10-06 19:19:33,484Z tomcat-http--36 vsphere.local        388087db-8176-41da-b260-56a6a9d89fb5 TRACE com.vmware.identity.sts.auth.impl.BSTAuthenticator] Authenticating by BST...

[2016-10-06 19:19:36,380Z tomcat-http--49 vsphere.local        ae8e2b2b-4886-405f-8731-214460d8ec93 TRACE com.vmware.identity.sts.auth.impl.BSTAuthenticator] Authenticating by BST...

[2016-10-06 19:19:37,101Z tomcat-http--38 vsphere.local        9083d604-f727-4d0e-8720-c4075f078d7b TRACE com.vmware.identity.sts.auth.impl.BSTAuthenticator] Authenticating by BST...

[2016-10-06 19:19:37,344Z tomcat-http--14 vsphere.local        84c84833-89cc-4496-9106-0063cd9a5970 TRACE com.vmware.identity.sts.auth.impl.BSTAuthenticator] Authenticating by BST...

[2016-10-06 19:19:44,314Z tomcat-http--31 vsphere.local        d59e3202-d276-42aa-b6f8-35f7afd5c485 TRACE com.vmware.identity.sts.auth.impl.BSTAuthenticator] Authenticating by BST...

[2016-10-06 19:19:50,071Z tomcat-http--12 vsphere.local        6bec09e4-9e6e-4d3a-a993-7961f0004808 TRACE com.vmware.identity.sts.auth.impl.BSTAuthenticator] Authenticating by BST...

[2016-10-06 19:19:53,549Z tomcat-http--48 vsphere.local        a442d8ea-0e18-410b-92fb-ee91dc0dd058 TRACE com.vmware.identity.sts.auth.impl.BSTAuthenticator] Authenticating by BST...

[2016-10-06 19:19:56,442Z tomcat-http--33 vsphere.local        aec686bd-0c73-4337-a102-81a379498fb4 TRACE com.vmware.identity.sts.auth.impl.BSTAuthenticator] Authenticating by BST...

[2016-10-06 19:19:57,168Z tomcat-http--4 vsphere.local        25794f2e-d6e7-4476-a26e-265b16bcdd91 TRACE com.vmware.identity.sts.auth.impl.BSTAuthenticator] Authenticating by BST...

[2016-10-06 19:20:00,067Z tomcat-http--28 vsphere.local        73192347-960e-4e8a-847d-e95c127521b4 TRACE com.vmware.identity.sts.auth.impl.BSTAuthenticator] Authenticating by BST...

[2016-10-06 19:20:04,388Z tomcat-http--45 vsphere.local        1050ec90-49da-482a-90db-5177a69d934c TRACE com.vmware.identity.sts.auth.impl.BSTAuthenticator] Authenticating by BST...

[2016-10-06 19:20:07,833Z tomcat-http--15 vsphere.local        62d11ed2-46cf-4d76-bae7-7ab5b209fb7e TRACE com.vmware.identity.sts.auth.impl.BSTAuthenticator] Authenticating by BST...

[2016-10-06 19:20:10,154Z tomcat-http--29 vsphere.local        32327146-ddd2-44b7-88df-4e6ba053fc3d TRACE com.vmware.identity.sts.auth.impl.BSTAuthenticator] Authenticating by BST...

[2016-10-06 19:20:13,609Z tomcat-http--20 vsphere.local        104c631d-b6e6-4576-b1a9-792a93f39224 TRACE com.vmware.identity.sts.auth.impl.BSTAuthenticator] Authenticating by BST...

[2016-10-06 19:20:16,506Z tomcat-http--50 vsphere.local        51aeb270-0266-4e55-9319-5d04fc9be3dd TRACE com.vmware.identity.sts.auth.impl.BSTAuthenticator] Authenticating by BST...

[2016-10-06 19:20:17,250Z tomcat-http--35 vsphere.local        710bde46-b45d-4e05-b14f-089ba3dc6944 TRACE com.vmware.identity.sts.auth.impl.BSTAuthenticator] Authenticating by BST...

[2016-10-06 19:20:24,465Z tomcat-http--44 vsphere.local        21a6668d-c9f0-49df-8c56-04f49ecd0ad0 TRACE com.vmware.identity.sts.auth.impl.BSTAuthenticator] Authenticating by BST...

[2016-10-06 19:20:30,217Z tomcat-http--47 vsphere.local        50cfdef7-a086-4471-88b0-1f5badd418e5 TRACE com.vmware.identity.sts.auth.impl.BSTAuthenticator] Authenticating by BST...

[2016-10-06 19:20:33,687Z tomcat-http--24 vsphere.local        f40877eb-df35-4fff-8047-23099c4044a7 TRACE com.vmware.identity.sts.auth.impl.BSTAuthenticator] Authenticating by BST...

[2016-10-06 19:20:36,572Z tomcat-http--27 vsphere.local        65df49a9-6aae-4672-ac58-360b46323469 TRACE com.vmware.identity.sts.auth.impl.BSTAuthenticator] Authenticating by BST...

[2016-10-06 19:20:37,325Z tomcat-http--40 vsphere.local        784e8a88-4087-47b1-8776-74a11075cc6e TRACE com.vmware.identity.sts.auth.impl.BSTAuthenticator] Authenticating by BST...

[2016-10-06 19:20:38,065Z tomcat-http--2 vsphere.local        5cca0e91-a164-4882-a4ba-e40a74fccca3 TRACE com.vmware.identity.sts.auth.impl.BSTAuthenticator] Authenticating by BST...

[2016-10-06 19:20:44,532Z tomcat-http--22 vsphere.local        d71139c2-9856-4f5b-9f6c-c6420367f571 TRACE com.vmware.identity.sts.auth.impl.BSTAuthenticator] Authenticating by BST...

[2016-10-06 19:20:50,299Z tomcat-http--37 vsphere.local        0378f771-8969-4b99-ada3-e62667002f4c TRACE com.vmware.identity.sts.auth.impl.BSTAuthenticator] Authenticating by BST...

[2016-10-06 19:20:53,746Z tomcat-http--32 vsphere.local        33e7b8c6-7466-48ac-83e1-c88986a90c5f TRACE com.vmware.identity.sts.auth.impl.BSTAuthenticator] Authenticating by BST...

[2016-10-06 19:20:56,630Z tomcat-http--43 vsphere.local        c00c811e-19d7-4388-a567-2e7e328eb432 TRACE com.vmware.identity.sts.auth.impl.BSTAuthenticator] Authenticating by BST...

[2016-10-06 19:20:57,390Z tomcat-http--7 vsphere.local        aa917b8c-aa64-4ec5-a745-ddf911895981 TRACE com.vmware.identity.sts.auth.impl.BSTAuthenticator] Authenticating by BST...

On the 'primary' CAFE appliance:   /var/log/vmware/vcac/catalina.out:

There are many many many authentication failure messages in catalina.out on the Cafe VA.  Here are some examples:

2016-10-06 14:15:16,010 vcac: [component="cafe:catalog" priority="INFO" thread="mainTaskExecutor-3" tenant=""] com.vmware.vcac.core.service.identity.config.ServiceIdentityInitializer.initServiceName:76 - The property serviceName initialized to: catalog-service.

2016-10-06 14:15:16,010 vcac: [component="cafe:catalog" priority="INFO" thread="mainTaskExecutor-3" tenant=""] com.vmware.vcac.core.service.registry.config.ServiceRegistryInitializer.call:261 - Service catalog-service registration started ...

2016-10-06 14:15:16,067 vcac: [component="cafe:catalog" priority="ERROR" thread="mainTaskExecutor-3" tenant=""] com.vmware.vim.sso.client.impl.SoapBindingImpl.sendMessage:133 - SOAP fault

javax.xml.ws.soap.SOAPFaultException: Invalid credentials

        at com.sun.xml.internal.ws.fault.SOAP11Fault.getProtocolException(SOAP11Fault.java:178)

        at com.sun.xml.internal.ws.fault.SOAPFaultBuilder.createException(SOAPFaultBuilder.java:117)

        at com.sun.xml.internal.ws.client.dispatch.DispatchImpl.doInvoke(DispatchImpl.java:184)

        at com.sun.xml.internal.ws.client.dispatch.DispatchImpl.invoke(DispatchImpl.java:203)

        at com.vmware.vim.sso.client.impl.SoapBindingImpl.sendMessage(SoapBindingImpl.java:130)

        at com.vmware.vim.sso.client.impl.SoapBindingImpl.sendMessage(SoapBindingImpl.java:81)

        at com.vmware.vim.sso.client.impl.SecurityTokenServiceImpl$RequestResponseProcessor.sendRequest(SecurityTokenServiceImpl.java:767)

        at com.vmware.vim.sso.client.impl.SecurityTokenServiceImpl$RequestResponseProcessor.executeRoundtrip(SecurityTokenServiceImpl.java:697)

        at com.vmware.vim.sso.client.impl.SecurityTokenServiceImpl.acquireTokenByCertificate(SecurityTokenServiceImpl.java:458)

        at com.vmware.vcac.authentication.sts.SamlTokenService$SolutionSamlTokensHolder.newValue(SamlTokenService.java:511)

        at com.vmware.vcac.authentication.sts.SamlTokenService$SolutionSamlTokensHolder.newValue(SamlTokenService.java:488)

        at com.vmware.vcac.platform.support.TemporalVariable.get(TemporalVariable.java:29)

        at com.vmware.vcac.authentication.sts.SamlTokenService.acquireSolutionToken(SamlTokenService.java:80)

        at com.vmware.vcac.core.componentregistry.rest.client.EndPointAuthenticationManager.getServiceSolutionToken(EndPointAuthenticationManager.java:293)

        at com.vmware.vcac.core.componentregistry.rest.client.EndPointAuthenticationManager.access$200(EndPointAuthenticationManager.java:36)

        at com.vmware.vcac.core.componentregistry.rest.client.EndPointAuthenticationManager$SolutionSsoAuthentication.getHoKToken(EndPointAuthenticationManager.java:414)

        at com.vmware.vcac.authentication.sso.client.BaseSsoAuthentication.hashCode(BaseSsoAuthentication.java:96)

        at com.vmware.vcac.core.componentregistry.rest.client.RestClientEndPointFactory.getComponentRegistryRestClient(RestClientEndPointFactory.java:216)

        at com.vmware.vcac.core.componentregistry.rest.client.RestClientEndPointFactory.getEndPointService(RestClientEndPointFactory.java:178)

        at com.vmware.vcac.core.componentregistry.rest.client.RestClientEndPointFactory.getDefaultRestClientByEndPointTypeWhenAvailable(RestClientEndPointFactory.java:265)

        at com.vmware.vcac.core.componentregistry.rest.client.SolutionRestClientManager.restClientForSolutionUserByTypeWhenAvailable(SolutionRestClientManager.java:86)

        at com.vmware.vcac.core.componentregistry.rest.client.SolutionRestClientManager.restClientForSolutionUserByTypeWhenAvailable(SolutionRestClientManager.java:80)

        at com.vmware.vcac.core.service.registry.config.ServiceRegistryInitializer$2.call(ServiceRegistryInitializer.java:262)

        at com.vmware.vcac.core.service.registry.config.ServiceRegistryInitializer$2.call(ServiceRegistryInitializer.java:258)

        at com.vmware.vcac.platform.rest.client.support.RetriableOperation.call(RetriableOperation.java:60)

        at java.util.concurrent.FutureTask.run(FutureTask.java:262)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

        at java.lang.Thread.run(Thread.java:745)

2016-10-06 14:15:16,068 vcac: [component="cafe:catalog" priority="INFO" thread="mainTaskExecutor-3" tenant=""] com.vmware.vim.sso.client.impl.SecurityTokenServiceImpl$RequestResponseProcessor.handleFaultCondition:834 - Provided credentials are not valid.

2016-10-06 14:15:16,068 vcac: [component="cafe:catalog" priority="WARN" thread="mainTaskExecutor-3" tenant=""] com.vmware.vcac.platform.rest.client.support.RetriableOperation.call:74 - Exception handled during retry operation with message: com.vmware.vim.sso.client.exception.AuthenticationFailedException: Provided credentials are not valid.

2016-10-06 14:15:16,068 vcac: [component="cafe:catalog" priority="INFO" thread="mainTaskExecutor-3" tenant=""] com.vmware.vcac.platform.rest.client.support.RetriableOperation.call:76 - Retries left: [20]. Sleeping for [20] seconds before the next retry attempt.

2016-10-06 14:15:22,351 vcac: [component="cafe:identity" priority="WARN" thread="tomcat-http--3" tenant="vsphere.local"] org.springframework.web.client.RestTemplate.handleResponseError:581 - GET request for "https://clouddev.na.com/component-registry/endpoints/types/sso" resulted in 503 (Service Temporarily Unavailable); invoking error handler

2016-10-06 14:15:22,358 vcac: [component="cafe:identity" priority="WARN" thread="tomcat-http--3" tenant="vsphere.local"] org.springframework.web.client.RestTemplate.handleResponseError:581 - GET request for "https://clouddev.na.com/component-registry/endpoints/types/sso" resulted in 503 (Service Temporarily Unavailable); invoking error handler

2016-10-06 14:15:22,359 vcac: [component="cafe:identity" priority="WARN" thread="tomcat-http--3" tenant="vsphere.local"] com.vmware.vcac.authentication.http.spring.SamlTokenAuthenticationFilter.handleException:115 - Authentication request for failed: Unable to authenticate with SamlToken

2016-10-06 14:15:22,359 vcac: [component="cafe:identity" priority="WARN" thread="tomcat-http--3" tenant="vsphere.local"] com.vmware.vcac.authentication.http.spring.SamlTokenAuthenticationFilter.handleException:116 - Failed:

org.springframework.security.authentication.BadCredentialsException: Unable to authenticate with SamlToken

        at com.vmware.vcac.authentication.http.spring.SamlTokenAuthenticationFilter.attemptAuthentication(SamlTokenAuthenticationFilter.java:51)

        at com.vmware.vcac.authentication.http.spring.BaseTokenAuthenticationFilter.doFilter(BaseTokenAuthenticationFilter.java:55)

        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)

        at com.vmware.vcac.authentication.http.tenancy.TenancyContextFilter.doFilterWithTenancyContext(TenancyContextFilter.java:67)

        at com.vmware.vcac.authentication.http.tenancy.TenancyContextFilter.doFilter(TenancyContextFilter.java:54)

        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)

        at org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter.doFilterInternal(WebAsyncManagerIntegrationFilter.java:50)

        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:108)

        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)

        at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:87)

        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)

        at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:192)

        at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:160)

        at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:344)

        at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:261)

        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)

        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)

        at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:88)

        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:108)

        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)

        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)

        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:220)

        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:122)

        at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:504)

        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:170)

        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)

        at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:950)

        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)

        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:421)

        at org.apache.coyote.ajp.AjpProcessor.process(AjpProcessor.java:190)

        at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:611)

        at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:314)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

        at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)

        at java.lang.Thread.run(Thread.java:745)

Caused by: com.vmware.vcac.authentication.http.SamlAuthenticationException: Failed to create IdP configuration for tenant vsphere.local

        at com.vmware.vcac.authentication.http.SamlTokenExtractor.extractToken(SamlTokenExtractor.java:223)

        at com.vmware.vcac.authentication.http.SamlTokenExtractor.extractToken(SamlTokenExtractor.java:221)

        at com.vmware.vcac.authentication.http.SamlTokenExtractor.extractSamlToken(SamlTokenExtractor.java:78)

        at com.vmware.vcac.authentication.http.spring.SamlTokenAuthenticationFilter.attemptAuthentication(SamlTokenAuthenticationFilter.java:34)

        ... 35 more

Caused by: java.lang.IllegalStateException: Failed to create IdP configuration for tenant vsphere.local

        at com.vmware.vcac.authentication.http.idp.IdPMetadataSettingManager.getOrCreateIdPConfigurationIfNeeded(IdPMetadataSettingManager.java:58)

        at com.vmware.vcac.authentication.http.SamlTokenExtractor.getRootCertificatesForTenant(SamlTokenExtractor.java:244)

        at com.vmware.vcac.authentication.http.SamlTokenExtractor.getVerifier(SamlTokenExtractor.java:234)

        at com.vmware.vcac.authentication.http.SamlTokenExtractor.extractToken(SamlTokenExtractor.java:212)

        ... 38 more

Caused by: org.springframework.web.client.HttpServerErrorException: 503 Service Temporarily Unavailable

        at org.springframework.web.client.DefaultResponseErrorHandler.handleError(DefaultResponseErrorHandler.java:94)

        at com.vmware.vcac.platform.rest.client.error.ResponseErrorHandler.handleError(ResponseErrorHandler.java:49)

        at org.springframework.web.client.RestTemplate.handleResponseError(RestTemplate.java:588)

        at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:546)

        at org.springframework.web.client.RestTemplate.execute(RestTemplate.java:517)

        at org.springframework.web.client.RestTemplate.getForObject(RestTemplate.java:255)

        at com.vmware.vcac.platform.rest.client.impl.RestClientImpl.get(RestClientImpl.java:237)

        at com.vmware.vcac.platform.rest.client.services.AbstractService.get(AbstractService.java:64)

        at com.vmware.vcac.core.componentregistry.rest.client.service.EndpointService.getSsoEndPoint(EndpointService.java:102)

        at com.vmware.vcac.core.componentregistry.rest.client.IdpMetadataSettingRetrieverImpl.getSsoEndpoint(IdpMetadataSettingRetrieverImpl.java:32)

        at com.vmware.vcac.core.componentregistry.rest.client.IdpMetadataSettingRetrieverImpl.getIdPConfiguration(IdpMetadataSettingRetrieverImpl.java:24)

        at com.vmware.vcac.authentication.http.idp.IdPMetadataSettingManager.getOrCreateIdPConfigurationIfNeeded(IdPMetadataSettingManager.java:40)

        ... 41 more

Reply
0 Kudos
1 Solution

Accepted Solutions
Ukusic
Enthusiast
Enthusiast
Jump to solution

Hi,

 

Have not looked thru your logs in detail but we had something similar, on 6.2.3.

 

"Login failed. Please contact your System Administrator and report error code..."

 

 

A internal user cert had expired after 2 years... They had fixed in 6.2.3 or something so the cert gets auto renewed. The problem was that stuff stopped working anyway bc the new cert and permissions broke 😞

 

This was the sulotion for us.

 

https://kb.vmware.com/kb/2116525

View solution in original post

Reply
0 Kudos
3 Replies
Ukusic
Enthusiast
Enthusiast
Jump to solution

Hi,

 

Have not looked thru your logs in detail but we had something similar, on 6.2.3.

 

"Login failed. Please contact your System Administrator and report error code..."

 

 

A internal user cert had expired after 2 years... They had fixed in 6.2.3 or something so the cert gets auto renewed. The problem was that stuff stopped working anyway bc the new cert and permissions broke 😞

 

This was the sulotion for us.

 

https://kb.vmware.com/kb/2116525

Reply
0 Kudos
VMpalavsachin27
Contributor
Contributor
Jump to solution

Yes, that KB did also help me once.

Another strange thing we had was the password expired for IAAS service account and after reseting and disabling password expiration this started working after complete restart.

Reply
0 Kudos
admin
Immortal
Immortal
Jump to solution

You should be able to find Q2n6XUV7 in the vra cataloina.out logs. If you search for that string, it will take you to the line of the error in the catalina.log file. Can you paste that here?

Reply
0 Kudos