Can't sign in with AD credentials after vCenter 6.5u3 update

[itcomputers]

Hello everyone,

First of all, I want to express my gratitude in advance to all the people who help on this forum.

My issue is as follows:

We have 2 vCenters in different datacenters, and in both of them, we upgraded to vCenter 6.5u3 (due to hardware constraints, we cannot upgrade to version 6.7 or later). The entire upgrade process went smoothly without any errors. However, the problem is that in one of them, we can no longer connect with any AD user accounts that we could previously (though we can still log in with local users like administrator@vsphere.local). When we try to connect, the vCenter web interface displays "invalid credentials."

The only difference between the two vCenters is that one has the hostname (checked from our vCenter Server Appliance at :5480) as vcenter1.domain2.loc, while the other (the one working correctly) has the hostname vcenter2.domain1.loc.

Additionally, when I run the following command on vCenter1, the result is:

root@vCenter1 [ /opt/likewise/bin ]# ./domainjoin-cli query
Name = vCenter1
Domain = domain1.loc
Distinguished Name = CN=vCenter1,OU=XXX,OU=DATACENTER,DC=domain1,DC=loc

And on vCenter2, the result is:

root@vCenter2 [ /opt/likewise/bin ]# ./domainjoin-cli query
Name = vCenter2
Domain = domain1.loc
Distinguished Name = CN=vCenter2,OU=XXX,OU=DATACENTER,DC=domain1,DC=loc

 The logs from /var/log/vmware/sso/websso.log on the vCenter1 server, which is experiencing authentication issues with AD, show the following:

[2023-09-06T11:16:16.774+02:00  tomcat-http--31                                       INFO  com.vmware.identity.SsoController] Welcome to SP-initiated AuthnRequest handler! The client locale is en_US, tenant is vsphere.local
[2023-09-06T11:16:16.774+02:00  tomcat-http--31                                       INFO  com.vmware.identity.SsoController] Request URL is https://vcenter1.domain2.loc/websso/SAML2/SSO/vsphere.local
[2023-09-06T11:16:16.821+02:00  tomcat-http--31  6aa64f1c-a2e6-43b6-a88f-6bd0d57c5cf9 INFO  com.vmware.identity.samlservice.impl.AuthnRequestStateValidator] Validating SAML AuthnRequest, ID: _23ac03ebd4b6ff47da2603ef1485864d
[2023-09-06T11:16:16.828+02:00  tomcat-http--31  6aa64f1c-a2e6-43b6-a88f-6bd0d57c5cf9 INFO  com.vmware.identity.samlservice.impl.AuthnRequestStateValidator] Authn request proxyCount= null set isProxying=false
[2023-09-06T11:16:16.838+02:00  tomcat-http--31  6aa64f1c-a2e6-43b6-a88f-6bd0d57c5cf9 INFO  com.vmware.identity.samlservice.impl.AuthnRequestStateValidator] Authentication request validation succeeded
[2023-09-06T11:16:16.841+02:00  tomcat-http--31  6aa64f1c-a2e6-43b6-a88f-6bd0d57c5cf9 INFO  com.vmware.identity.SsoController] Server SPN is HTTP/vcenter1.domain1.loc
[2023-09-06T11:16:16.842+02:00  tomcat-http--31  6aa64f1c-a2e6-43b6-a88f-6bd0d57c5cf9 INFO  com.vmware.identity.SsoController] Accessing Tenant vsphere.local, brand name string null
[2023-09-06T11:16:23.892+02:00  tomcat-http--35                                       INFO  com.vmware.identity.SsoController] Welcome to SP-initiated AuthnRequest handler! The client locale is en_US, tenant is vsphere.local
[2023-09-06T11:16:23.892+02:00  tomcat-http--35                                       INFO  com.vmware.identity.SsoController] Request URL is https://vcenter1.domain2.loc/websso/SAML2/SSO/vsphere.local
[2023-09-06T11:16:23.953+02:00  tomcat-http--35  547d652e-b37d-4b88-b8e0-ca894f5907d7 INFO  com.vmware.identity.samlservice.impl.AuthnRequestStateValidator] Validating SAML AuthnRequest, ID: _23ac03ebd4b6ff47da2603ef1485864d
[2023-09-06T11:16:23.962+02:00  tomcat-http--35  547d652e-b37d-4b88-b8e0-ca894f5907d7 INFO  com.vmware.identity.samlservice.impl.AuthnRequestStateValidator] Authn request proxyCount= null set isProxying=false
[2023-09-06T11:16:23.972+02:00  tomcat-http--35  547d652e-b37d-4b88-b8e0-ca894f5907d7 INFO  com.vmware.identity.samlservice.impl.AuthnRequestStateValidator] Authentication request validation succeeded
[2023-09-06T11:16:24.012+02:00  tomcat-http--35  547d652e-b37d-4b88-b8e0-ca894f5907d7 ERROR com.vmware.identity.samlservice.impl.CasIdmAccessor] Caught exception.
com.vmware.identity.idm.IDMLoginException: Native platform error [code: 851968][null][null]
        at com.vmware.identity.idm.server.ServerUtils.getRemoteException(ServerUtils.java:118) ~[?:?]
        at com.vmware.identity.idm.server.IdentityManager.authenticate(IdentityManager.java:9825) ~[?:?]
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_341]
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_341]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_341]
        at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_341]
        at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:357) ~[?:1.8.0_341]
        at sun.rmi.transport.Transport$1.run(Transport.java:200) ~[?:1.8.0_341]
        at sun.rmi.transport.Transport$1.run(Transport.java:197) ~[?:1.8.0_341]
        at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_341]
        at sun.rmi.transport.Transport.serviceCall(Transport.java:196) ~[?:1.8.0_341]
        at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:573) ~[?:1.8.0_341]
        at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:834) ~[?:1.8.0_341]
        at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.lambda$run$0(TCPTransport.java:688) ~[?:1.8.0_341]
        at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_341]
        at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:687) ~[?:1.8.0_341]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_341]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_341]
        at java.lang.Thread.run(Thread.java:750) ~[?:1.8.0_341]
        at sun.rmi.transport.StreamRemoteCall.exceptionReceivedFromServer(StreamRemoteCall.java:303) ~[?:1.8.0_341]
        at sun.rmi.transport.StreamRemoteCall.executeCall(StreamRemoteCall.java:279) ~[?:1.8.0_341]
        at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:163) ~[?:1.8.0_341]
        at java.rmi.server.RemoteObjectInvocationHandler.invokeRemoteMethod(RemoteObjectInvocationHandler.java:235) ~[?:1.8.0_341]
        at java.rmi.server.RemoteObjectInvocationHandler.invoke(RemoteObjectInvocationHandler.java:180) ~[?:1.8.0_341]
        at com.sun.proxy.$Proxy302.authenticate(Unknown Source) ~[?:?]
        at com.vmware.identity.idm.client.CasIdmClient.authenticate(CasIdmClient.java:1311) ~[vmware-identity-idm-client.jar:?]
        at com.vmware.identity.samlservice.impl.CasIdmAccessor.authenticate(CasIdmAccessor.java:467) [websso.jar:?]
        at com.vmware.identity.samlservice.impl.AuthnRequestStatePasswordAuthenticationFilter.authenticate(AuthnRequestStatePasswordAuthenticationFilter.java:95) [websso.jar:?]
        at com.vmware.identity.samlservice.impl.AuthnRequestStatePasswordAuthenticationFilter.authenticate(AuthnRequestStatePasswordAuthenticationFilter.java:45) [websso.jar:?]
        at com.vmware.identity.samlservice.impl.AuthnRequestStateCookieWrapper.authenticate(AuthnRequestStateCookieWrapper.java:149) [websso.jar:?]
        at com.vmware.identity.samlservice.impl.AuthnRequestStateCookieWrapper.authenticate(AuthnRequestStateCookieWrapper.java:43) [websso.jar:?]
        at com.vmware.identity.samlservice.AuthnRequestState.authenticate(AuthnRequestState.java:467) [websso.jar:?]
        at com.vmware.identity.BaseSsoController.processSsoRequest(BaseSsoController.java:88) [websso.jar:?]
        at com.vmware.identity.SsoController.sso(SsoController.java:100) [websso.jar:?]
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_341]
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_341]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_341]
        at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_341]
        at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:204) [spring-web-4.3.30.RELEASE.jar:4.3.30.RELEASE]
        at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:132) [spring-web-4.3.30.RELEASE.jar:4.3.30.RELEASE]
        at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:97) [spring-webmvc-4.3.30.RELEASE.jar:4.3.30.RELEASE]
        at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:854) [spring-webmvc-4.3.30.RELEASE.jar:4.3.30.RELEASE]
        at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:765) [spring-webmvc-4.3.30.RELEASE.jar:4.3.30.RELEASE]
        at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:85) [spring-webmvc-4.3.30.RELEASE.jar:4.3.30.RELEASE]
        at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:967) [spring-webmvc-4.3.30.RELEASE.jar:4.3.30.RELEASE]
        at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:901) [spring-webmvc-4.3.30.RELEASE.jar:4.3.30.RELEASE]
        at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:970) [spring-webmvc-4.3.30.RELEASE.jar:4.3.30.RELEASE]
        at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:872) [spring-webmvc-4.3.30.RELEASE.jar:4.3.30.RELEASE]
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:681) [servlet-api.jar:?]
        at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:846) [spring-webmvc-4.3.30.RELEASE.jar:4.3.30.RELEASE]
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:764) [servlet-api.jar:?]
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231) [catalina.jar:8.5.81]
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [catalina.jar:8.5.81]
        at com.vmware.identity.SecurityRequestWrapperFilter.doFilterInternal(SecurityRequestWrapperFilter.java:49) [websso.jar:?]
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-4.3.30.RELEASE.jar:4.3.30.RELEASE]
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [catalina.jar:8.5.81]
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [catalina.jar:8.5.81]
        at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52) [tomcat-websocket.jar:8.5.81]
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [catalina.jar:8.5.81]
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [catalina.jar:8.5.81]
        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:197) [catalina.jar:8.5.81]
        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:97) [catalina.jar:8.5.81]
        at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:543) [catalina.jar:8.5.81]
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:135) [catalina.jar:8.5.81]
        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92) [catalina.jar:8.5.81]
        at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:698) [catalina.jar:8.5.81]
        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:78) [catalina.jar:8.5.81]
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:367) [catalina.jar:8.5.81]
        at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:639) [tomcat-coyote.jar:8.5.81]
        at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65) [tomcat-coyote.jar:8.5.81]
        at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:882) [tomcat-coyote.jar:8.5.81]
        at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1691) [tomcat-coyote.jar:8.5.81]
        at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49) [tomcat-coyote.jar:8.5.81]
        at org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1191) [tomcat-util.jar:8.5.81]
        at org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:659) [tomcat-util.jar:8.5.81]
        at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) [tomcat-util.jar:8.5.81]
        at java.lang.Thread.run(Thread.java:750) [?:1.8.0_341]
[2023-09-06T11:16:24.021+02:00  tomcat-http--35  547d652e-b37d-4b88-b8e0-ca894f5907d7 INFO  auditlogger] {"user":"user@domain1.loc","client":"192.168.134.118","timestamp":"09/06/2023 11:16:24 GMT+02:00","description":"User user@domain1.loc@192.168.134.118 failed to log in with response code 401","eventSeverity":"INFO","type":"com.vmware.sso.LoginFailure"}
[2023-09-06T11:16:24.021+02:00  tomcat-http--35  547d652e-b37d-4b88-b8e0-ca894f5907d7 ERROR com.vmware.identity.samlservice.AuthnRequestState] Caught Saml Service Exception from authenticate com.vmware.identity.samlservice.SamlServiceException
[2023-09-06T11:16:24.021+02:00  tomcat-http--35  547d652e-b37d-4b88-b8e0-ca894f5907d7 ERROR com.vmware.identity.BaseSsoController] Sending error to browser. ERROR: 401, message

Clarifications:

• For security and privacy reasons, I have changed the username, our domains, and the vCenter names in this post.
• I have already restarted the problematic vCenter several times without success.
• I have checked and the time and NTP settings are correct.
• I have reviewed the DNS/FQDN and they are accurate.

Could someone shed some light on this issue?

Thank you very much to everyone,

[CallistoJag]

During the upgrade did someone accidentally enter .domain2.loc at any point? Seems you need to find where this is referenced from and remove it.

[itcomputers]

Hello @CallistoJag , the vCenter1 was configured in our .domain2.loc from the very beggining and always worked fine like that, so I believe this is not the problem.

[Sachchidanand]

From the first screen, i can see the domain name as domain1.loc while in logs it's showing domain2.loc, see the line:

https://vcenter1.domain2.loc/websso/SAML2/SSO/vsphere.local

It seems due to mismatch of domain, you are getting this issue.

Please rectify it and see if it works.

Regards,

Sachchidanand

[CallistoJag]

did domain2.loc used to be a trusted domain in your org and it no longer is? This might explain the issue. I am still sure domain2.loc is the issue. DNS records could be checked/re-set? Maybe set an alias for domain1.loc for the vCenter1, better still set it so that both are domain1.loc  

[itcomputers]

Hello,

Thank you for your support, @CallistoJag and @Sachchidanand.

Regarding vCenter1, it has been a part of domain2.loc from the very beginning, and that has never been an issue.

Regarding DNS, I have the following records/aliases:

• vcenter1.domain1.loc
• vcenter1.domain2.loc
• vcenter2.domain1.loc
• vcenter2.domain2.loc

As for the question, "Did domain2.loc used to be a trusted domain in your organization and it no longer is?" --> domain2.loc has always been a trusted domain; we haven't made any changes to that. It's just one of several domains we have in our company.

Do you think this workaround would solve my issue? You can find it here: https://bhanuwriter.com/unable-to-login-into-vcenter-server-with-ad-credentials/ 

Thanks again 

[CallistoJag]

Kerberos has always been at the route of AD issues I have had, usually due to hardening. So it is certainly worth looking at this workaround, the checks will do no harm at least. Good luck!

[itcomputers · Accepted solution]

Thanks @CallistoJag !

I have a doubt with that workaround: do you think it's a good practice to remove manually the Computer AD Object and the DNS records related of this vCenter1 after leaving it from domain? Or it's not necessary?

I will follow this official VMware guide: https://kb.vmware.com/s/article/50112055 

[CallistoJag]

These steps are not an issue if you have access to re-add them, but if you are not comfortable with this, of course first try to just leave and rejoin first, it does no harm

[itcomputers]

Finally solved it!!! It worked like a charm the workaround.

For all who can has the same issue, here my steps done:

1. Connect to the vCenter Server console or SSH session and log in using root credentials.
2. Run this command to disjoin the Appliance from the domain: /opt/likewise/bin/domainjoin-cli leave
3. Verify the status using "/opt/likewise/bin/domainjoin-cli query" command.
4. Run this command to restart the vCenter Server services: "service-control --stop --all" and "service-control --start --all"
5. Delete manually the computer object from AD
6. /opt/likewise/bin/domainjoin-cli join domain.com Administrator (or other user with privileges you have)
7. Run this command to restart the vCenter Server services: "service-control --stop --all" and "service-control --start --all"
8. Successful login to vCenter with AD credentials 

Thanks a lot @CallistoJag and @Sachchidanand