I recently updated a vCenter Server Appliance (with embedded PSC) from 6.5 to 7.0, and after the upgrade the DRS cluster is trying to deploy the new vCLS machines but repeatedly failing with the error "A general system error occurred: PBM error occurred during PreCreateCheckCallback: Invalid state". I'm also getting the same error when I try to vMotion to this cluster, or when vMotioning to the other non-DRS cluster that's in the same datacenter on this same vCenter server.
When I check the SPS log on the server console, it's being flooded with errors:
2022-05-18T14:07:42.070Z [pool-3-thread-1] ERROR opId=4c18e540-01-01-01 com.vmware.vim.storage.common.security.CommonSessionManager - Login by token failed
com.vmware.vim.sso.client.exception.MalformedTokenException: Error parsing SAML token.
at com.vmware.identity.token.impl.SamlTokenImpl.parseTokenXmlToDom(SamlTokenImpl.java:759)
at com.vmware.identity.token.impl.SamlTokenImpl.<init>(SamlTokenImpl.java:274)
at com.vmware.vim.sso.client.DefaultTokenFactory.parseToken(DefaultTokenFactory.java:69)
at com.vmware.vim.sso.client.DefaultTokenFactory.parseToken(DefaultTokenFactory.java:77)
at com.vmware.vim.storage.common.security.CommonSessionManager.parseSamlToken(CommonSessionManager.java:258)
at com.vmware.vim.storage.common.security.CommonSessionManager.loginByToken(CommonSessionManager.java:158)
at com.vmware.pbm.auth.impl.SessionManagerImpl.loginByToken(SessionManagerImpl.java:44)
at sun.reflect.GeneratedMethodAccessor650.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.vmware.vim.vmomi.server.impl.InvocationTask.run(InvocationTask.java:99)
at com.vmware.vim.vmomi.server.common.impl.RunnableWrapper$1.run(RunnableWrapper.java:47)
at com.vmware.vim.storage.common.task.opctx.RunnableOpCtxDecorator.run(RunnableOpCtxDecorator.java:38)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.xml.sax.SAXParseException; lineNumber: 1; columnNumber: 1; Content is not allowed in prolog.
at com.sun.org.apache.xerces.internal.parsers.DOMParser.parse(DOMParser.java:257)
at com.sun.org.apache.xerces.internal.jaxp.DocumentBuilderImpl.parse(DocumentBuilderImpl.java:339)
at com.vmware.identity.token.impl.SamlTokenImpl.parseTokenXmlToDom(SamlTokenImpl.java:754)
... 17 more
2022-05-18T14:07:42.070Z [pool-3-thread-1] ERROR opId=4c18e540-01-01-01 com.vmware.vim.storage.common.VmodlErrorStrings - Login to PBM failed
com.vmware.vim.storage.common.fault.LoginException: com.vmware.vim.sso.client.exception.MalformedTokenException: Error parsing SAML token.
at com.vmware.vim.storage.common.security.CommonSessionManager.loginByToken(CommonSessionManager.java:176)
at com.vmware.pbm.auth.impl.SessionManagerImpl.loginByToken(SessionManagerImpl.java:44)
at sun.reflect.GeneratedMethodAccessor650.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.vmware.vim.vmomi.server.impl.InvocationTask.run(InvocationTask.java:99)
at com.vmware.vim.vmomi.server.common.impl.RunnableWrapper$1.run(RunnableWrapper.java:47)
at com.vmware.vim.storage.common.task.opctx.RunnableOpCtxDecorator.run(RunnableOpCtxDecorator.java:38)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: com.vmware.vim.sso.client.exception.MalformedTokenException: Error parsing SAML token.
at com.vmware.identity.token.impl.SamlTokenImpl.parseTokenXmlToDom(SamlTokenImpl.java:759)
at com.vmware.identity.token.impl.SamlTokenImpl.<init>(SamlTokenImpl.java:274)
at com.vmware.vim.sso.client.DefaultTokenFactory.parseToken(DefaultTokenFactory.java:69)
at com.vmware.vim.sso.client.DefaultTokenFactory.parseToken(DefaultTokenFactory.java:77)
at com.vmware.vim.storage.common.security.CommonSessionManager.parseSamlToken(CommonSessionManager.java:258)
at com.vmware.vim.storage.common.security.CommonSessionManager.loginByToken(CommonSessionManager.java:158)
... 12 more
Caused by: org.xml.sax.SAXParseException; lineNumber: 1; columnNumber: 1; Content is not allowed in prolog.
at com.sun.org.apache.xerces.internal.parsers.DOMParser.parse(DOMParser.java:257)
at com.sun.org.apache.xerces.internal.jaxp.DocumentBuilderImpl.parse(DocumentBuilderImpl.java:339)
at com.vmware.identity.token.impl.SamlTokenImpl.parseTokenXmlToDom(SamlTokenImpl.java:754)
... 17 more
This was the second vCenter server upgraded within our environment, it shares a SSO domain with another vCenter server that is working fine. When we upgraded this second vCenter server (that is now having problems), the pre-checks indicated a certificate error and specified to use lsdoctor --trustfix, which ran without errors and fixed 3 services, and then the pre-check came back clean the second time.
Logging into the web client with the VMWare SSO domain administrator account works on either vCenter server. How can I fix this impacted vCenter server so that it can deploy new machines / receive incoming vMotions again?
