DaIceMan
Enthusiast
Enthusiast

vCenter 6/6.5 upgrade to 6.7U3 fails during vmidentity-firstboot.py - self.checkSTS

I'm encountering this error attempting to upgrade a 6.0U3 (Windows install) to 6.7. I was able to successfully upgrade the 6.0 install to 6.5 (tried this because direct to 6.7 didn't work with same error) but I still get this error when stepping up to 6.7 latest build. The fbinstall.json log shows:

Traceback (most recent call last):\n File \"C:\\Program Files\\VMware\\vCenter Server\\firstboot\\vmidentity-firstboot.py\", line 1641, in main\n vmidentityFB.boot()\n File \"C:\\Program Files\\VMware\\vCenter Server\\firstboot\\vmidentity-firstboot.py\", line 345, in boot\n self.checkSTS(self.__stsRetryCount, self.__stsRetryInterval)\n File \"C:\\Program Files\\VMware\\vCenter Server\\firstboot\\vmidentity-firstboot.py\", line 1179, in checkSTS\n raise Exception('Failed to initialize Secure Token Server.')\nException: Failed to initialize Secure Token Server.\n"

And going further in depth in the collected vmidentity-firstboot.py log file:

.......
giu 24, 2021 4:08:24 PM com.sun.xml.internal.messaging.saaj.soap.MessageImpl saveChanges
SEVERE: SAAJ0540: Error during saving a multipart message
Failed to check VMware STS.
com.vmware.vim.sso.client.exception.ServerCommunicationException: Error communicating to the remote server https://SRVFQDN:7444/sts/STSService/vsphere.local
at com.vmware.vim.sso.client.impl.SecurityTokenServiceImpl$RequestResponseProcessor.sendRequest(SecurityTokenServiceImpl.java:940)
at com.vmware.vim.sso.client.impl.SecurityTokenServiceImpl$RequestResponseProcessor.executeRoundtrip(SecurityTokenServiceImpl.java:856)
at com.vmware.vim.sso.client.impl.SecurityTokenServiceImpl.acquireToken(SecurityTokenServiceImpl.java:144)
at com.vmware.identity.installer.STSInstaller.check_sts_endpoints(STSInstaller.java:424)
at com.vmware.identity.installer.STSInstaller.check_sts(STSInstaller.java:388)
at com.vmware.identity.installer.STSInstaller.check_sts(STSInstaller.java:1194)
at com.vmware.identity.installer.STSInstaller.main(STSInstaller.java:1106)

Caused by: com.vmware.vim.sso.client.exception.ServerCommunicationException: Error communicating to the remote server https://SRVFQDN:7444/sts/STSService/vsphere.local
at com.vmware.vim.sso.client.impl.SoapBindingImpl.sendMessage(SoapBindingImpl.java:178)
at com.vmware.vim.sso.client.impl.SoapBindingImpl.sendMessage(SoapBindingImpl.java:114)
at com.vmware.vim.sso.client.impl.SecurityTokenServiceImpl$RequestResponseProcessor.sendRequest(SecurityTokenServiceImpl.java:927)
... 6 more
Caused by: javax.xml.ws.WebServiceException: com.sun.xml.internal.messaging.saaj.SOAPExceptionImpl: Error during saving a multipart message
at com.sun.xml.internal.ws.client.dispatch.SOAPMessageDispatch.toReturnValue(SOAPMessageDispatch.java:91)
at com.sun.xml.internal.ws.client.dispatch.SOAPMessageDispatch.toReturnValue(SOAPMessageDispatch.java:60)
at com.sun.xml.internal.ws.client.dispatch.DispatchImpl.doInvoke(DispatchImpl.java:274)
at com.sun.xml.internal.ws.client.dispatch.DispatchImpl.invoke(DispatchImpl.java:289)
at com.vmware.vim.sso.client.impl.SoapBindingImpl.sendMessage(SoapBindingImpl.java:161)
... 8 more
Caused by: com.sun.xml.internal.messaging.saaj.SOAPExceptionImpl: Error during saving a multipart message
at com.sun.xml.internal.messaging.saaj.soap.MessageImpl.saveChanges(MessageImpl.java:1204)
at com.sun.xml.internal.ws.api.message.saaj.SAAJFactory.readAsSOAPMessage(SAAJFactory.java:277)
at com.sun.xml.internal.ws.api.message.saaj.SAAJFactory.readAsSAAJ(SAAJFactory.java:197)
at com.sun.xml.internal.ws.api.message.saaj.SAAJFactory.read(SAAJFactory.java:186)
at com.sun.xml.internal.ws.message.AbstractMessageImpl.toSAAJ(AbstractMessageImpl.java:217)
at com.sun.xml.internal.ws.api.message.MessageWrapper.readAsSOAPMessage(MessageWrapper.java:156)
at com.sun.xml.internal.ws.client.dispatch.SOAPMessageDispatch.toReturnValue(SOAPMessageDispatch.java:89)
... 12 more
Caused by: java.lang.NoClassDefFoundError: org/apache/xml/serializer/TreeWalker
at org.apache.xalan.processor.TransformerFactoryImpl.newTransformer(TransformerFactoryImpl.java:823)
at com.sun.xml.internal.messaging.saaj.util.transform.EfficientStreamingTransformer.materialize(EfficientStreamingTransformer.java:106)
at com.sun.xml.internal.messaging.saaj.util.transform.EfficientStreamingTransformer.setOutputProperty(EfficientStreamingTransformer.java:190)
at com.sun.xml.internal.messaging.saaj.soap.impl.EnvelopeImpl.output(EnvelopeImpl.java:260)
at com.sun.xml.internal.messaging.saaj.soap.impl.EnvelopeImpl.output(EnvelopeImpl.java:302)
at com.sun.xml.internal.messaging.saaj.soap.SOAPPartImpl.getContentAsStream(SOAPPartImpl.java:311)
at com.sun.xml.internal.messaging.saaj.soap.MessageImpl.getHeaderBytes(MessageImpl.java:1015)
at com.sun.xml.internal.messaging.saaj.soap.MessageImpl.saveChanges(MessageImpl.java:1166)
... 18 more
Caused by: java.lang.ClassNotFoundException: org.apache.xml.serializer.TreeWalker
at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:355)
at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
... 26 more
giu 24, 2021 4:08:54 PM com.sun.xml.internal.messaging.saaj.soap.MessageImpl saveChanges
SEVERE: SAAJ0540: Error during saving a multipart message
Failed to check VMware STS.

and so forth repeating and then:

2021-06-24T14:12:56.113Z Checking VMware STS...
Built URI https://VCENTERSRVFQDN:7444/sso-adminserver/sdk/vsphere.local
Built URI https://VCENTERSRVFQDN:7444/sts/STSService/vsphere.local
Sleeping for [30] seconds
Checking VMware STS...

10 Times every 30 seconds and finally

2021-06-24T14:12:56.113Z <<<<stdout
2021-06-24T14:12:56.113Z ===Return code: 1
2021-06-24T14:12:56.113Z VMware Identity Service bootstrap failed.
2021-06-24T14:12:56.114Z Exception: Traceback (most recent call last):
File "C:\Program Files\VMware\vCenter Server\firstboot\vmidentity-firstboot.py", line 1641, in main
vmidentityFB.boot()
File "C:\Program Files\VMware\vCenter Server\firstboot\vmidentity-firstboot.py", line 345, in boot
self.checkSTS(self.__stsRetryCount, self.__stsRetryInterval)
File "C:\Program Files\VMware\vCenter Server\firstboot\vmidentity-firstboot.py", line 1179, in checkSTS
raise Exception('Failed to initialize Secure Token Server.')
Exception: Failed to initialize Secure Token Server.

Apparently the STS is not responding at all or not responding correctly. Naturally if I enter that URL before the upgrade it responds and all is well. I have found no further clues at what could be the culprit, I checked the certificates which I had previously renewed (self) and removed the old ones and all are valid. The VMFAD Boots successfully and the 4 ceritificates are valid. Additionally from the vmware-sts-idmd.log:

 

......
2021-06-24T16:08:22.816+02:00 INFO ] [Util] Reading resources from zip file path=[/C:/Program%20Files/VMware/vCenter%20Server/vmware-sso/commonlib/wstClient.jar]
[2021-06-24T16:08:22.817+02:00 INFO ] [Util] Reading resources from decoded zip file path=[/C:/Program Files/VMware/vCenter Server/vmware-sso/commonlib/wstClient.jar]
[2021-06-24T16:08:24.118+02:00 ERROR] [SoapBindingImpl] Error communicating to the remote server https://VCENTERFQDN:7444/sts/STSService/vsphere.local
javax.xml.ws.WebServiceException: com.sun.xml.internal.messaging.saaj.SOAPExceptionImpl: Error during saving a multipart message
at com.sun.xml.internal.ws.client.dispatch.SOAPMessageDispatch.toReturnValue(SOAPMessageDispatch.java:91) ~[?:1.8.0_281]
at com.sun.xml.internal.ws.client.dispatch.SOAPMessageDispatch.toReturnValue(SOAPMessageDispatch.java:60) ~[?:1.8.0_281]
at com.sun.xml.internal.ws.client.dispatch.DispatchImpl.doInvoke(DispatchImpl.java:274) ~[?:1.8.0_281]
at com.sun.xml.internal.ws.client.dispatch.DispatchImpl.invoke(DispatchImpl.java:289) ~[?:1.8.0_281]
at com.vmware.vim.sso.client.impl.SoapBindingImpl.sendMessage(SoapBindingImpl.java:161) [wstClient.jar:?]
at com.vmware.vim.sso.client.impl.SoapBindingImpl.sendMessage(SoapBindingImpl.java:114) [wstClient.jar:?]
at com.vmware.vim.sso.client.impl.SecurityTokenServiceImpl$RequestResponseProcessor.sendRequest(SecurityTokenServiceImpl.java:927) [wstClient.jar:?]
at com.vmware.vim.sso.client.impl.SecurityTokenServiceImpl$RequestResponseProcessor.executeRoundtrip(SecurityTokenServiceImpl.java:856) [wstClient.jar:?]
at com.vmware.vim.sso.client.impl.SecurityTokenServiceImpl.acquireToken(SecurityTokenServiceImpl.java:144) [wstClient.jar:?]
at com.vmware.identity.installer.STSInstaller.check_sts_endpoints(STSInstaller.java:424) [vmware-identity-install.jar:?]
at com.vmware.identity.installer.STSInstaller.check_sts(STSInstaller.java:388) [vmware-identity-install.jar:?]
at com.vmware.identity.installer.STSInstaller.check_sts(STSInstaller.java:1194) [vmware-identity-install.jar:?]
at com.vmware.identity.installer.STSInstaller.main(STSInstaller.java:1106) [vmware-identity-install.jar:?]
Caused by: com.sun.xml.internal.messaging.saaj.SOAPExceptionImpl: Error during saving a multipart message
at com.sun.xml.internal.messaging.saaj.soap.MessageImpl.saveChanges(MessageImpl.java:1204) ~[?:1.8.0_281]
at com.sun.xml.internal.ws.api.message.saaj.SAAJFactory.readAsSOAPMessage(SAAJFactory.java:277) ~[?:1.8.0_281]
at com.sun.xml.internal.ws.api.message.saaj.SAAJFactory.readAsSAAJ(SAAJFactory.java:197) ~[?:1.8.0_281]
at com.sun.xml.internal.ws.api.message.saaj.SAAJFactory.read(SAAJFactory.java:186) ~[?:1.8.0_281]
at com.sun.xml.internal.ws.message.AbstractMessageImpl.toSAAJ(AbstractMessageImpl.java:217) ~[?:1.8.0_281]
at com.sun.xml.internal.ws.api.message.MessageWrapper.readAsSOAPMessage(MessageWrapper.java:156) ~[?:1.8.0_281]
at com.sun.xml.internal.ws.client.dispatch.SOAPMessageDispatch.toReturnValue(SOAPMessageDispatch.java:89) ~[?:1.8.0_281]
.....

And so forth until the IDM server is stopped.

Does anyone have any pointers? Could it be a Java component mix related issue (this server was upgraded from v5.5)? A certificate issue (doesn't seem so)?

Thanks!

0 Kudos
3 Replies
Ank_S
Enthusiast
Enthusiast

Hello @DaIceMan,

 

Follow the below mentioned KB and see if it helps :


https://kb.vmware.com/s/article/76144 

 

PS: Mark kudos or correct answer as appropriate

0 Kudos
DaIceMan
Enthusiast
Enthusiast

Hi Ank, yes I already saw that KB and checked it out, but as you see it is referred to the VCSA not the Windows VIM install. Also from my logs, there is no mention of the STS certificate not being validated. As I stated, my certificates are fine, I have double checked that there are no expired stray ones in the store and root store.

0 Kudos
DaIceMan
Enthusiast
Enthusiast

Digging deeper into the error, it was obvious that it was triggered by a Java Class incompatibility as the STS service WAS responding and running correctly on port 7444. Searching for similar errors I finally found a vmware kb referencing an error during the upgrade process from 6.5 to 6.7U3, specifically kb75289:

"Failed to initialize Secure Token Server""Installation of component VCSServiceManager failed with e...

While this was not my exact error (I never got reported back error code 1603 from the installer), it did raise a red flag as the errors were identical so I investigated the mentioned directory for any outdated jars and found one dated 2015 (xalan-2.7.1.jar) which was exactly the one mentioned in the above kb. I stopped the Vmware Identity Service and dependant ones and removed the jar from the folder and restarted the services. Everything looked fine so I went ahead and ran the update, which this time completed successfully. I have no idea why the error was apparently different, but the culprit was the same outdated jar.

0 Kudos