Upgrade VCSA 6.7 to VCSA 7.0 fails on Stage 2 - Starting VMware Security Token Service...

[Lukas_Ro]

Hello,

I tried to upgrade vCSA 6.7.0.46000  to vCSA 7.0.1.00200 on January 2021 and it fails at Stage 2 -> A problem occured while - Starting VMware Security Token Service. I didn't have time to deal with it. 

I got to it again last week and tried everything with new versions. So i tried from vCSA 6.7.0.47000 to vCSA 7.0.2.00000, but with the same result. All pre-upgrade check passed. Upgrading via build-in account administrator@vsphere.local.

Error:

Encountered an internal error. Traceback (most recent call last): File "/usr/lib/vmware/site-packages/cis/cisreglib.py", line 249, in securityctx_modifier yield File "/usr/lib/vmware/site-packages/cis/cisreglib.py", line 348, in add_securityctx_to_requests return req_method(self, *args, **kargs) File "/usr/lib/vmware/site-packages/cis/cisreglib.py", line 360, in register_service svc_create_spec) File "/usr/lib/vmware/site-packages/pyVmomi/VmomiSupport.py", line 583, in <lambda> self.f(*(self.args + (obj,) + args), **kwargs) File "/usr/lib/vmware/site-packages/pyVmomi/VmomiSupport.py", line 373, in _InvokeMethod return self._stub.InvokeMethod(self, info, args) File "/usr/lib/vmware/site-packages/pyVmomi/SoapAdapter.py", line 1570, in InvokeMethod raise obj # pylint: disable-msg=E0702 pyVmomi.VmomiSupport.vmodl.fault.SecurityError: (vmodl.fault.SecurityError) { dynamicType = <unset>, dynamicProperty = (vmodl.DynamicProperty) [], msg = '', faultCause = <unset>, faultMessage = (vmodl.LocalizableMessage) [] } During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/lib/vmidentity/firstboot/vmidentity-firstboot.py", line 1861, in main vmidentityFB.boot() File "/usr/lib/vmidentity/firstboot/vmidentity-firstboot.py", line 369, in boot self.registerTokenServiceWithLookupService() File "/usr/lib/vmidentity/firstboot/vmidentity-firstboot.py", line 656, in registerTokenServiceWithLookupService raise e File "/usr/lib/vmidentity/firstboot/vmidentity-firstboot.py", line 652, in registerTokenServiceWithLookupService dynVars=dynVars) File "/usr/lib/vmware-cm/bin/cloudvmcisreg.py", line 710, in cloudvm_sso_cm_register serviceId = do_lsauthz_operation(cisreg_opts_dict) File "/usr/lib/vmware/site-packages/cis/cisreglib.py", line 1118, in do_lsauthz_operation ls_obj.register_service(svc_id, svc_create_spec) File "/usr/lib/vmware/site-packages/cis/cisreglib.py", line 348, in add_securityctx_to_requests return req_method(self, *args, **kargs) File "/usr/lib/python3.7/contextlib.py", line 161, in __exit__ raise RuntimeError("generator didn't stop after throw()") RuntimeError: generator didn't stop after throw()

Resolution

This is an unrecoverable error, please retry install. If you encounter this error again, please search for these symptoms in the VMware Knowledge Base for any known issues and possible resolutions. If none can be found, collect a support bundle and open a support request

Do you have any idea?

[Ajay1988]

Yes.. Any default solution users removed from SSO groups will cause issues. 

Be careful of taking cold snapshots >> https://virtual-power.in/f/29-vcenter-in-enhanced-linked-mode-best-practice-before-changes

[TirTul]

Thanks @Ajay1988 . We have been very diligent with doing cold snapshots and verifying replication status before starting the upgrades and after each vCenter in the sso was upgraded.

So we have successfully upgraded 5 out of the 6 vCenters in ELM. So I'm not sure how the registrations for this one vCenter got the way it is. If what I found is the issue, we're still waiting on support to get back to us...while all six still are running on snapshots.

[JairoCA]

I had the same error. my troubleshooting was:

For sure I will make it work without the DNS..... 

2 hrs later,,,,, mmmm maybe the DNS  could help.....

2 hrs later...... Ok, I will install the DNS on a VM with Winserver.....

15 min later...... it worked!!!!!!!

So.... do not loose your time... this error is fixed using the DNS...... it is an obligation.....

Hope this helps...

Jairo 

[msardar21]

Thanks a ton @Ajay1988 It did work like a charm.

I had issue upgrading VCSA7.x to 8.x managed by VCF5.x and this did help me finally. Used the below command on source(old) VCSA to add the missing vSphere local Administrator user.

/usr/lib/vmware-vmafd/bin/dir-cli group modify --name Administrators --add Administrator

[msardar21]

It did work like a charm Thank you @Ajay1988