VMware Cloud Community
Lukas_Ro
Contributor
Contributor
Jump to solution

Upgrade VCSA 6.7 to VCSA 7.0 fails on Stage 2 - Starting VMware Security Token Service...

Hello,

I tried to upgrade vCSA 6.7.0.46000  to vCSA 7.0.1.00200 on January 2021 and it fails at Stage 2 -> A problem occured while - Starting VMware Security Token Service. I didn't have time to deal with it. 

I got to it again last week and tried everything with new versions. So i tried from vCSA 6.7.0.47000 to vCSA 7.0.2.00000, but with the same result. All pre-upgrade check passed. Upgrading via build-in account administrator@vsphere.local.

Error:

Encountered an internal error. Traceback (most recent call last): File "/usr/lib/vmware/site-packages/cis/cisreglib.py", line 249, in securityctx_modifier yield File "/usr/lib/vmware/site-packages/cis/cisreglib.py", line 348, in add_securityctx_to_requests return req_method(self, *args, **kargs) File "/usr/lib/vmware/site-packages/cis/cisreglib.py", line 360, in register_service svc_create_spec) File "/usr/lib/vmware/site-packages/pyVmomi/VmomiSupport.py", line 583, in <lambda> self.f(*(self.args + (obj,) + args), **kwargs) File "/usr/lib/vmware/site-packages/pyVmomi/VmomiSupport.py", line 373, in _InvokeMethod return self._stub.InvokeMethod(self, info, args) File "/usr/lib/vmware/site-packages/pyVmomi/SoapAdapter.py", line 1570, in InvokeMethod raise obj # pylint: disable-msg=E0702 pyVmomi.VmomiSupport.vmodl.fault.SecurityError: (vmodl.fault.SecurityError) { dynamicType = <unset>, dynamicProperty = (vmodl.DynamicProperty) [], msg = '', faultCause = <unset>, faultMessage = (vmodl.LocalizableMessage) [] } During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/lib/vmidentity/firstboot/vmidentity-firstboot.py", line 1861, in main vmidentityFB.boot() File "/usr/lib/vmidentity/firstboot/vmidentity-firstboot.py", line 369, in boot self.registerTokenServiceWithLookupService() File "/usr/lib/vmidentity/firstboot/vmidentity-firstboot.py", line 656, in registerTokenServiceWithLookupService raise e File "/usr/lib/vmidentity/firstboot/vmidentity-firstboot.py", line 652, in registerTokenServiceWithLookupService dynVars=dynVars) File "/usr/lib/vmware-cm/bin/cloudvmcisreg.py", line 710, in cloudvm_sso_cm_register serviceId = do_lsauthz_operation(cisreg_opts_dict) File "/usr/lib/vmware/site-packages/cis/cisreglib.py", line 1118, in do_lsauthz_operation ls_obj.register_service(svc_id, svc_create_spec) File "/usr/lib/vmware/site-packages/cis/cisreglib.py", line 348, in add_securityctx_to_requests return req_method(self, *args, **kargs) File "/usr/lib/python3.7/contextlib.py", line 161, in __exit__ raise RuntimeError("generator didn't stop after throw()") RuntimeError: generator didn't stop after throw()

Resolution

This is an unrecoverable error, please retry install. If you encounter this error again, please search for these symptoms in the VMware Knowledge Base for any known issues and possible resolutions. If none can be found, collect a support bundle and open a support request

Lukas_Ro_0-1616411601553.png

Do you have any idea?

 

Labels (4)
Reply
0 Kudos
1 Solution

Accepted Solutions
Ajay1988
Expert
Expert
Jump to solution

Looks like u have sso admin user missing from  SSO Administrators group.

Ajay1988_0-1616516024990.png

 

If you think your queries have been answered
Mark this response as "Correct" or "Helpful".

Regards,
AJ

View solution in original post

24 Replies
Ajay1988
Expert
Expert
Jump to solution

Can u please try to go  for   7.0 update 1d (7.0.1.00300) ?

If you think your queries have been answered
Mark this response as "Correct" or "Helpful".

Regards,
AJ
Reply
0 Kudos
Lukas_Ro
Contributor
Contributor
Jump to solution

Hi Ajay,

with installer 7.0 update 1d got same result:

Encountered an internal error. Traceback (most recent call last): File "/usr/lib/vmware/site-packages/cis/cisreglib.py", line 249, in securityctx_modifier yield File "/usr/lib/vmware/site-packages/cis/cisreglib.py", line 348, in add_securityctx_to_requests return req_method(self, *args, **kargs) File "/usr/lib/vmware/site-packages/cis/cisreglib.py", line 358, in register_service svc_create_spec) File "/usr/lib/vmware/site-packages/pyVmomi/VmomiSupport.py", line 592, in <lambda> self.f(*(self.args + (obj,) + args), **kwargs) File "/usr/lib/vmware/site-packages/pyVmomi/VmomiSupport.py", line 381, in _InvokeMethod return self._stub.InvokeMethod(self, info, args) File "/usr/lib/vmware/site-packages/pyVmomi/SoapAdapter.py", line 1608, in InvokeMethod raise obj # pylint: disable-msg=E0702 pyVmomi.VmomiSupport.vmodl.fault.SecurityError: (vmodl.fault.SecurityError) { dynamicType = <unset>, dynamicProperty = (vmodl.DynamicProperty) [], msg = '', faultCause = <unset>, faultMessage = (vmodl.LocalizableMessage) [] } During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/lib/vmidentity/firstboot/vmidentity-firstboot.py", line 1843, in main vmidentityFB.boot() File "/usr/lib/vmidentity/firstboot/vmidentity-firstboot.py", line 372, in boot self.registerTokenServiceWithLookupService() File "/usr/lib/vmidentity/firstboot/vmidentity-firstboot.py", line 659, in registerTokenServiceWithLookupService raise e File "/usr/lib/vmidentity/firstboot/vmidentity-firstboot.py", line 655, in registerTokenServiceWithLookupService dynVars=dynVars) File "/usr/lib/vmware-cm/bin/cloudvmcisreg.py", line 710, in cloudvm_sso_cm_register serviceId = do_lsauthz_operation(cisreg_opts_dict) File "/usr/lib/vmware/site-packages/cis/cisreglib.py", line 1116, in do_lsauthz_operation ls_obj.register_service(svc_id, svc_create_spec) File "/usr/lib/vmware/site-packages/cis/cisreglib.py", line 348, in add_securityctx_to_requests return req_method(self, *args, **kargs) File "/usr/lib/python3.7/contextlib.py", line 161, in __exit__ raise RuntimeError("generator didn't stop after throw()") RuntimeError: generator didn't stop after throw()

Lukas_Ro_0-1616498321275.png

 

Reply
0 Kudos
Ajay1988
Expert
Expert
Jump to solution

Hmm. I would suggest to open a case with VMware Support and upload the  failure log bundle. 

Share the SR number if possible or upload the log bundle here or attach a link to download the bundle

If you think your queries have been answered
Mark this response as "Correct" or "Helpful".

Regards,
AJ
Reply
0 Kudos
Lukas_Ro
Contributor
Contributor
Jump to solution

We bought licenses from HPE, so i must contact HPE support. When I have more informations, i will put it here.

Log bundle is uploaded

Reply
0 Kudos
Ajay1988
Expert
Expert
Jump to solution

Looks like u have sso admin user missing from  SSO Administrators group.

Ajay1988_0-1616516024990.png

 

If you think your queries have been answered
Mark this response as "Correct" or "Helpful".

Regards,
AJ
Lukas_Ro
Contributor
Contributor
Jump to solution

Hi Ajay,

I don't understand how it happened. But after re-adding Administartor@vshpere.local to SSO Administrators group everything works. Thank you very much for your help!

Lukas_Ro_0-1616534605691.png

 

Lukas

 

vjrk83
Enthusiast
Enthusiast
Jump to solution

I have the same issue , but I see the SSO administrator is part of the Administrator Group and also DNS / NTP is correct, but still see the issues. Any thoughts ? 

 

 

Reply
0 Kudos
vjrk83
Enthusiast
Enthusiast
Jump to solution

I see the same issue. Have raised the VMware support case - 21207589703  and attached the logs and bundles. Would you be able to look at that ? 

Reply
0 Kudos
Ajay1988
Expert
Expert
Jump to solution

You have a different issue altogether .

Error : ""2021-03-25T03:20:55.832Z Failed to Reregister STS with Lookup Service."

raise Exception('Unable to find the sso endpoint for reregistering') Exception: Unable to find the sso endpoint for reregistering

Seems you have an endpoint issue here. Please collect source vcsa bundle and ldif(https://kb.vmware.com/s/article/2146046) and upload to SR .

If you think your queries have been answered
Mark this response as "Correct" or "Helpful".

Regards,
AJ
Reply
0 Kudos
vjrk83
Enthusiast
Enthusiast
Jump to solution

Hello Ajay, Thanks for your reply. I've uploaded the support bundle from source vcsa to the SR. Also the vcsa is added to only AD as windows integrated authentication and not over LDAP. I suspect the name mismatch since the FQDN is different from the Primary network Identifier. Would it be a suggested way to remove the VCSA from AD upgrade and then add back to AD on the new VCSA on 7.0 ? Please let me know your thoughts . Thanks. 

 

- Vijay 

Reply
0 Kudos
Ajay1988
Expert
Expert
Jump to solution

""FQDN is different from the Primary network Identifier"" . This was  exactly what I suspected from upgrade logs but wanted to be sure .  Make sure the PNID and FQDN is same before upgrade. Take cold snapshot > Do changes and then  take another cold snapshot and upgrade.

https://blogs.vmware.com/vsphere/2019/08/changing-your-vcenter-servers-fqdn.html

If you think your queries have been answered
Mark this response as "Correct" or "Helpful".

Regards,
AJ
vjrk83
Enthusiast
Enthusiast
Jump to solution

Hi Ajay, 

I wasn't able to update the FQDN , since it doesnt allow to update the FQDN with DNS matching the IP. Instead, I could update the PNID  to match the FQDN. 

/usr/lib/vmware-vmafd/bin/vmafd-cli set-pnid --server-name localhost --pnid <FQDN name> 

I did that before upgrade, but it was failing at the time of starting the vmware secure token services with error "Unable to find the sso endpoint for reregistering" . My question was would it harm to Leave the VCSA from AD , upgrade and Add to AD on new vcsa to overcome this mismatch ? I still need to update PNID to match anyways. 

Reply
0 Kudos
Ajay1988
Expert
Expert
Jump to solution

"Leave VCSA from AD" >> Doesn't matter here.   Doing so will cause no harm and u can add that later. 

Ur issue is more on the  FQDN and PNID and that needs to be fixed prior upgrade. Please note that PNID cmd shared is not supported anymore and if you need to change PNID then I already shared the doc with you last .  Let Support team check and help as u already have the SR

If you think your queries have been answered
Mark this response as "Correct" or "Helpful".

Regards,
AJ
Reply
0 Kudos
vjrk83
Enthusiast
Enthusiast
Jump to solution

The issue is fixed. I had to remove the 2nd NIC DNS entry and updated the hostname to match the primary network identifier and also reverted the PNID to original name to match the SSO endpoints. Everything else was fine and after upgrade put the chnages back . However the NIC configurations has to be done manually during and after the upgrade like updating the gateway, making sure the right VLAN Is set on the new VCSA etc. 

 

Thanks for your help. 

Reply
0 Kudos
Ajay1988
Expert
Expert
Jump to solution

Cool. Having second NIC is not supported for VCSA  6.x . Not following the docs will introduce new issues

If you think your queries have been answered
Mark this response as "Correct" or "Helpful".

Regards,
AJ
Reply
0 Kudos
iliketurbos
Enthusiast
Enthusiast
Jump to solution

Ajay1988 you hit the nail right on the head, we had sts and dns mismatch had to lsdoctor and , after correcting still getting issue above from administrator@vsphere.local no longer in administrators group, corrected and good to go. Thank you!!

TirTul
Contributor
Contributor
Jump to solution

@Ajay1988 We have a similar issue during our upgrade. SR 22351970308

 

 

Reply
0 Kudos
Ajay1988
Expert
Expert
Jump to solution

You have a different issue and I see work is in progress via SR.

If you think your queries have been answered
Mark this response as "Correct" or "Helpful".

Regards,
AJ
TirTul
Contributor
Contributor
Jump to solution

Thanks. @Ajay1988 Communication from support/engineering has been poor.

 

I used jxplorer and determined that the service registrations are missing from a couple of groups. I removed the guid the refers to the vCSA in question. Would these cause a upgrade to fail?

"CN=vpxd-extension-*****,CN=ServicePrincipals,DC=vsphere,DC=local" is missing from LicenseService.Administrators and ServiceProviderUsers

"CN=machine-*******,CN=ServicePrincipals,DC=vsphere,DC=local" is missing from Administrators and SyncUsers

"CN=vsphere-webclient-*****,CN=ServicePrincipals,DC=vsphere,DC=local" is missing from vSphereClientSolutionUsers

Reply
0 Kudos