1 2 Previous Next 17 Replies Latest reply on Sep 5, 2020 3:17 AM by EdwinMenard

    upgrade 6.7u3 to 7.0 cert issues

    gjbrown Novice

      attempting to upgrade my lab from 6.7u3.latest to 7.0.latest

      new VCSA VM deploys ok, but during pre-check get the following error:

       

      Error

      A vCenter Single Sign-On endpoint certificate validation error has occurred.

       

       

      Resolution

      Ensure that the endpoint service registrations in vmdir match their corrsponding machine SSL certificates in VECS. For more information, see Knowledge Base article KB 2121701.

       

       

       

       

       

      I have already gone through the KB to no avail.  I have also gone through and reset all certs (cert manager option 8).

       

      Anyone have any guidance or suggestions?

      Thanks,

      -GB

       

       

       

       

        • 2. Re: upgrade 6.7u3 to 7.0 cert issues
          gjbrown Novice

          thanks for the suggestion, but tried that as well.

          GB

          • 3. Re: upgrade 6.7u3 to 7.0 cert issues
            scott28tt Guru
            Community WarriorsVMware EmployeesUser Moderators

            Moderator: Thread moved to the vSphere Upgrade & Install area.

            • 4. Re: upgrade 6.7u3 to 7.0 cert issues
              harry89 Enthusiast

              This issue mostly occurs if the SSL trust of the services registered on PSC are having different than the SSL certificate of the node (of which the services is registered).

               

              Please follow steps of the below article

               

              VMware Knowledge Base

               

              you have to basically get the old thumbprint and update the services with ls update cert  script using the new SSL certificate which is currently present

               

              This command will give you all the services registered along with SSL trust they have .

              /usr/lib/vmidentity/tools/scripts/lstool.py list --url https://localhost/lookupservice/sdk --no-check-cert --ep-type com.vmware.cis.cs.identity.sso 2>/dev/null

               

               

               

              *Please mark the answer as correct if it solves your query

              • 5. Re: upgrade 6.7u3 to 7.0 cert issues
                gjbrown Novice

                Thanks harry89, I went through the KB no errors, replaced 3 certificates but still the same issue when I attempt to upgrade.

                 

                -GB

                • 6. Re: upgrade 6.7u3 to 7.0 cert issues
                  harry89 Enthusiast

                  Can u send the log snippet

                  • 7. Re: upgrade 6.7u3 to 7.0 cert issues
                    gjbrown Novice

                    harry89 which log snip you want? the log bundle compressed is 16mb and I am sure you don't want to deal with all of it.

                     

                    Thx

                    • 8. Re: upgrade 6.7u3 to 7.0 cert issues
                      sudeshnas Enthusiast
                      VMware Employees

                      Hi gjbrown,

                       

                      You can run the following command to check if the certificates of the existing environment is fine and valid or not .

                      #for i in $(/usr/lib/vmware-vmafd/bin/vecs-cli store list); do echo $i; /usr/lib/vmware-vmafd/bin/vecs-cli entry list --store $i --text | grep -i "not after"; done;

                      If the certs are fine and you continue to face the same issue please go ahead and replace the certificates using the option 8 in the certificate-manager tool.

                      VMware Knowledge Base

                      Then continue with the upgrade again.

                      It still you run into any issue please open a support request with us.

                       

                       

                      Regards,

                      Sudeshna Sarkar

                      Install-Upgrade Specialist

                       

                      • 9. Re: upgrade 6.7u3 to 7.0 cert issues
                        gjbrown Novice

                        Hi sudeshnas

                         

                        When I ran the command you provided it only returned back to a prompt with no output.  Not sure if that is good or bad.

                         

                         

                        I ran through cert replacement, option 8 again, even though I have done already.

                         

                        Updated 5 service(s)

                        Status : 60% Completed [Reset vpxd-extension Cert...]

                        2020-07-22T15:14:46.910Z  Updating certificate for "com.vmware.imagebuilder" extension

                         

                         

                        Reset status : 100% Completed [Reset completed successfully]

                        --obviously this is good.

                         

                        but upgrade still fails

                        • 10. Re: upgrade 6.7u3 to 7.0 cert issues
                          sudeshnas Enthusiast
                          VMware Employees

                          Hi gjbrown,

                           

                          I have attached a script here.

                          Please download the script and run it on the source machine to fix any ssl trust mismatch in lookup service registrations.

                          Please take a snapshot before proceeding.

                          Copy the file to lstool scripts folder.

                          For vCSA path:

                          # /usr/lib/vmidentity/tools/scripts

                          Run the below commands:

                          # python ls_ssltrust_fixer.py -f scan

                          #python ls_ssltrust_fixer.py -f fix

                           

                          Then try running the upgrade.

                           

                          Note: Make sure you take necessary backup/snapshot. Please try this ls_ssltrust_fixer.py in test environment, do not try this in production environment. Please raise a support request to validate before executing this script in production environment.

                           

                          Regards,

                          Sudeshna Sarkar

                          Install-Upgrade Specialist

                           

                           

                          _______________________________________________________________________________________________________

                          "Did you find this helpful? Let us know by completing this survey (takes 1 minute!)"

                          1 person found this helpful
                          • 11. Re: upgrade 6.7u3 to 7.0 cert issues
                            harry89 Enthusiast

                            There are possibilities that when u ran the reset all the certificates  , some of the endpoints are still having the older machine SSL cert as ssl trust .

                             

                            This is fairly common occurrence .

                             

                            But was this done before starting the upgrade or after . (reset all certificates).

                             

                            If this was done to try to mitigate the issue and solve the upgrade problem , then not sure if this right direction because we need to be sure that prior to upgrade some cert in vecs-cli was surely expired and that was machine ssl .

                            • 12. Re: upgrade 6.7u3 to 7.0 cert issues
                              gjbrown Novice

                              Hi sudeshnas

                              The script worked and found 31 mismatches. I ran the fix which let me run the upgrade but failed @ error#2, 89%.  Here is the error

                              Error

                               

                               

                              WCP service installation failed : Traceback (most recent call last): File "/usr/lib/vmware-wcp/firstboot/wcp-firstboot.py", line 50, in proxy return func(*args, **kwargs) File "/usr/lib/vmware-wcp/firstboot/wcp-firstboot.py", line 71, in configure wcpconfigure.configure_service() File "/usr/lib/vmware-wcp/py-modules/wcpconfigure.py", line 442, in configure_service create_storage_identity() File "/usr/lib/vmware-wcp/py-modules/wcpconfigure.py", line 438, in create_storage_identity SsoUser(_STORAGE_USER).create_storage_user_and_assign() File "/usr/lib/vmware-wcp/py-modules/wcpconfigure.py", line 330, in create_storage_user_and_assign self._create_storage_user() File "/usr/lib/vmware-wcp/py-modules/wcpconfigure.py", line 298, in _create_storage_user password = svcacctmgmt_client.create_svc_account(self._user_name) File "/usr/lib/vmware-wcp/py-modules/svcacctmgmt.py", line 90, in create_svc_account raise er File "/usr/lib/vmware-wcp/py-modules/svcacctmgmt.py", line 84, in create_svc_account svcacct_pwd_out = svcacct_client.create(create_spec) File "/usr/lib/vmware-wcp/py-modules/vapi-bindings/com/vmware/vcenter/svcaccountmgmt_client.py", line 368, in create 'create_spec': create_spec, File "/usr/lib/vmware-vapi/lib/python/vapi_runtime-2.100.0-py2.py3-none-any.whl/vmware/vapi/bindings/stub.py", line 345, in _invoke return self._api_interface.native_invoke(ctx, _method_name, kwargs) File "/usr/lib/vmware-vapi/lib/python/vapi_runtime-2.100.0-py2.py3-none-any.whl/vmware/vapi/bindings/stub.py", line 298, in native_invoke self._rest_converter_mode) com.vmware.vapi.std.errors_client.InternalServerError: {messages : [LocalizableMessage(id='com.vmware.vapi.authorization.permission.error', default_message='Could not validate permission information for operation com.vmware.vcenter.svcaccountmgmt.service_account.create invocation.', args=['com.vmware.vcenter.svcaccountmgmt.service_account.create'], params=None, localized=None)], data : None, error_type : None}

                              Resolution

                               

                               

                              This is an unrecoverable error, please retry install. If you encounter this error again, please search for these symptoms in the VMware Knowledge Base for any known issues and possible resolutions. If none can be found, collect a support bundle and open a support request.

                               

                              I do have SR 20142056507 open, but just getting started if you would like to review any logs.

                               

                              Thank you for the help with this.

                              • 13. Re: upgrade 6.7u3 to 7.0 cert issues
                                sudeshnas Enthusiast
                                VMware Employees

                                Hi gjbrown,

                                 

                                Thank you for opening a ticket with us.

                                I have gone through the logs and the errors/backtrace reported.

                                Well upon researching I see that similar issue has been reported by the other customer too and currently we are working internally to get it fixed.

                                You will receive all the updates on the ticket.

                                 

                                 

                                Regards,

                                Sudeshna Sarkar

                                Install-Upgrade Specialist

                                • 14. Re: upgrade 6.7u3 to 7.0 cert issues
                                  gjbrown Novice

                                  sudeshnas  Thanks for digging into this.  I'll see what GSS says via ticket.  I'll update this thread with info to guide others towards a KB or solution.

                                   

                                  Again thanks for the help and time with this.

                                   

                                  -GB

                                  1 2 Previous Next