VMware Cloud Community
gjbrown
Enthusiast
Enthusiast
Jump to solution

upgrade 6.7u3 to 7.0 cert issues

attempting to upgrade my lab from 6.7u3.latest to 7.0.latest

new VCSA VM deploys ok, but during pre-check get the following error:

Error

A vCenter Single Sign-On endpoint certificate validation error has occurred.

Resolution

Ensure that the endpoint service registrations in vmdir match their corrsponding machine SSL certificates in VECS. For more information, see Knowledge Base article KB 2121701.

I have already gone through the KB to no avail.  I have also gone through and reset all certs (cert manager option 8).

Anyone have any guidance or suggestions?

Thanks,

-GB

1 Solution

Accepted Solutions
sudeshnas
VMware Employee
VMware Employee
Jump to solution

Hi gjbrown,

I have attached a script here.

Please download the script and run it on the source machine to fix any ssl trust mismatch in lookup service registrations.

Please take a snapshot before proceeding.

Copy the file to lstool scripts folder.

For vCSA path:

# /usr/lib/vmidentity/tools/scripts

Run the below commands:

# python ls_ssltrust_fixer.py -f scan

#python ls_ssltrust_fixer.py -f fix

Then try running the upgrade.

Note: Make sure you take necessary backup/snapshot. Please try this ls_ssltrust_fixer.py in test environment, do not try this in production environment. Please raise a support request to validate before executing this script in production environment.

Regards,

Sudeshna Sarkar

Install-Upgrade Specialist

_______________________________________________________________________________________________________

"Did you find this helpful? Let us know by completing this survey (takes 1 minute!)"

View solution in original post

31 Replies
Alex_Romeo
Leadership
Leadership
Jump to solution

Hi,

VMware Knowledge Base

ARomeo

Blog: https://www.aleadmin.it/
Reply
0 Kudos
gjbrown
Enthusiast
Enthusiast
Jump to solution

thanks for the suggestion, but tried that as well.

GB

Reply
0 Kudos
scott28tt
VMware Employee
VMware Employee
Jump to solution

Moderator: Thread moved to the vSphere Upgrade & Install area.


-------------------------------------------------------------------------------------------------------------------------------------------------------------

Although I am a VMware employee I contribute to VMware Communities voluntarily (ie. not in any official capacity)
VMware Training & Certification blog
harry89
Enthusiast
Enthusiast
Jump to solution

This issue mostly occurs if the SSL trust of the services registered on PSC are having different than the SSL certificate of the node (of which the services is registered).

Please follow steps of the below article

VMware Knowledge Base

you have to basically get the old thumbprint and update the services with ls update cert  script using the new SSL certificate which is currently present

This command will give you all the services registered along with SSL trust they have .

/usr/lib/vmidentity/tools/scripts/lstool.py list --url https://localhost/lookupservice/sdk --no-check-cert --ep-type com.vmware.cis.cs.identity.sso 2>/dev/null

*Please mark the answer as correct if it solves your query

Harry
VCIX-DCV6.5 ,VCIX-NV6 , VCAP-CMA7
Mark answer as correct/helpful if it solves your query
Reply
0 Kudos
gjbrown
Enthusiast
Enthusiast
Jump to solution

Thanks harry89​, I went through the KB no errors, replaced 3 certificates but still the same issue when I attempt to upgrade.

-GB

Reply
0 Kudos
harry89
Enthusiast
Enthusiast
Jump to solution

Can u send the log snippet

Harry
VCIX-DCV6.5 ,VCIX-NV6 , VCAP-CMA7
Mark answer as correct/helpful if it solves your query
Reply
0 Kudos
gjbrown
Enthusiast
Enthusiast
Jump to solution

harry89​ which log snip you want? the log bundle compressed is 16mb and I am sure you don't want to deal with all of it.

Thx

Reply
0 Kudos
sudeshnas
VMware Employee
VMware Employee
Jump to solution

Hi gjbrown,

You can run the following command to check if the certificates of the existing environment is fine and valid or not .

#for i in $(/usr/lib/vmware-vmafd/bin/vecs-cli store list); do echo $i; /usr/lib/vmware-vmafd/bin/vecs-cli entry list --store $i --text | grep -i "not after"; done;

If the certs are fine and you continue to face the same issue please go ahead and replace the certificates using the option 8 in the certificate-manager tool.

VMware Knowledge Base

Then continue with the upgrade again.

It still you run into any issue please open a support request with us.

Regards,

Sudeshna Sarkar

Install-Upgrade Specialist

Reply
0 Kudos
gjbrown
Enthusiast
Enthusiast
Jump to solution

Hi sudeshnas

When I ran the command you provided it only returned back to a prompt with no output.  Not sure if that is good or bad.

I ran through cert replacement, option 8 again, even though I have done already.

Updated 5 service(s)

Status : 60% Completed [Reset vpxd-extension Cert...]

2020-07-22T15:14:46.910Z  Updating certificate for "com.vmware.imagebuilder" extension

Reset status : 100% Completed [Reset completed successfully]

--obviously this is good.

but upgrade still fails

Reply
0 Kudos
sudeshnas
VMware Employee
VMware Employee
Jump to solution

Hi gjbrown,

I have attached a script here.

Please download the script and run it on the source machine to fix any ssl trust mismatch in lookup service registrations.

Please take a snapshot before proceeding.

Copy the file to lstool scripts folder.

For vCSA path:

# /usr/lib/vmidentity/tools/scripts

Run the below commands:

# python ls_ssltrust_fixer.py -f scan

#python ls_ssltrust_fixer.py -f fix

Then try running the upgrade.

Note: Make sure you take necessary backup/snapshot. Please try this ls_ssltrust_fixer.py in test environment, do not try this in production environment. Please raise a support request to validate before executing this script in production environment.

Regards,

Sudeshna Sarkar

Install-Upgrade Specialist

_______________________________________________________________________________________________________

"Did you find this helpful? Let us know by completing this survey (takes 1 minute!)"

harry89
Enthusiast
Enthusiast
Jump to solution

There are possibilities that when u ran the reset all the certificates  , some of the endpoints are still having the older machine SSL cert as ssl trust .

This is fairly common occurrence .

But was this done before starting the upgrade or after . (reset all certificates).

If this was done to try to mitigate the issue and solve the upgrade problem , then not sure if this right direction because we need to be sure that prior to upgrade some cert in vecs-cli was surely expired and that was machine ssl .

Harry
VCIX-DCV6.5 ,VCIX-NV6 , VCAP-CMA7
Mark answer as correct/helpful if it solves your query
Reply
0 Kudos
gjbrown
Enthusiast
Enthusiast
Jump to solution

Hi sudeshnas

The script worked and found 31 mismatches. I ran the fix which let me run the upgrade but failed @ error#2, 89%.  Here is the error

Error

WCP service installation failed : Traceback (most recent call last): File "/usr/lib/vmware-wcp/firstboot/wcp-firstboot.py", line 50, in proxy return func(*args, **kwargs) File "/usr/lib/vmware-wcp/firstboot/wcp-firstboot.py", line 71, in configure wcpconfigure.configure_service() File "/usr/lib/vmware-wcp/py-modules/wcpconfigure.py", line 442, in configure_service create_storage_identity() File "/usr/lib/vmware-wcp/py-modules/wcpconfigure.py", line 438, in create_storage_identity SsoUser(_STORAGE_USER).create_storage_user_and_assign() File "/usr/lib/vmware-wcp/py-modules/wcpconfigure.py", line 330, in create_storage_user_and_assign self._create_storage_user() File "/usr/lib/vmware-wcp/py-modules/wcpconfigure.py", line 298, in _create_storage_user password = svcacctmgmt_client.create_svc_account(self._user_name) File "/usr/lib/vmware-wcp/py-modules/svcacctmgmt.py", line 90, in create_svc_account raise er File "/usr/lib/vmware-wcp/py-modules/svcacctmgmt.py", line 84, in create_svc_account svcacct_pwd_out = svcacct_client.create(create_spec) File "/usr/lib/vmware-wcp/py-modules/vapi-bindings/com/vmware/vcenter/svcaccountmgmt_client.py", line 368, in create 'create_spec': create_spec, File "/usr/lib/vmware-vapi/lib/python/vapi_runtime-2.100.0-py2.py3-none-any.whl/vmware/vapi/bindings/stub.py", line 345, in _invoke return self._api_interface.native_invoke(ctx, _method_name, kwargs) File "/usr/lib/vmware-vapi/lib/python/vapi_runtime-2.100.0-py2.py3-none-any.whl/vmware/vapi/bindings/stub.py", line 298, in native_invoke self._rest_converter_mode) com.vmware.vapi.std.errors_client.InternalServerError: {messages : [LocalizableMessage(id='com.vmware.vapi.authorization.permission.error', default_message='Could not validate permission information for operation com.vmware.vcenter.svcaccountmgmt.service_account.create invocation.', args=['com.vmware.vcenter.svcaccountmgmt.service_account.create'], params=None, localized=None)], data : None, error_type : None}

Resolution

This is an unrecoverable error, please retry install. If you encounter this error again, please search for these symptoms in the VMware Knowledge Base for any known issues and possible resolutions. If none can be found, collect a support bundle and open a support request.

I do have SR 20142056507 open, but just getting started if you would like to review any logs.

Thank you for the help with this.

Reply
0 Kudos
sudeshnas
VMware Employee
VMware Employee
Jump to solution

Hi gjbrown,

Thank you for opening a ticket with us.

I have gone through the logs and the errors/backtrace reported.

Well upon researching I see that similar issue has been reported by the other customer too and currently we are working internally to get it fixed.

You will receive all the updates on the ticket.

Regards,

Sudeshna Sarkar

Install-Upgrade Specialist

Reply
0 Kudos
gjbrown
Enthusiast
Enthusiast
Jump to solution

sudeshnas​  Thanks for digging into this.  I'll see what GSS says via ticket.  I'll update this thread with info to guide others towards a KB or solution.

Again thanks for the help and time with this.

-GB

Reply
0 Kudos
ErichWeihrauch
Contributor
Contributor
Jump to solution

sudeshnas, that script worked perfectly for me, thank you!

I had some invalid cert, that not even regenerating and resetting existing certs worked to resolve.

Gave your script a shot, and bam!

Reply
0 Kudos
vt-vmwaresjo
Contributor
Contributor
Jump to solution

Perfect, worked for me - Thanks

Reply
0 Kudos
EdwinMenard
Contributor
Contributor
Jump to solution

Hi,

I have the same error : WCP service installation failed.

Where can i find the solution for this problem ?

Thanks for help.

Reply
0 Kudos
AleksejsV
Contributor
Contributor
Jump to solution

Hello,

 

your script returns the following error:

root@vcenter [ /usr/lib/vmidentity/tools/scripts ]# python ls_ssltrust_fixer_p3.py -f fix
Running function 'fix'
Fix phase 1: Reading IDs with incorrect certificate from scan results
Using mismatch ID list from: /var/log/ls_ssltrust_fixer/mismatchIDs
SSO administrator user (Default:Administrator@vsphere.local):administrator@vsphere.local
Traceback (most recent call last):
File "ls_ssltrust_fixer_p3.py", line 368, in <module>
main()
File "ls_ssltrust_fixer_p3.py", line 360, in main
_doFix()
File "ls_ssltrust_fixer_p3.py", line 297, in _doFix
user=input("SSO administrator user (Default:Administrator@vsphere.local):") or "Administrator@vsphere.local"
File "<string>", line 1
administrator@vsphere.local
^
SyntaxError: invalid syntax

Reply
0 Kudos
nclifton
Contributor
Contributor
Jump to solution

I ran into the same issue.. Adding " " around the user name worked for me.  i.e. "administrator@vsphere.local"