VMware Cloud Community
EmirKanioz
Contributor
Contributor

Regenerating STS Signing Certificate and Other Certificates

We used VMware vCenter Server version 7.0.2. We regenerated STS Signing Certificate via using fixsts.sh script as showed in (https://kb.vmware.com/s/article/76719) article. Then we followed that article (https://kb.vmware.com/s/article/2112283) for regeneration of VMCA Root Certificate, Machine Certificate and other kind of certificates. We tried option 8, then option 4 in /usr/lib/vmware-vmca/bin/certificate-manager location. Also, we tried both options seperately to get different results. After we tried to reach our vSphere Client via using web browser, we can not access. After that, we checked services that worked in that time, we faced some of services can not open even if we wait 2-3 hours more. Could you help us to solve this regeneration and services process before it is expire?

10 Replies
EmirKanioz
Contributor
Contributor

If you have any question, we will explain it in a detail way. Could you please help us to solve this regeneration process?

Reply
0 Kudos
a_p_
Leadership
Leadership

Did the STS certificate re-generation throw an error? Please provide error details (the script's outout).
If the re-generation succeeded, did you perform the required service stop/start (see KB article) as the final step?

André

 

Reply
0 Kudos
EmirKanioz
Contributor
Contributor

After I ran fixsts script, I can not take any error and I restart services via service-control --stop --all & service-control --start --all. Then I ran option 8 or option 4 for updating other certificates such as Machine SSL and Solution User according to this https://kb.vmware.com/s/article/2097936 article. After that I got below errors when the some services don' t get up. 

Service-control failed. Error: Failed to start services in profile ALL. RC=1, stderr=Failed to start sps, content-library, vstats, vpxd, updatemgr, vsan-health, wcp services. Error: Operation timed out

Error: The same certificate cannot be used by multiple services [wcp-44265dc0-104e-11ea-8e23-000c29ef723e].

2023-11-08T13:38:27.654Z ERROR certificate-manager Error: The same certificate cannot be used by multiple services [wcp-44265dc0-104e-11ea-8e23-000c29ef723e].

Error while performing Cert Replacement operation, please see /var/log/vmware/vmcad/certificate-manager.log for more information.

Reply
0 Kudos
a_p_
Leadership
Leadership

Let's g with one step aftre the other.
Since the certificates have not expired yet, I'd suggest the following steps.

  1. Cleanup the backup certificate store (https://kb.vmware.com/s/article/82560)
  2. Run the checksts script (https://kb.vmware.com/s/article/79248) to see whether the previous try to renew the STS certificate was successful
  3. Run the fixsts script (https://kb.vmware.com/s/article/76719) only if necessary, i.e. if your previous attempt did not work.
  4. In case you need to renew the STS certificate, restart the services after doing this and run checksts to verify the results.
  5. Once the STS certificate has successfully been replaced, recreate the remaining certificates using option the certificate-manager (https://kb.vmware.com/s/article/2097936).

If that solves the issue, make sure that you run the cleanup of the backup store once again to avoid messages regarding expiring certificates in the GUI.

André

Reply
0 Kudos
EmirKanioz
Contributor
Contributor

Thanks for your instructions. I will apply this steps on Thursday.

I can renew the STS certificate via using (https://kb.vmware.com/s/article/76719) the article. I do not take any error after the STS certificate regeneration.

On the other hand, I tried option 4 and 8 in the certificate-manager for updating Machine and User Solution certificates, but it did not work and try to reset services in the vSphere. During the services getting up, some required services did not get up. That's why I need to come back to snapshot every time. 

 

My question is, would you recommend any other option/way for after trying the 5th step which you mentioned about? Also would you recommend any option in certificate-manager?

Certificates Expire Date In CLI (1).png

Reply
0 Kudos
EmirKanioz
Contributor
Contributor

I applied the steps which you mentioned above. I updated certificates but I realized that same services did not get up after doing step 5 (https://kb.vmware.com/s/article/2097936). 

I faced with "rollback after certificate-manager options" attachment after using option 3. After that, I tried option 8 and faced with "option 8 to fail" attachment. Then I checked the services and I got "services after certificate generation options" attachment.

Then I tried to enter vSphere Client and vCenter Server Management on web, but I took 2 different errors. One of them exists in 1vCenter error" and the other one is "500 error" attachment.

I shared with you my current certification situation. I took snapshot before the 5th step for come back in the future. Right now I back to the snapshot and I can access the vSphere Client and vCenter Server Management, but I can not update the other certificates which I shared you below.

EmirKanioz_0-1700174915394.pngEmirKanioz_1-1700175123717.png

 

Reply
0 Kudos
navina
Enthusiast
Enthusiast

The issue is not with STS

"Error: The same certificate cannot be used by multiple services [wcp-44265dc0-104e-11ea-8e23-000c29ef723e]."

Use https://kb.vmware.com/s/article/90561?lang=en_US

 

Regards,
Navin A
Reply
0 Kudos
EmirKanioz
Contributor
Contributor

Since we came back to snapshot, we do not get this errors right now. Our current condition we regenerated the STS certificate and cleaned the STORE BACKUP_STORE. Also we can not generate other certificates like the one below. Would you recommend us to apply this https://kb.vmware.com/s/article/90561?lang=en_US kb after applying options in certificate-manager? 

If you have a different road map/steps for following we would like to try it. 

EmirKanioz_2-1700549716480.png

EmirKanioz_0-1700549440390.png

 

Reply
0 Kudos
navina
Enthusiast
Enthusiast

The https://kb.vmware.com/s/article/90561?lang=en_US is an alternative to certificate-manager

Regards,
Navin A
Reply
0 Kudos
EmirKanioz
Contributor
Contributor

We ran "python fixcerts.py replace --certType machinessl" to replace Machine SSL Certificate and we successfully replaced it.

Then we need to update Solution User certificates, and we ran "python fixcerts.py replace --certType solutionusers" to replace remaining certificates. However, we faced with fail. I shared two screenshots of it. In one of them it replaced certificates but took fails. The other one I shared the detailed information of that error.

At the end of the day I need to run our services and system so I back to the snapshot, and "current status of certificates in CLI" screenshot is the current condition of my environment.

Could we have a chance to update Solution User Certificates one by one?

I have only 4 days to replace it. Please could you help me to solve this issue. If you want me to additional information please inform me I can answer as soon as possible.

 

Reply
0 Kudos