Trying to follow KB: 2118939 - Replacing the Lookup Service SSL certificate on a Platform Services Controller 6.0 - ls_update_certs.py - FAILURE

[Samsonite801]

EDIT: Posted wrong KB in subject line and below (corrected KB the link shown below, was not able to edit Subject field above).

I am attempting to follow KB 2109074 - VMware KB: vCenter Server or Platform Services Controller certificate validation error for external ...

My steps relate to the 2 KB's linked to in the main KB article: 2109074

Everything goes fine in preparation to run the final command like getting the old certificate fingerprint, getting the new certificate file, etc.

When attempting to run the actual command in Windows (the most successful version of the command I am running as follows):

"C:\Program Files\VMware\vCenter Server\python\python.exe" ls_update_certs.py --url https://vcenter.domain.local/lookupservice/sdk --fingerprint b1:35:c1:9c:a5:59:dd:ab:3d:c2:50:e7:92:79:82:f0:b6:85:7d:c8 --certfile C:\certificates\new_machine.crt --user Administrator@vsphere.local --password "Passw0rd&"

BTW, the VMware KB says:

"Note: On Windows systems, enclose the password in double quotes."

I get is this error (fails on get-site-id) :

----------------------------------------------------------------------------------------------------------

Traceback (most recent call last):

  File "ls_update_certs.py", line 19, in <module>

    args.password)

  File "C:\Program Files\VMware\vCenter Server\VMware Identity Services\lstool\s

cripts\lstoolutil.py", line 79, in modify_svc_ep_certs

    raise Exception("'lstool get-site-id' failed: %d" % rc)

Exception: 'lstool get-site-id' failed: 1

----------------------------------------------------------------------------------------------------------

I've tried this on 2 different vCenter servers (both at 6.0u2) and get the same behavior, I've tried every combo of passwords for PSC/SSO like Passw0rd!   Pass-w0rd    P@ssw0rd  W34df*fdc4  ..etc, and tried it with or without quotes (2 quotes, 4 quotes), tried bash escape after password like: \   ...and nothing works. I'm not even sure if this is a password issue. A few lines above I see things showing this:

----------------------------------------------------------------------------------------------------------

Caused by: com.vmware.vim.vmomi.core.exception.CertificateValidationException: Server certificate chain not verified

Caused by: javax.net.ssl.SSLPeerUnverifiedException: peer not authenticated at sun.security.ssl.SSLSessionImpl.getPeerCertificates(SSLSessionImpl.java:421) at com.vmware.vim.vmomi.client.http.impl.ThumbprintTrustManager$Hostname Verifier.verify(ThumbprintTrustManager.java:296)

----------------------------------------------------------------------------------------------------------

However, I can successfully run this command (which doesn't seem to require a password):

"C:\Program Files\VMware\vCenter Server\python\python.exe" "C:\Program Files\VMware\vCenter Server\VMware Identity Services\lstool\scripts\lstool.py" get-site-id --url https://vcenter.domain.local/lookupservice/sdk"

..It works fine and shows the SSO name of default-sso-site.

All other aspects of the signed certificate install have succeed except the Lookup Service which is causing my NSX Manager install to not connect back up to the Lookup Service..

I also am getting the error from the Web Client showing this:

"Error occurred while processing request. Check vSphere WebClient logs for details."     (Refer to KB: https://kb.vmware.com/kb/2129053 )   ...caused by same issue as well:

Anyone else see this behavior or know of any possible way to recover from this without having to re-install? How to debug the .py scripts better? Is there any better documentation from VMware on how these scripts work? Can the Lookup Service be re-installed?

Any assistance is greatly appreciated!

[Samsonite801]

Here's the full output from running the command (showing the errors). I only scrubbed the FQDN:

C:\Program Files\VMware\vCenter Server\python\python.exe" ls_update_certs.py --u

rl https://vcenter.domain.local/lookupservice/sdk --fingerprint a1:35:e1:9c:a4:59:d

d:cb:3d:c1:50:e7:82:78:81:f0:b6:85:7d:b8 --certfile C:\certificates\new_machine.

crt --user Administrator@vsphere.local --password "Passw0rd!"

Get site name

2016-04-07 09:35:44,077 INFO  com.vmware.vim.vmomi.core.types.impl.VmodlContextI

mpl$NonValidatingClassPathXmlApplicationContext - Refreshing com.vmware.vim.vmom

i.core.types.impl.VmodlContextImpl$NonValidatingClassPathXmlApplicationContext@7

52d2e33: startup date [Thu Apr 07 09:35:44 MDT 2016]; root of context hierarchy

2016-04-07 09:35:44,223 INFO  org.springframework.beans.factory.xml.XmlBeanDefin

itionReader - Loading XML bean definitions from class path resource [com/vmware/

vim/binding/vmodl/context_v2.xml]

2016-04-07 09:35:44,868 INFO  com.vmware.vim.vmomi.core.types.impl.VmodlContextI

mpl$NonValidatingClassPathXmlApplicationContext - Closing com.vmware.vim.vmomi.c

ore.types.impl.VmodlContextImpl$NonValidatingClassPathXmlApplicationContext@752d

2e33: startup date [Thu Apr 07 09:35:44 MDT 2016]; root of context hierarchy

2016-04-07 09:35:44,877 INFO  com.vmware.vim.vmomi.core.types.impl.VmodlContextI

mpl$NonValidatingClassPathXmlApplicationContext - Refreshing com.vmware.vim.vmom

i.core.types.impl.VmodlContextImpl$NonValidatingClassPathXmlApplicationContext@e

06234e: startup date [Thu Apr 07 09:35:44 MDT 2016]; root of context hierarchy

2016-04-07 09:35:44,879 INFO  org.springframework.beans.factory.xml.XmlBeanDefin

itionReader - Loading XML bean definitions from class path resource [com/vmware/

vim/binding/vmodl/context_v2.xml]

2016-04-07 09:35:44,931 INFO  com.vmware.vim.vmomi.core.types.impl.VmodlContextI

mpl$NonValidatingClassPathXmlApplicationContext - Closing com.vmware.vim.vmomi.c

ore.types.impl.VmodlContextImpl$NonValidatingClassPathXmlApplicationContext@e062

34e: startup date [Thu Apr 07 09:35:44 MDT 2016]; root of context hierarchy

2016-04-07 09:35:44,938 INFO  com.vmware.vim.vmomi.core.types.impl.VmodlContextI

mpl$NonValidatingClassPathXmlApplicationContext - Refreshing com.vmware.vim.vmom

i.core.types.impl.VmodlContextImpl$NonValidatingClassPathXmlApplicationContext@1

9306a99: startup date [Thu Apr 07 09:35:44 MDT 2016]; root of context hierarchy

2016-04-07 09:35:44,941 INFO  org.springframework.beans.factory.xml.XmlBeanDefin

itionReader - Loading XML bean definitions from class path resource [com/vmware/

vim/binding/lookup/context.xml]

2016-04-07 09:35:45,117 INFO  com.vmware.vim.vmomi.core.types.impl.VmodlContextI

mpl$NonValidatingClassPathXmlApplicationContext - Closing com.vmware.vim.vmomi.c

ore.types.impl.VmodlContextImpl$NonValidatingClassPathXmlApplicationContext@1930

6a99: startup date [Thu Apr 07 09:35:44 MDT 2016]; root of context hierarchy

2016-04-07 09:35:46,410 WARN  com.vmware.vim.vmomi.client.http.impl.HttpConfigur

ationCompilerBase$ConnectionMonitorThreadBase - Shutting down the connection mon

itor.

com.vmware.vim.vmomi.client.exception.SslException: com.vmware.vim.vmomi.core.ex

ception.CertificateValidationException: Server certificate chain not verified

        at com.vmware.vim.vmomi.client.common.impl.ResponseImpl.setError(Respons

eImpl.java:251)

        at com.vmware.vim.vmomi.client.http.impl.HttpExchange.run(HttpExchange.j

ava:54)

        at com.vmware.vim.vmomi.client.http.impl.HttpProtocolBindingBase.execute

Runnable(HttpProtocolBindingBase.java:186)

        at com.vmware.vim.vmomi.client.http.impl.HttpProtocolBindingImpl.send(Ht

tpProtocolBindingImpl.java:115)

        at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl$C

allExecutor.sendCall(MethodInvocationHandlerImpl.java:581)

        at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl$C

allExecutor.executeCall(MethodInvocationHandlerImpl.java:562)

        at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl.c

ompleteCall(MethodInvocationHandlerImpl.java:348)

        at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl.i

nvokeOperation(MethodInvocationHandlerImpl.java:308)

        at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl.i

nvoke(MethodInvocationHandlerImpl.java:182)

        at com.sun.proxy.$Proxy23.getSiteId(Unknown Source)

        at com.vmware.vim.lookup.client.tool.command.GetSiteIdCommand$1.execute(

GetSiteIdCommand.java:46)

        at com.vmware.vim.lookup.client.tool.command.GetSiteIdCommand$1.execute(

GetSiteIdCommand.java:43)

        at com.vmware.vim.lookup.client.tool.command.Command.callLsEx(Command.ja

va:183)

        at com.vmware.vim.lookup.client.tool.command.Command.callLs(Command.java

:154)

        at com.vmware.vim.lookup.client.tool.command.GetSiteIdCommand.execute(Ge

tSiteIdCommand.java:43)

        at com.vmware.vim.lookup.client.tool.LsTool.app(LsTool.java:67)

        at com.vmware.vim.lookup.client.tool.LsTool.main(LsTool.java:103)

Caused by: com.vmware.vim.vmomi.core.exception.CertificateValidationException: S

erver certificate chain not verified

        at com.vmware.vim.vmomi.client.http.impl.ThumbprintTrustManager$Hostname

Verifier.verify(ThumbprintTrustManager.java:308)

        at com.vmware.vim.vmomi.client.http.impl.VlsiSslSocketFactory.connectSoc

ket(VlsiSslSocketFactory.java:121)

        at org.apache.http.conn.ssl.SSLSocketFactory.connectSocket(SSLSocketFact

ory.java:401)

        at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnect

ion(DefaultClientConnectionOperator.java:177)

        at org.apache.http.impl.conn.AbstractPoolEntry.open(AbstractPoolEntry.ja

va:144)

        at org.apache.http.impl.conn.AbstractPooledConnAdapter.open(AbstractPool

edConnAdapter.java:131)

        at org.apache.http.impl.client.DefaultRequestDirector.tryConnect(Default

RequestDirector.java:611)

        at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultReq

uestDirector.java:446)

        at org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttp

Client.java:863)

        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttp

Client.java:82)

        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttp

Client.java:57)

        at com.vmware.vim.vmomi.client.http.impl.HttpExchange.run(HttpExchange.j

ava:48)

        ... 15 more

Caused by: javax.net.ssl.SSLPeerUnverifiedException: peer not authenticated

        at sun.security.ssl.SSLSessionImpl.getPeerCertificates(SSLSessionImpl.ja

va:421)

        at com.vmware.vim.vmomi.client.http.impl.ThumbprintTrustManager$Hostname

Verifier.verify(ThumbprintTrustManager.java:296)

        ... 26 more

Traceback (most recent call last):

  File "ls_update_certs.py", line 19, in <module>

    args.password)

  File "C:\Program Files\VMware\vCenter Server\VMware Identity Services\lstool\s

cripts\lstoolutil.py", line 79, in modify_svc_ep_certs

    raise Exception("'lstool get-site-id' failed: %d" % rc)

Exception: 'lstool get-site-id' failed: 1

C:\Program Files\VMware\vCenter Server\VMware Identity Services\lstool\scripts>

[Samsonite801]

So I may have made some progress...

The part I don't think I mentioned before, is that I was also having previous failure when trying to use the certificate-manager.bat method to install the signed machine certificate. So I instead had to find a 3rd-party tool to install the certs this last time (there was a time when the certificate-manager worked).

Supposedly vCenter 6.0 U2 is said to fix the issue with the Lookup Service 'certificate thumbprint registrations' not updating, in order for the external services to work. But they said it only works if you use the certificate-manager.bat method to update, but I was getting the 'error with rollback' messages using it. So when trying to follow these other KBs on how to use the ls_update_certs.py method to fix it I was also getting errors.

certificate-manager.log:

2016-04-07T19:08:02.522Z INFO certificate-manager MACHINE_SSL_CERT certificate replacement failed. SerialNumber and Thumbprint not changed after replacement, certificates are same before and after

2016-04-07T19:08:02.525Z ERROR certificate-manager Error while replacing Machine SSL Cert, please see C:\ProgramData\VMware\vCenterServer\logs\vmca\certificate-manager.log for more information.

2016-04-07T19:08:02.525Z ERROR certificate-manager 'exceptions.Exception' object has no attribute 'appendErrorStack'

2016-04-07T19:08:02.528Z INFO certificate-manager Performing rollback of Machine SSL Cert...

This article seems to provide some potential clue as to what I need to do:  VCSA 6.0: Replacing external-facing SSL certificates by CA signed certs

The only problem there is that my vCenter / PSC is running on Windows and the steps they are suggesting are for the vCSA appliance..

So I will need to research the correct conversions of the commands unless someone who reads this may have some experience with the procedure detailed there, or perhaps have enough background with both systems to provide some direction..

[Samsonite801]

No, I guess I spoke too soon..

My errors and their errors are caused by different things.. The last issue I posted was caused because I had been trying to replace the cert with the same cert so it couldn't change it..

The underlying issue in my environment is still the same error as in the first post:

ERROR certificate-manager 'lstool get-site-id' failed: 1

Same error on both tools the ls_update_certs.py (which invokes lstool.py) and the certificate-manager.bat

And the underlying error is still related to java:

com.vmware.vim.vmomi.client.exception.SslException: com.vmware.vim.vmomi.core.exception.CertificateValidationException: Server certificate chain not verified

Caused by: javax.net.ssl.SSLPeerUnverifiedException: peer not authenticated at sun.security.ssl.SSLSessionImpl.getPeerCertificates(SSLSessionImpl.java:421)

thus causing the error:

Exception: 'lstool get-site-id' failed: 1

I've even tried adding the certs into Java for both VMware Java cacert store and the machine install of Java cacert store and it doesn't help.

If I can't figure this out it would be a shame to have to re-install 2 vCenters in this cloud POC all for such a silly problem.. Man why are certificate issues always such a challenge? ..hehe

[Samsonite801 · Accepted solution]

Ahhhh-hah... I figured out the problem.

If I manually run the command:

"C:\Program Files\VMware\vCenter Server\python\python.exe" "C:\Program Files\VMware\vCenter Server\VMware Identity Services\lstool\scripts\lstool.py" list --url https://vcenter.domain.local/lookupservice/sdk"

Then I get good output.. Command works awesome.

But just as a hunch I had to check something out, I decided to see what happens when I run the same command with the '--no-check-cert' switch like as follows:

"C:\Program Files\VMware\vCenter Server\python\python.exe" "C:\Program Files\VMware\vCenter Server\VMware Identity Services\lstool\scripts\lstool.py" list --no-check-cert --url https://vcenter.domain.local/lookupservice/sdk"

...and guess what? I got the SAME java error as in all the other scripts: "peer not authenticated "

So then I went over to the VMware lstool scripts folder ( C:\Program Files\VMware\vCenter Server\VMware Identity Services\lstool\scripts\ 😞 and opened lstool.py with notepad and could see that it simply invokes another script in the same folder called: lstoolutil.py

I then opened that script lstoolutil.py in notepad and did a search for the string: --no-check-cert   ...and there were 5 instances where various commands were calling this switch. So I commented out ( # ) all 5 lines calling this switch, saved file, and re-ran the original script: ls_update_certs.py ...and WHAH-LAHH  Hooray!

Line 52:  #                                "--no-check-cert",

Line 74:  #                                "--no-check-cert",

Line 85:  #                                "--no-check-cert",

Line 121: #                                "--no-check-cert",

Line 139: #                                "--no-check-cert",

Then just to check I run again: "C:\Program Files\VMware\vCenter Server\python\python.exe" "C:\Program Files\VMware\vCenter Server\VMware Identity Services\lstool\scripts\lstool.py" list --url https://vcenter.domain.local/lookupservice/sdk"

...and can confirm that all of the service registrations show the 'SSL Trust' field using the new certificate key.

Problem solved.. Let this simply a a journal for others to follow in case they encounter the same issue...

[Samsonite801]

BTW, here was the KB to follow the steps to run ls_update_certs.py   to fix the service registration SSL Trust certs:

VMware KB: vCenter Server certificate validation error for external solutions in environments with E...

[Samsonite801]

Essentially the most part of that KB is so you can retrieve your old key Thumbprint, and get the new certificate file ready so you can ultimately run the following command:

"C:\Program Files\VMware\vCenter Server\python\python.exe" "C:\Program Files\VMware\vCenter Server\VMware Identity Services\lstool\scripts\ls_update_certs.py" --url https://vcenter.domain.local/lookupservice/sdk --fingerprint a535d17cb453dacb3dc170e7227481f0b6452db8 --certfile C:\certificates\new_machine.crt --user Administrator@vsphere.local --password "Passw0rd!"

..which is the script that actually swap the old SSL Trust key on all the service registrations so your external solutions like NSX Manager, SRM, etc can register again..

[Samsonite801]

A couple other things to mention here as well. It may or may not matter, but one step I performed before reaching the above solution was to add the certificate chain pem (mine was in form of root_signing_cert.cer  (if you open in text editor you should see all of the individual certs begin-end-begin-end-begin-end   etc, and copy that file to C:\  (that only what I did, and then I added those to the Java trusted cert store so that when I commented out those lines for --no-check-cert so that java wouldn't throw a fit:

"C:\Program Files (x86)\Java\jre1.8.0_73\bin\keytool.exe" -import -file "C:\root_signing_chain.cer" -keystore cacerts

"C:\Program Files (x86)\Java\jre1.8.0_73\bin\keytool.exe" -list -keystore cacerts

"C:\Program Files\VMware\vCenter Server\jre\bin\keytool.exe" -import -file "C:\root_signing_chain.cer" -keystore cacerts

"C:\Program Files\VMware\vCenter Server\jre\bin\keytool.exe" -list -keystore cacerts

Just as good measure I did one set for the machine installed java, and other set for the VMware installed java, but most likely only needs to be added to the VMware one really..

The last thing I had to do once I got NSX Manager to register again, was I had to fix the vpxd-extension certificate registry so the NSX VXLAN vib would install and communicate again.

Followed this article for that part: http://www.virtually-limitless.com/nsx/unable-to-deploy-nsx-vibs-after-updating-certificates-in-vsph...

[Samsonite801]

Well the good news is, that most of this is all fixed now..

After everything I've been through above, NSX Manager is back, the NSX Controller VIB's are communicating again, all SSL Trust Anchors are all showing updated in the Managed Object Browser, and all end-user browser certs are green.

But there is still one remnant left behind that I cannot seem to fix (which all the KB's say the steps I did were supposed to fix this, and vSphere 6.0u2, what I am running, is supposed to resolve these issues as well), but when logging into the Web Client I still get the error:

"Error occurred while processing request. Check vSphere WebClient logs for details."

And if you go to 'Administration > Customer Experience Improvement Program'  ...you can still see the same error shown in there and the 'Join' and 'Leave' buttons are still greyed out..

So as far as I know, this is the only annoyance left behind and I do not believe there are any otherwise functional problems other than this trace left behind.

Is this just a signature 'scar' that I have to live with every time I log into the Web Client, or is there any manual way to bump it so that the Customer Experience Improvement Program can connect again?

It's ironic that this piece is broken, because I couldn't even report back to VMware through the Customer Experience Improvement Program about how this is broken if I wanted to join hehe..

[ArildS]

Had a similar error. Turned out my password had a problematic first letter which didn't pass to pyton correctly.

Had to edit the script file:

D:\Program Files\VMware\vCenter Server\VMware Identity Services\lstool\scripts\lstoolutil.py

Lines 107 and 108:

From

   update55_ct = _modify_svc_eps(lsUrl, True, ids55, _update_svc_spec, username, password)

   update60_ct = _modify_svc_eps(lsUrl, False, ids60, _update_svc_spec, username, password)

To

   update55_ct = _modify_svc_eps(lsUrl, True, ids55, _update_svc_spec, username, 'MyStupidPassword')

   update60_ct = _modify_svc_eps(lsUrl, False, ids60, _update_svc_spec, username, 'MyStupidPassword')

Shouldn't hit too many out there, but nice to know.