NSX 6.3.5 Unable to deploy NSX controllers

[future2000]

Hi,

I have a new deployment of NSX 6.3.5. I cannot deploy any NSX controllers the OVA deployment shows the following error..

Operation failed on VC. For more details, refer to the rootCauseString or the VC logs

NSX Manager controller log shows the following...

Caused by: javax.net.ssl.SSLHandshakeException: java.security.cert.CertificateException: Server Certificate's thumbprint:CC:62:42:E1:9A:E0:40:E6:0A:67:C1:E9:12:FF:8C:A2:47:1D:B0:CFdoesn't match any of the Registered thumbprint Set:[06:26:65:80:AA:65:A7:83:C4:0C:C0:22:CB:45:1E:07:CD:02:BC:41]

        at sun.security.ssl.Alerts.getSSLException(Alerts.java:192)

        at sun.security.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1959)

        at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:302)

        at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:296)

        at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1514)

        at sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:216)

        at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1026)

        at sun.security.ssl.Handshaker.process_record(Handshaker.java:961)

        at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1072)

        at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1385)

        at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1413)

        at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1397)

        at sun.net.www.protocol.https.HttpsClient.afterConnect(HttpsClient.java:559)

        at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:185)

        at sun.net.www.protocol.http.HttpURLConnection.getOutputStream0(HttpURLConnection.java:1316)

        at sun.net.www.protocol.http.HttpURLConnection.getOutputStream(HttpURLConnection.java:1291)

        at sun.net.www.protocol.https.HttpsURLConnectionImpl.getOutputStream(HttpsURLConnectionImpl.java:250)

        at com.vmware.vshield.vsm.inventory.vcoperations.impl.ResourcePoolVcOperationsImpl.pushFile_aroundBody0(ResourcePoolVcOperationsImpl.java:152)

        ... 20 more

Important things to note...

• vCenter, PSC and all Machine SSL Certificates have been changed
• NSX Manager SSL Certificate has been changed.
• All SSL Certificates are signed by a Intermediate Windows 2012 R2 CA. All are trusted.
• SSO NTP source identical to NSX Manager time source. Both show accurate time.
• Those SSL Cert thumbprints in the error are not the thumbprints of either my vCenter or NSX manager.

Spent nearly a day on this and its driving me crazy. Anyone seen this?

Cheers

[future2000]

Ok, so the fix. This is interesting...

It turned out the SSL cert thumbprints the NSX manager was seeing were the old ESXi host SSL certs which I changed to signed certs a couple of days back. Since then I hadn't rebooted the hosts I had simply restarted hostd and vpxa. A restart still didn't fix the issues. It did do one thing though. A restart of ESXi 6.5U1 deleted the backups I had made in /etc/vmware/ssl/backups. The entire directory had gone! So I couldn't go back to the old certs to fix the issue.

So I simply disconnected the ESXi hosts from vCenter and reconnected them. Problem solved. NSX controllers deployed without issue. vCenter had been rebooted many times but for whatever reason it still though the ESXi hosts were connected with a different thumbprint, which must have been where NSX manager got the thumbprint from. We live and learn!

[tanurkov]

The fix is disconnect  ESXI form VC ----> when you do that password between VC and ESXi is changed and regenerated and NSX manager as informed about his change.

and then NSX manager start using/leverage new password generated between VC and ESXi.

Regards Dmitri 

[sachinsharma711]

It works for me NSX 6.4.5