I get the following error when configuring SSO in NSX
NSX Management Service operation failed.( Initialization of Admin Registration Service Provider failed. Root Cause: Error occurred while registration of lookup service, com.vmware.vim.sso.admin.exception.InternalError: javax.net.ssl.SSLHandshakeException: com.vmware.vim.vmomi.client.exception.VlsiCertificateException: Server certificate chain is not trusted and thumbprint doesn't match )
Is the time synchronized between Vcenter and Nsx Manager?
One other possibility is that the DNS forward and reverse entries exist and resolvable by both NSX Manger and Vcenter?
As canero mentioned, check time sync and NTP and DNS forward and reverse.
There are some other possibilities like connectivity issues. Check this KB for possible resolution steps: VMware Knowledge Base