Timeout on building connection between VSM and new deployed controller

[shobhitsinha]

While deploying the nsx controller I am getting "Timeout on building connection between VSM and new deployed controller" warning and then the controller is delete. Any idea why?

[tanurkov]

HI

please use show manager log follow on NSX Manager

there is reason for not connecting between NSX manager and NSX controller usualy ports should be 443 to NSX manager and 1234 to ESXi host where the controller is deploy.

Review logs and you will find the particular issue .

Regards Dmitri

[shobhitsinha]

Hi

I downloaded the nsx manager log and I came across this

2018-03-21 21:08:35.029 GMT  INFO http-nio-127.0.0.1-7441-exec-1 VcConnection$VimClient:1258 - Successfully created vimclient for uri:https://<my-vcsa-ip>/sdk/vimService

2018-03-21 21:08:35.152 GMT  INFO http-nio-127.0.0.1-7441-exec-1 VcConnection:645 - Session info : Session key [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] for User [VSPHERE.LOCAL\Administrator]]

2018-03-21 21:08:35.220 GMT  INFO http-nio-127.0.0.1-7441-exec-1 VcConnection:645 - Session info : Session key [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] for User [VSPHERE.LOCAL\Administrator]]

2018-03-21 21:08:35.221 GMT  INFO http-nio-127.0.0.1-7441-exec-1 VcConnection:656 - Logout from Vc Connection

2018-03-21 21:08:41.584 GMT  INFO http-nio-127.0.0.1-7441-exec-2 VcAuthenticationProvider:166 - There are no SSO Groups with role on vSM

After this I'm getting a warning  -  WARN taskScheduler-18 ControllerPoweronAdvisor:292 - Timeout on building connection between VSM and new deployed controller controller-8, then remove it

and also -  "VXLAN Controller controller-8 has been removed due to the connection cant be built, please check controller IP configuration and deploy again."

Also I have checked the controller IP everything is proper.

Any idea how can I fix it?

Thanks

[tanurkov]

Did the IP is duplicated

or pool is out of IP addresses ?

It simply says Manager cant reach Controller IP it is network issue

Regards Dmitri

[shobhitsinha]

Hi

The ip is not duplicated and it is specifically allocated for controllers from the pool.

Thanks

[amolnjadhav]

Hi,

  Do you have NSX Manager and NSX Controller are in same subnet?

  if it manager &controller are in different subnet, please try below command from NSX Manager :

   Manager#show arp

   Manager#show ip route

   Manager#ping manager_gateway

   Manager#ping controller_gateway

   I hope above commands will help to resolve your issue.

[bayupw]

I'm wondering if this error is the cause of the issue:

2018-03-21 21:08:41.584 GMT  INFO http-nio-127.0.0.1-7441-exec-2 VcAuthenticationProvider:166 - There are no SSO Groups with role on vSM

Do you have a dedicated user for registering NSX Manager to vCenter and PSC/Lookup Service URL?
Do you use same users? Check from NSX Manager web interface that both vCenter Server and Lookup Service URL status are Connected and green

Make sure you use a user that has been added into SSO Admin group in PSC

The the vCenter User Name that you put under the NSX Management Service will be used by NSX to do vSphere related tasks such as deploying NSX VMs/components or preparing the hosts

Make sure the clock/time between vCenter/PSC and NSX Manager are synchronised

[shobhitsinha]

Hi

Yes i have a dedicated user for registering NSX Manager to vCenter and PSC/Lookup Service URL and I'm using the same user(administrator). Both of them are connected and green.

Also my vCenter and nsx manager are synchronized. I'm usind a ntp server for both of them.

Is there anything else that I can try to make it work?

Thanks

[shobhitsinha]

Hi

They are in the same subnet.

[bayupw]

You mentioned you are using dedicated user, is it an Active Directory user e.g. domain\administrator ? or SSO default admin, administrator@vsphere.local?

Make sure the user is part of SSO Administrators, see below screenshot

I had same issue with same error "There are no SSO Groups with role on vSM" but I forgot what was the root cause

I think it was DNS, Time/NTP or SSO Admin issue.

This KB also highlight about time settings (check timezone too) and DNS to make sure that FQDN of VC/SSO, NSX Manager and ESXi hosts can be resolved: SSO and NSX/vShield Manager Integration (2131860)

Assuming the issue is related with the SSO, then the KB list the common problems as below

Common Problems Encountered in Troubleshooting:

• The host is unreachable or Hostname cannot be resolved.
• Invalid user or credentials.
• The User does not have proper SSO administrative rights.
• Time Sync error, request expired error.

[shobhitsinha]

Hi
I'm using the default administrator@vsphere.local user.

Here's the screenshot of my users and groups.

Thanks

[bayupw]

You mentioned you are using dedicated account, I thought you are using a user other than the administrator@vsphere.local

The administrator@vsphere.local is the SSO admin so there should not be an issue on that.

How about the other things that I mentioned like DNS? Can the vCenter FQDN, ESXi hosts FQDN be resolved from the NSX Manager?

And time to double check including the timezone?

[shobhitsinha]

Hi

Yes I have verified that the vCenter FQDN, ESXi hosts FQDN can be resolved from the NSX Manager. I am able to ping both of them using their FQDN.

All the above three are synced with the same ntp server and the timezone is UTC in all of them. Both nsxmanager and vcenter show the exact same time but the physical host where the controllers are to be deployed is 18 minutes ahead.

Is there anything that I can try to fix it?

Thanks

[bayupw]

The time skew on the ESXi host could be the issue.

For the resolution, you can set the ESXi time to use NTP server

or set the ESXi time manually to match the NSX Manager and vCenter Server's time first as a workaround

[shobhitsinha]

Hi

The esxi host is also synced with the same ntp server. 

However I configured the time manually to remove the time skew but still getting the same error.

[bayupw]

Those are the things that described in the documentation: NSX Controller Deployment Issues

Re: DNS, you mentioned that you check from NSX Manager that you can resolve ESXi and vCenter FQDN. Did you check from ESXi and from vCenter too?

Ensure that ESXi can reach NSX Manager and vCenter FQDN and vCenter can resolve NSX Manager and ESXi.

Did you still see the same error on NSX Manager log?

2018-03-21 21:08:41.584 GMT  INFO http-nio-127.0.0.1-7441-exec-2 VcAuthenticationProvider:166 - There are no SSO Groups with role on vSM

Did you deploy the NSX Controller node in the same subnet as vCenter and NSX Manager?

[shobhitsinha]

Hi

Yes I have verified that ESXi can reach NSX Manager and vCenter FQDN and vCenter can resolve NSX Manager and ESXi.

Did you still see the same error on NSX Manager log? - Yes

And yes I am deploying the nsx controller in the same subnet as vCenter and NSX Manager.