7 Replies Latest reply on Jul 2, 2018 10:10 AM by vswitchzero

    NSX 6.3.3 Controllers Disconnected

    romuald_geo Lurker

      Hello,

       

      I have NSX manager version 6.3.3 Build 6276725 deployed with 3 controllers.In Networking & Security > Installation >Management all the controllers status is Disconnected. According to KB2151719 it's known bug. I followed the procedures as in the article, however when run API call of the step 3 I'm getting status code 200 but the body of the reply is the following:

       

      # of controllers: 3

       

      ------------------------------------------------------------------

      Fix controller: {Controller: 10.60.1.223 [controller-9], apiUser: admin }

        live prompt: <Password: >, SEND reply: <??>

        live prompt: <Password: >, SEND reply: <??>

        live prompt: <Password: >, SEND reply: <??>

        live prompt: <Password: >, SEND reply: <??>

        live prompt: <Password: >, SEND reply: <??>

      ERROR in connect: SSH_MSG_DISCONNECT: 2 Too many authentication failures

      script failed: com.jcraft.jsch.JSchException: SSH_MSG_DISCONNECT: 2 Too many authentication failures

       

      ------------------------------------------------------------------

      Fix controller: {Controller: 10.60.1.222 [controller-8], apiUser: admin }

        live prompt: <Password: >, SEND reply: <??>

        live prompt: <Password: >, SEND reply: <??>

        live prompt: <Password: >, SEND reply: <??>

        live prompt: <Password: >, SEND reply: <??>

        live prompt: <Password: >, SEND reply: <??>

      ERROR in connect: SSH_MSG_DISCONNECT: 2 Too many authentication failures

      script failed: com.jcraft.jsch.JSchException: SSH_MSG_DISCONNECT: 2 Too many authentication failures

       

      ------------------------------------------------------------------

      Fix controller: {Controller: 10.60.1.221 [controller-7], apiUser: admin }

        live prompt: <Password: >, SEND reply: <??>

        live prompt: <Password: >, SEND reply: <??>

        live prompt: <Password: >, SEND reply: <??>

        live prompt: <Password: >, SEND reply: <??>

        live prompt: <Password: >, SEND reply: <??>

      ERROR in connect: SSH_MSG_DISCONNECT: 2 Too many authentication failures

      script failed: com.jcraft.jsch.JSchException: SSH_MSG_DISCONNECT: 2 Too many authentication failures

      Patch script completed successfully.

       

      And all the controllers remain in disconnected status. Any ideas what could be causing the issue? What credentials are used when NSX Manager tries to connect to controllers and where can it be verified?

       

      Thanks

        • 2. Re: NSX 6.3.3 Controllers Disconnected
          romuald_geo Lurker

          Thanks for your time rajeevsrikant.

          It's the same issue described in KB2151719 and I followed the procedures as in the resolution, restarting the API servers on the controllers as well.But to no avail, they are still in the Disconnected state. 

          • 3. Re: NSX 6.3.3 Controllers Disconnected
            rajeevsrikant Expert
            Community Warriors

            VMware Knowledge Base

             

             

             

            Kindly go through this KB, in this tehy have mentioned about the below step. Could you please try this ?

             

             

            Note: As a part of Step 3, the script will set a temporary password on the Controller, log in to the root shell and change the password for the user account back to the original password set during initial Controller deployment. If any or all of the Controllers are re-deployed, repeat the preceding steps again.

            • 4. Re: NSX 6.3.3 Controllers Disconnected
              romuald_geo Lurker

              Scripts in KB51144 are the same as in previously mentioned KBs. When I run the script in step 2 I get 500 Internal Server Error. The very script I ran for the first time returned status code 200. When I issue API call from step 3 I get the same result 

              # of controllers: 3

               

              ------------------------------------------------------------------

              Fix controller: {Controller: 10.60.1.223 [controller-9], apiUser: admin }

                live prompt: <Password: >, SEND reply: <??>

                live prompt: <Password: >, SEND reply: <??>

                live prompt: <Password: >, SEND reply: <??>

                live prompt: <Password: >, SEND reply: <??>

                live prompt: <Password: >, SEND reply: <??>

              ERROR in connect: SSH_MSG_DISCONNECT: 2 Too many authentication failures

              script failed: com.jcraft.jsch.JSchException: SSH_MSG_DISCONNECT: 2 Too many authentication failures

               

              ------------------------------------------------------------------

              Fix controller: {Controller: 10.60.1.222 [controller-8], apiUser: admin }

                live prompt: <Password: >, SEND reply: <??>

                live prompt: <Password: >, SEND reply: <??>

                live prompt: <Password: >, SEND reply: <??>

                live prompt: <Password: >, SEND reply: <??>

                live prompt: <Password: >, SEND reply: <??>

              ERROR in connect: SSH_MSG_DISCONNECT: 2 Too many authentication failures

              script failed: com.jcraft.jsch.JSchException: SSH_MSG_DISCONNECT: 2 Too many authentication failures

               

              ------------------------------------------------------------------

              Fix controller: {Controller: 10.60.1.221 [controller-7], apiUser: admin }

                live prompt: <Password: >, SEND reply: <??>

                live prompt: <Password: >, SEND reply: <??>

                live prompt: <Password: >, SEND reply: <??>

                live prompt: <Password: >, SEND reply: <??>

                live prompt: <Password: >, SEND reply: <??>

              ERROR in connect: SSH_MSG_DISCONNECT: 2 Too many authentication failures

              script failed: com.jcraft.jsch.JSchException: SSH_MSG_DISCONNECT: 2 Too many authentication failures

              Patch script completed successfully.

              • 5. Re: NSX 6.3.3 Controllers Disconnected
                romuald_geo Lurker

                In my opinion the problem here is that NSX Manager is providing incorrect credentials when establishing SSH connection with controllers. I wonder if it's possible to configure which credentials are used or in general, what credentials are used for such communication.

                Thanks 

                • 6. Re: NSX 6.3.3 Controllers Disconnected
                  romuald_geo Lurker

                  Hi All,

                   

                  Any ideas ?

                  • 7. Re: NSX 6.3.3 Controllers Disconnected
                    vswitchzero Novice
                    vExpert

                    Hi romuaid_geo,

                     

                    I'd say your assessment is probably correct. The current controller 'admin' account password may not match what NSX manager thinks it should be. Did you happen to change the password from an earlier login attempt when prompted? If so, VMware GSS may be able to assist in setting it back to it's previous value to match again.

                     

                    If that proves problematic and you have already upgraded NSX Manager to one of the 're-released' 6.3.3 or 6.3.4 builds (or 6.3.5 and later) it may be easiest to just delete all three controller nodes and then re-deploy them. Keep in mind that if you don't want to upgrade the entire environment, upgrading manager to 6.3.3 build 7087283 won't require the upgrade of any other components aside from the control cluster). With their current state, a force-delete of all three controllers before re-deployment may be necessary. If you do proceed with this, make sure you do it during a maintenance window and be careful to ensure the appliances are deleted as part of the process. If not, they'll need to be manually removed. A force sync of routing and VXLAN services of all NSX prepared clusters would be a good idea once finished as well.

                     

                    Either way, I'd recommend opening an SR with VMware's NSX support team. They can help to debug the API call and try to get you back on track. If the API fix can't be used, they can also provide more guidance on the re-deployment process I outlined above.

                     

                    I hope this helps.

                     

                    Regards,

                    Mike