VMware Networking Community
rajkumar49
Contributor
Contributor
Jump to solution

After rebooting NSX-T 3.0 manager, UI is not coming up- error code 101

Error message in the web page :

Some appliance components are not functioning properly.

Component health: SEARCH:UP, POLICY:UP, MANAGER:UNKNOWN, UI:UP, NODE_MGMT:UP.

Error code: 101

any idea to fix this ?

Tags (1)
Reply
0 Kudos
1 Solution

Accepted Solutions
Lalegre
Virtuoso
Virtuoso
Jump to solution

You are following a procedure for a wrong version. The command "detach management-plane" is mentioned to use in the documentation till version 2.3 of NSX-T. As you mentioned that you are facing issues with NSX-T 3.0 you should follow the procedure that involves the "del nsx" command as mentioned here: Uninstall NSX-T Data Center from a Host in a vSphere Cluster

This command will completely wipe all the vibs and configurations from the ESXi. Please make sure that your VMkernels are in a VSS or VDS and not on a N-VDS: Verify Host Network Mappings for Uninstall

View solution in original post

Reply
0 Kudos
9 Replies
rajkumar49
Contributor
Contributor
Jump to solution

syslog :

2020-08-19T11:37:57.547Z nsx-manager NSX 6600 - [nsx@6876 comp="nsx-manager" level="WARNING" subcomp="policy"] ConnectionFuture completed with error io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: /127.0.0.1:9831. Retrying...

<182>1 2020-08-19T11:37:57.598Z nsx-manager NSX 1662 - [nsx@6876 comp="nsx-controller" level="INFO" subcomp="adapt

Reply
0 Kudos
Lalegre
Virtuoso
Virtuoso
Jump to solution

It seems that you are facing some issues with the Manager services. If you run: get services. What is the output you receive?

If the manager services is down tray to start it manually and if it fails please paste ERROR output of the manager.log using this command: get log-file manager.log

Reply
0 Kudos
rajkumar49
Contributor
Contributor
Jump to solution

I restarted the nsx-manager service, but still no luck. the login page is fine in the UI, after logging in only error comes.

nsx-manager> get services

Service name:                            applianceproxy

Service state:                           running

Service name:                            async_replicator

Service state:                           running

Logging level:                           info

Service name:                            cluster_manager

Service state:                           running

Service name:                            cm-inventory

Service state:                           running

Service name:                            controller

Service state:                           running

Listen address:

Service name:                            datastore

Service state:                           running

Service name:                            http

Service state:                           running

Logging level:                           info

Session timeout:                         1800

Connection timeout:                      30

Redirect host:                           (not configured)

Client API rate limit:                   100 requests/sec

Client API concurrency limit:            40

Global API concurrency limit:            199

Basic authentication:                    enabled

Cookie-based authentication:             enabled

Service name:                            idps-reporting

Service state:                           running

Service name:                            install-upgrade

Service state:                           running

Enabled on:

Service name:                            intelligence-upgrade-coordinator

Service state:                           running

Service name:                            liagent

Service state:                           stopped

Service name:                            manager

Service state:                           running

Logging level:                           info

Service name:                            mgmt-plane-bus

Service state:                           running

Service name:                            migration-coordinator

Service state:                           stopped

Service name:                            node-mgmt

Service state:                           running

Service name:                            node-stats

Service state:                           running

Service name:                            nsx-message-bus

Service state:                           stopped

Service name:                            nsx-platform-client

Service state:                           running

Service name:                            nsx-upgrade-agent

Service state:                           running

Service name:                            ntp

Service state:                           running

Start on boot:                           True

Service name:                            policy

Service state:                           running

Logging level:                           info

Service name:                            search

Service state:                           running

Service name:                            snmp

Service state:                           stopped

Start on boot:                           False

Service name:                            ssh

Service state:                           running

Start on boot:                           True

Service name:                            syslog

Service state:                           running

Service name:                            telemetry

Service state:                           running

Service name:                            ui-service

Service state:                           running

nsx-manager>

manager.log:

2020-08-20T06:44:38.276Z  INFO http-nio-127.0.0.1-7440-exec-2 AuditingServiceImpl - - [nsx@6876 audit="true" comp="nsx-manager" level="INFO" reqId="0cf484cb-ed28-41e1-8513-a7417eb726b7" subcomp="manager" username="node-mgmt"] UserName="node-mgmt", ModuleName="AAA", Operation="ListRolesInfo", Operation status="failure"

2020-08-20T06:44:38.276Z  INFO http-nio-127.0.0.1-7440-exec-2 NsxBaseRestController - - [nsx@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Error in API /nsxapi/api/v1/aaa/roles-with-feature-permissions caused by exception com.vmware.nsx.management.common.exceptions.BaseException:  {"moduleName":"common-services","errorCode":280,"errorMessage":"Application is not ready"}

2020-08-20T06:44:40.446Z  WARN http-nio-127.0.0.1-7440-exec-11 PageNotFound - No mapping for GET /nsxapi/api/v1/fabric/nodes

2020-08-20T06:44:42.866Z  WARN http-nio-127.0.0.1-7440-exec-7 AAAFacadeImpl - SYSTEM [nsx@6876 comp="nsx-manager" level="WARNING" reqId="cd07e62f-3644-4227-a393-266685f8dca3" subcomp="manager" username="node-mgmt"] Application not ready

2020-08-20T06:44:42.866Z  INFO http-nio-127.0.0.1-7440-exec-7 AuditingServiceImpl - - [nsx@6876 audit="true" comp="nsx-manager" level="INFO" reqId="cd07e62f-3644-4227-a393-266685f8dca3" subcomp="manager" username="node-mgmt"] UserName="node-mgmt", ModuleName="AAA", Operation="GetAllRoleBindings", Operation status="failure", New value=[{"page_size":1000}]

2020-08-20T06:44:42.867Z  INFO http-nio-127.0.0.1-7440-exec-7 NsxBaseRestController - - [nsx@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Error in API /nsxapi/api/v1/aaa/role-bindings caused by exception com.vmware.nsx.management.common.exceptions.BaseException:  {"moduleName":"common-services","errorCode":280,"errorMessage":"Application is not ready"}

2020-08-20T06:44:43.284Z  WARN http-nio-127.0.0.1-7440-exec-3 AAAFacadeImpl - SYSTEM [nsx@6876 comp="nsx-manager" level="WARNING" reqId="496a397a-9302-444e-8db2-8fa6ccb19b2b" subcomp="manager" username="node-mgmt"] Application not ready

2020-08-20T06:44:43.284Z  INFO http-nio-127.0.0.1-7440-exec-3 AuditingServiceImpl - - [nsx@6876 audit="true" comp="nsx-manager" level="INFO" reqId="496a397a-9302-444e-8db2-8fa6ccb19b2b" subcomp="manager" username="node-mgmt"] UserName="node-mgmt", ModuleName="AAA", Operation="ListRolesInfo", Operation status="failure"

2020-08-20T06:44:43.284Z  INFO http-nio-127.0.0.1-7440-exec-3 NsxBaseRestController - - [nsx@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Error in API /nsxapi/api/v1/aaa/roles-with-feature-permissions caused by exception com.vmware.nsx.management.common.exceptions.BaseException:  {"moduleName":"common-services","errorCode":280,"errorMessage":"Application is not ready"}

2020-08-20T06:44:43.674Z  WARN http-nio-127.0.0.1-7440-exec-6 PageNotFound - No mapping for GET /nsxapi/api/v1/fabric/nodes

2020-08-20T06:44:43.689Z  INFO http-nio-127.0.0.1-7440-exec-5 AuditingServiceImpl - SYSTEM [nsx@6876 audit="true" comp="nsx-manager" level="INFO" reqId="ea98eaba-583f-4f1d-8b44-e0bc143b7446" subcomp="manager" username="admin"] UserName="admin", ModuleName="ClusterManagement", Operation="ListClusterNodeConfigs", Operation status="success"

2020-08-20T06:44:45.475Z  WARN http-nio-127.0.0.1-7440-exec-4 PageNotFound - No mapping for GET /nsxapi/api/v1/fabric/nodes

2020-08-20T06:44:46.612Z  WARN ShardingService ShardingServiceImpl - - [nsx@6876 comp="nsx-manager" level="WARNING" subcomp="manager"] Node has not joined the cluster yet. Skipping: self node: ClusterNodeConfigModel/f3562e42-2f54-07c5-5cd2-7985e32954b2, current cluster membership: []

Reply
0 Kudos
rajkumar49
Contributor
Contributor
Jump to solution

we are okay with redeploying the NSX-T manager appliance.

is there any way to safely remove the configurations of the current NSX manager using CLI command ?

Reply
0 Kudos
Lalegre
Virtuoso
Virtuoso
Jump to solution

Hello rajkumar49​,

If you are going to re-install NSX-T you should not worry about the configurations inside NSX-T, the only configurations you should worry are the ones on your Transport Nodes (ESXi or KVM)

I encourage you to read this full section: Uninstalling NSX-T Data Center from a Host Transport Node

Reply
0 Kudos
rajkumar49
Contributor
Contributor
Jump to solution

hi,

thanks for the info,

we have put the Transport node ESXi into maintenance mode as per the guide.

then executed the below command in ESXi :

Command :

detach management-plane <MANAGER> username <ADMIN-USER> password <ADMIN-PASSWORD> thumbprint <MANAGER-THUMBPRINT>

Error :

% Node unregistration failed: 'the manager API server rejected the deregistration attempt'

any suggestion ?

Reply
0 Kudos
Lalegre
Virtuoso
Virtuoso
Jump to solution

You are following a procedure for a wrong version. The command "detach management-plane" is mentioned to use in the documentation till version 2.3 of NSX-T. As you mentioned that you are facing issues with NSX-T 3.0 you should follow the procedure that involves the "del nsx" command as mentioned here: Uninstall NSX-T Data Center from a Host in a vSphere Cluster

This command will completely wipe all the vibs and configurations from the ESXi. Please make sure that your VMkernels are in a VSS or VDS and not on a N-VDS: Verify Host Network Mappings for Uninstall

Reply
0 Kudos
xuewenchen
Contributor
Contributor
Jump to solution

me too,

I seem to be having the same problem,

How do you solve it right now?

Reply
0 Kudos
lucsan
Contributor
Contributor
Jump to solution

Hi, 

what is the deployed manager size you have in your environment? you might need to increase the memory on the VM,  power off the managers one by one and edit settings on the vm and increase the mem. 

 

Reply
0 Kudos