I wrote a same question but it seemed that it disappeared with unknown reason so I'm posting this again.(still the same question)
I'm currently using VCF4.1 with NSX-T 3.0 on a consolidated single cluster to build the supervisor cluster for future vSphere Pod service and am facing workload management hanging at "configuring" issue, which is the similar problem described in https://communities.vmware.com/t5/VMware-vSphere-Discussions/Enable-workload-management-hangs-on-con...
The edge cluster utilized for the workload management deployment is large size(8 vCPU + 32GB ram) of two VMs, and they are deployed via sddc manager.
The error message is shown in figure below:
In order to solve this, I did check some places:
1. ensured that VDS and physical switch port/routing MTU are set to 9000; all the uplink profiles for NSX-T TNs are 9000.
2. turned on the "enable trust" in compute manager setting of NSX-T for vCenter.
3. the NTP service is correct on each ESXi hosts.
4. the NSX-T manager VMs did not have NTP configured but they got time sync with esxi hosts.(NSX-T was deployed via cloud builder so there shouldn't be any time sync problem)
5. checked the error messages in logs: domainmanager, operationsmanager, nsxapi and wcpsvc but the root cause is still unclear for me.
some of the messages in wcpsvc.log are listed below:
-----------------------------------------------------------------------------
2021-01-22T02:31:41.365Z error wcp [opID=60079059] Error occurred sending Principal Identity request to NSX: principal identity already created
2021-01-22T02:31:41.385Z debug wcp [opID=60079059] NSX HTTP Request is: &{Method:POST URL:https://10.21.0.32:443/api/v1/trust-management/token-principal-identities/ Proto:HTTP/1.1 ProtoMajor:1 ProtoMinor:1 Header:map[] Body:{Reader:{"description":"Principal Identity for WCP service","display_name":"wcp-239f1518-8a0e-4da7-a38f-6900a5f3e575","id":"wcp-239f1518-8a0e-4da7-a38f-6900a5f3e575","name":"wcp-239f1518-8a0e-4da7-a38f-6900a5f3e575","node_id":"wcp-239f1518-8a0e-4da7-a38f-6900a5f3e575"}} GetBody:0x843110 ContentLength:261 TransferEncoding:[] Close:false Host:10.21.0.32:443 Form:map[] PostForm:map[] MultipartForm:<nil> Trailer:map[] RemoteAddr: RequestURI: TLS:<nil> Cancel:<nil> Response:<nil> ctx:0xc000056058}
2021-01-22T02:31:41.391Z debug wcp [opID=6007971d] Getting HOK signer; store: wcp, alias: wcp
2021-01-22T02:31:41.513Z debug wcp [opID=60079059] NSX HTTP Response is: &{409 409 HTTP/1.1 1 1 map[Cache-Control:[no-cache, no-store, max-age=0, must-revalidate] Content-Type:[application/json] Date:[Fri, 22 Jan 2021 02:31:41 GMT] Expires:[0] Keep-Alive:[timeout=60] Pragma:[no-cache] Server:[NSX] Set-Cookie:[JSESSIONID=6D583AE427E4579888B9A195D2D212BA; Path=/; Secure; HttpOnly] Strict-Transport-Security:[max-age=31536000 ; includeSubDomains] Vary:[accept-encoding] X-Content-Type-Options:[nosniff] X-Frame-Options:[SAMEORIGIN] X-Nsx-Requestid:[a8d8f6d5-8d94-4661-a229-cee313d97677] X-Xss-Protection:[1; mode=block]] 0xc004a82800 -1 [chunked] false true map[] 0xc002a00d00 0xc00114c840}, {
"httpStatus" : "CONFLICT",
"error_code" : 2039,
"module_name" : "internal-framework",
"error_message" : "Principal TokenBasedPrincipalIdentityEntity{schemaValue=, identifier=null/null, touched=false, revision=0, displayName=wcp-239f1518-8a0e-4da7-a38f-6900a5f3e575, description=Principal Identity for WCP service, createUser=null, lastModifiedUser=null, createTime=null, lastModifiedTime=null, systemResourceFlag=false, tags=[], name=wcp-239f1518-8a0e-4da7-a38f-6900a5f3e575, nodeId=wcp-239f1518-8a0e-4da7-a38f-6900a5f3e575, dataProtected=true} already exists."
}
2021-01-22T02:31:41.513Z error wcp [opID=60079059] Error occurred sending Principal Identity request to NSX: principal identity already created
2021-01-22T02:31:41.534Z debug wcp [opID=60079059] NSX HTTP Request is: &{Method:POST URL:https://10.21.0.33:443/api/v1/trust-management/token-principal-identities/ Proto:HTTP/1.1 ProtoMajor:1 ProtoMinor:1 Header:map[] Body:{Reader:{"description":"Principal Identity for WCP service","display_name":"wcp-239f1518-8a0e-4da7-a38f-6900a5f3e575","id":"wcp-239f1518-8a0e-4da7-a38f-6900a5f3e575","name":"wcp-239f1518-8a0e-4da7-a38f-6900a5f3e575","node_id":"wcp-239f1518-8a0e-4da7-a38f-6900a5f3e575"}} GetBody:0x843110 ContentLength:261 TransferEncoding:[] Close:false Host:10.21.0.33:443 Form:map[] PostForm:map[] MultipartForm:<nil> Trailer:map[] RemoteAddr: RequestURI: TLS:<nil> Cancel:<nil> Response:<nil> ctx:0xc000056058}
2021-01-22T02:31:41.591Z debug wcp [opID=6007971e] Getting HOK signer; store: wcp, alias: wcp
2021-01-22T02:31:41.679Z debug wcp [opID=60079059] NSX HTTP Response is: &{409 409 HTTP/1.1 1 1 map[Cache-Control:[no-cache, no-store, max-age=0, must-revalidate] Content-Type:[application/json] Date:[Fri, 22 Jan 2021 02:31:41 GMT] Expires:[0] Keep-Alive:[timeout=60] Pragma:[no-cache] Server:[NSX] Set-Cookie:[JSESSIONID=8CCCF4469755AE4AA1EB21C65B5C8886; Path=/; Secure; HttpOnly] Strict-Transport-Security:[max-age=31536000 ; includeSubDomains] Vary:[accept-encoding] X-Content-Type-Options:[nosniff] X-Frame-Options:[SAMEORIGIN] X-Nsx-Requestid:[f8098fc9-b4ff-43e3-890c-0081c611191a] X-Xss-Protection:[1; mode=block]] 0xc0003f8760 -1 [chunked] false true map[] 0xc002a01400 0xc00114cfd0}, {
"httpStatus" : "CONFLICT",
"error_code" : 2039,
"module_name" : "internal-framework",
"error_message" : "Principal TokenBasedPrincipalIdentityEntity{schemaValue=, identifier=null/null, touched=false, revision=0, displayName=wcp-239f1518-8a0e-4da7-a38f-6900a5f3e575, description=Principal Identity for WCP service, createUser=null, lastModifiedUser=null, createTime=null, lastModifiedTime=null, systemResourceFlag=false, tags=[], name=wcp-239f1518-8a0e-4da7-a38f-6900a5f3e575, nodeId=wcp-239f1518-8a0e-4da7-a38f-6900a5f3e575, dataProtected=true} already exists."
}
2021-01-22T02:31:41.679Z error wcp [opID=60079059] Error occurred sending Principal Identity request to NSX: principal identity already created
2021-01-22T02:31:41.698Z debug wcp [opID=60079059] NSX HTTP Request is: &{Method:POST URL:https://10.21.0.31:443/api/v1/trust-management/token-principal-identities/ Proto:HTTP/1.1 ProtoMajor:1 ProtoMinor:1 Header:map[] Body:{Reader:{"description":"Principal Identity for WCP service","display_name":"wcp-239f1518-8a0e-4da7-a38f-6900a5f3e575","id":"wcp-239f1518-8a0e-4da7-a38f-6900a5f3e575","name":"wcp-239f1518-8a0e-4da7-a38f-6900a5f3e575","node_id":"wcp-239f1518-8a0e-4da7-a38f-6900a5f3e575"}} GetBody:0x843110 ContentLength:261 TransferEncoding:[] Close:false Host:10.21.0.31:443 Form:map[] PostForm:map[] MultipartForm:<nil> Trailer:map[] RemoteAddr: RequestURI: TLS:<nil> Cancel:<nil> Response:<nil> ctx:0xc000056058}
2021-01-22T02:31:41.79Z debug wcp [opID=6007971f] Getting HOK signer; store: wcp, alias: wcp
2021-01-22T02:31:41.839Z debug wcp [opID=60079059] NSX HTTP Response is: &{409 409 HTTP/1.1 1 1 map[Cache-Control:[no-cache, no-store, max-age=0, must-revalidate] Content-Type:[application/json] Date:[Fri, 22 Jan 2021 02:31:41 GMT] Expires:[0] Keep-Alive:[timeout=60] Pragma:[no-cache] Server:[NSX] Set-Cookie:[JSESSIONID=C7409B5E046EF445A77617B9235E25DA; Path=/; Secure; HttpOnly] Strict-Transport-Security:[max-age=31536000 ; includeSubDomains] Vary:[accept-encoding] X-Content-Type-Options:[nosniff] X-Frame-Options:[SAMEORIGIN] X-Nsx-Requestid:[bf28ac5e-20db-4b73-9813-36d3fc270b59] X-Xss-Protection:[1; mode=block]] 0xc005147ba0 -1 [chunked] false true map[] 0xc002b17600 0xc001fef3f0}, {
"httpStatus" : "CONFLICT",
"error_code" : 2039,
"module_name" : "internal-framework",
"error_message" : "Principal TokenBasedPrincipalIdentityEntity{schemaValue=, identifier=null/null, touched=false, revision=0, displayName=wcp-239f1518-8a0e-4da7-a38f-6900a5f3e575, description=Principal Identity for WCP service, createUser=null, lastModifiedUser=null, createTime=null, lastModifiedTime=null, systemResourceFlag=false, tags=[], name=wcp-239f1518-8a0e-4da7-a38f-6900a5f3e575, nodeId=wcp-239f1518-8a0e-4da7-a38f-6900a5f3e575, dataProtected=true} already exists."
}
2021-01-22T02:31:41.839Z error wcp [opID=60079059] Error occurred sending Principal Identity request to NSX: principal identity already created
2021-01-22T02:31:41.839Z error wcp [opID=60079059] Failed to create PI in NSX managers. Err: principal identity already created
2021-01-22T02:31:41.839Z debug wcp [opID=60079059] WCP service principal identity already created.
2021-01-22T02:31:41.857Z debug wcp [opID=60079059] NSX HTTP Request is: &{Method:POST URL:https://10.21.0.34:443/api/v1/trust-management/token-principal-identities/ Proto:HTTP/1.1 ProtoMajor:1 ProtoMinor:1 Header:map[] Body:{Reader:{"description":"Principal Identity for WCP cluster service account: wcp-cluster-user-domain-c8-239f1518-8a0e-4da7-a38f-6900a5f3e575","display_name":"wcp-cluster-user-domain-c8-239f1518-8a0e-4da7-a38f-6900a5f3e575","id":"wcp-cluster-user-domain-c8-239f1518-8a0e-4da7-a38f-6900a5f3e575","name":"wcp-cluster-user-domain-c8-239f1518-8a0e-4da7-a38f-6900a5f3e575","node_id":"wcp-cluster-user-domain-c8-239f1518-8a0e-4da7-a38f-6900a5f3e575"}} GetBody:0x843110 ContentLength:434 TransferEncoding:[] Close:false Host:10.21.0.34:443 Form:map[] PostForm:map[] MultipartForm:<nil> Trailer:map[] RemoteAddr: RequestURI: TLS:<nil> Cancel:<nil> Response:<nil> ctx:0xc000056058}
2021-01-22T02:31:41.982Z debug wcp [opID=60079059] NSX HTTP Response is: &{409 409 HTTP/1.1 1 1 map[Cache-Control:[no-cache, no-store, max-age=0, must-revalidate] Content-Type:[application/json] Date:[Fri, 22 Jan 2021 02:31:41 GMT] Expires:[0] Keep-Alive:[timeout=60] Pragma:[no-cache] Server:[NSX] Set-Cookie:[JSESSIONID=DBE79E8792E458B7EBF1ACD49742A322; Path=/; Secure; HttpOnly] Strict-Transport-Security:[max-age=31536000 ; includeSubDomains] Vary:[accept-encoding] X-Content-Type-Options:[nosniff] X-Frame-Options:[SAMEORIGIN] X-Nsx-Requestid:[32d05af2-0211-41ac-9042-12d7df04fd35] X-Xss-Protection:[1; mode=block]] 0xc000bf8d00 -1 [chunked] false true map[] 0xc002a01f00 0xc0052b0a50}, {
"httpStatus" : "CONFLICT",
"error_code" : 2039,
"module_name" : "internal-framework",
"error_message" : "Principal TokenBasedPrincipalIdentityEntity{schemaValue=, identifier=null/null, touched=false, revision=0, displayName=wcp-cluster-user-domain-c8-239f1518-8a0e-4da7-a38f-6900a5f3e575, description=Principal Identity for WCP cluster service account: wcp-cluster-user-domain-c8-239f1518-8a0e-4da7-a38f-6900a5f3e575, createUser=null, lastModifiedUser=null, createTime=null, lastModifiedTime=null, systemResourceFlag=false, tags=[], name=wcp-cluster-user-domain-c8-239f1518-8a0e-4da7-a38f-6900a5f3e575, nodeId=wcp-cluster-user-domain-c8-239f1518-8a0e-4da7-a38f-6900a5f3e575, dataProtected=true} already exists."
}
2021-01-22T02:31:41.982Z error wcp [opID=60079059] Error occurred sending Principal Identity request to NSX: principal identity already created
2021-01-22T02:31:41.991Z debug wcp [opID=60079720] Getting HOK signer; store: wcp, alias: wcp
2021-01-22T02:31:42.001Z debug wcp [opID=60079059] NSX HTTP Request is: &{Method:POST URL:https://10.21.0.32:443/api/v1/trust-management/token-principal-identities/ Proto:HTTP/1.1 ProtoMajor:1 ProtoMinor:1 Header:map[] Body:{Reader:{"description":"Principal Identity for WCP cluster service account: wcp-cluster-user-domain-c8-239f1518-8a0e-4da7-a38f-6900a5f3e575","display_name":"wcp-cluster-user-domain-c8-239f1518-8a0e-4da7-a38f-6900a5f3e575","id":"wcp-cluster-user-domain-c8-239f1518-8a0e-4da7-a38f-6900a5f3e575","name":"wcp-cluster-user-domain-c8-239f1518-8a0e-4da7-a38f-6900a5f3e575","node_id":"wcp-cluster-user-domain-c8-239f1518-8a0e-4da7-a38f-6900a5f3e575"}} GetBody:0x843110 ContentLength:434 TransferEncoding:[] Close:false Host:10.21.0.32:443 Form:map[] PostForm:map[] MultipartForm:<nil> Trailer:map[] RemoteAddr: RequestURI: TLS:<nil> Cancel:<nil> Response:<nil> ctx:0xc000056058}
2021-01-22T02:31:42.12Z debug wcp [opID=60079059] NSX HTTP Response is: &{409 409 HTTP/1.1 1 1 map[Cache-Control:[no-cache, no-store, max-age=0, must-revalidate] Content-Type:[application/json] Date:[Fri, 22 Jan 2021 02:31:41 GMT] Expires:[0] Keep-Alive:[timeout=60] Pragma:[no-cache] Server:[NSX] Set-Cookie:[JSESSIONID=E9FDEBD11DCBB90F40487CCB29BA424D; Path=/; Secure; HttpOnly] Strict-Transport-Security:[max-age=31536000 ; includeSubDomains] Vary:[accept-encoding] X-Content-Type-Options:[nosniff] X-Frame-Options:[SAMEORIGIN] X-Nsx-Requestid:[ed260536-907d-4921-93a6-acfc8276a6bd] X-Xss-Protection:[1; mode=block]] 0xc002d71d60 -1 [chunked] false true map[] 0xc0029d5800 0xc0052b0e70}, {
"httpStatus" : "CONFLICT",
"error_code" : 2039,
"module_name" : "internal-framework",
"error_message" : "Principal TokenBasedPrincipalIdentityEntity{schemaValue=, identifier=null/null, touched=false, revision=0, displayName=wcp-cluster-user-domain-c8-239f1518-8a0e-4da7-a38f-6900a5f3e575, description=Principal Identity for WCP cluster service account: wcp-cluster-user-domain-c8-239f1518-8a0e-4da7-a38f-6900a5f3e575, createUser=null, lastModifiedUser=null, createTime=null, lastModifiedTime=null, systemResourceFlag=false, tags=[], name=wcp-cluster-user-domain-c8-239f1518-8a0e-4da7-a38f-6900a5f3e575, nodeId=wcp-cluster-user-domain-c8-239f1518-8a0e-4da7-a38f-6900a5f3e575, dataProtected=true} already exists."
}
--------------------------------------------------------------------------------
I would like to clarify some doubts when trying to input the parameters during the deployment of supervisor cluster:
1. as shown in figure below, is api server the single entry point of supervisor VMs? do i need to set FQDN name record on DNS server beforehand if I fill in the FQDN for it? (i never saw any "api server" being deployed)
2. regarding to the above figure, should the "ingress" and "egress" CIDR use VLAN subnets that don't need to be configured in NSX-T manager as IP range pool?(according to the "network topology" shown in the deployment page, it seemed that they should be the external VLAN network.)
3. The NSX-T manager VIP is accessible but it seemed that there's problem talking to the manager API during the deployment of supervisor VMs, why?
4. is the ssl certificate necessary for the vCenter, sddc manager, nsx-t manager...etc? I never configure certificate for them therefor the webpage would be unsafe to browse. (Not sure whether this would impact on the deployment of supervisor cluster or not.)
Note that i uploaded the latest 4 logs (as mentioned above) into a zip file.
can anyone prompt me?