It's a Nested environment with VC and ESXi 7.0 U1
Using HA proxy as load balancer
Changed Security Promiscuous mode. MAC address changes and Forged transmits setting on nested ESXi Portgroup and HA Proxy Portgroups to ACCEPT
MTU is set to 1600 on both ESXi and workload mgmt vDS
Nested ESXi vDS MTU
Workload mgmt vDS MTU
Log snippet from wcpsvc.log
2020-10-15T00:05:02.674Z info wcp [opID=vCLS] Agency &{0xc002dd5900 {Agency 907a5a3a-e965-4efd-8f4a-3921f7c0cee4}} status is green on cluster domain-c8
2020-10-15T00:05:03.954Z debug wcp [opID=EAMAgent] Ignore non WCP agency vCLS
2020-10-15T00:05:03.999Z error wcp [opID=licenseRefreshMonitor] Supervisor control plane failed: No connectivity to API Master: connectivity Get https://10.155.124.67:6
443/healthz?timeout=5s: context deadline exceeded (Client.Timeout exceeded while awaiting headers), config status ERROR
2020-10-15T00:05:04.007Z debug wcp informer.processLoop() lister.List() returned
2020-10-15T00:05:06.999Z error wcp [opID=licenseRefreshMonitor] Supervisor control plane failed: No connectivity to API Master: connectivity Get https://10.155.124.67:6
443/healthz?timeout=5s: context deadline exceeded (Client.Timeout exceeded while awaiting headers), config status ERROR
2020-10-15T00:05:09.999Z error wcp [opID=licenseRefreshMonitor] Supervisor control plane failed: No connectivity to API Master: connectivity Get https://10.155.124.67:6
443/healthz?timeout=5s: context deadline exceeded (Client.Timeout exceeded while awaiting headers), config status ERROR
2020-10-15T00:05:11.486Z debug wcp [opID=5f816ea2] No notifications. seqNum: 524, Current seqNum: 523
2020-10-15T00:05:12.112Z error wcp Failed to get Kubernetes healthz results on server, 10.155.124.67: Get https://10.155.124.67:6443/healthz?timeout=5s: dial tcp 10.155
.124.67:6443: connect: connection refused
2020-10-15T00:05:12.112Z debug wcp healthz for 10.155.124.67 = "Get https://10.155.124.67:6443/healthz?timeout=5s: dial tcp 10.155.124.67:6443: connect: connection refu
sed"
2020-10-15T00:05:12.112Z debug wcp Publish change event: &cdc.ChangeLogChangeEvent{Resource:std.DynamicID{Type_:"ClusterComputeResource", Id:"domain-c8"}, Kind:"UPDATE"
, Properties:[]string{"kubernetes_status"}, ParentResources:[]std.DynamicID(nil)}
2020-10-15T00:05:12.999Z error wcp [opID=licenseRefreshMonitor] Supervisor control plane failed: No connectivity to API Master: connectivity Get https://10.155.124.67:6
443/healthz?timeout=5s: dial tcp 10.155.124.67:6443: connect: connection refused, config status ERROR
2020-10-15T00:05:14.022Z debug wcp [opID=EAMAgent] Ignore non WCP agency vCLS
2020-10-15T00:05:14.074Z debug wcp informer.processLoop() lister.List() returned
2020-10-15T00:05:15.999Z error wcp [opID=licenseRefreshMonitor] Supervisor control plane failed: No connectivity to API Master: connectivity Get https://10.155.124.67:6
443/healthz?timeout=5s: dial tcp 10.155.124.67:6443: connect: connection refused, config status ERROR
2020-10-15T00:05:18.999Z error wcp [opID=licenseRefreshMonitor] Supervisor control plane failed: No connectivity to API Master: connectivity Get https://10.155.124.67:6
443/healthz?timeout=5s: dial tcp 10.155.124.67:6443: connect: connection refused, config status ERROR
2020-10-15T00:05:21.486Z debug wcp [opID=5f816ea2] seqNum: 524, Current seqNum: 524, CL: &{525 [{{ClusterComputeResource domain-c8} UPDATE [kubernetes_status] []}]}
2020-10-15T00:05:21.486Z debug wcp [opID=5f816ea2] GetChanges result for sequence token: 524, &{525 [{{ClusterComputeResource domain-c8} UPDATE [kubernetes_status] []}]
}
2020-10-15T00:05:21.486Z debug wcp [opID=vapi] Validating output
2020-10-15T00:05:21.486Z debug wcp [opID=vapi] Request processing complete
2020-10-15T00:05:21.487Z debug wcp [opID=vapi] Sending response with output {"output":{"STRUCTURE":{"com.vmware.cdc.change_log.change_list":{"changes":{"OPTIONAL":[{"ST
RUCTURE":{"com.vmware.cdc.change_log.change_event":{"kind":"UPDATE","parent_resources":{"OPTIONAL":null},"properties":{"OPTIONAL":["kubernetes_status"]},"resource":{"ST
RUCTURE":{"com.vmware.vapi.std.dynamic_ID":{"id":"domain-c8","type":"ClusterComputeResource"}}}}}}]},"sequence":"525"}}}}
2020-10-15T00:05:21.999Z error wcp [opID=licenseRefreshMonitor] Supervisor control plane failed: No connectivity to API Master: connectivity Get https://10.155.124.67:6
443/healthz?timeout=5s: dial tcp 10.155.124.67:6443: connect: connection refused, config status ERROR
10.155.124.67:6443: connect: connection refused"
can you confirm the service is listening on port 6443 or something else is using this port?
thanks,
MS
2020-10-15T15:18:52.288Z error wcp [opID=5f822198-domain-c8] Failed to create VSphere Distributed Network object. Err Post https://10.155.124.67:6443/apis/netoperator.vmware.com/v1alpha1/vspheredistributednetworks?timeout=2... dial tcp 10.155.124.67:6443: connect: connection refused
2020-10-15T15:18:52.288Z error wcp [opID=5f822198-domain-c8] Error setting up cluster network: Post https://10.155.124.67:6443/apis/netoperator.vmware.com/v1alpha1/vspheredistributednetworks?timeout=2... dial tcp 10.155.124.67:6443: connect: connection refused
2020-10-15T15:18:52.288Z error wcp [opID=5f822198-domain-c8] Error configuring cluster NIC on master VM vm-105: Post https://10.155.124.67:6443/apis/netoperator.vmware.com/v1alpha1/vspheredistributednetworks?timeout=2... dial tcp 10.155.124.67:6443: connect: connection refused
2020-10-15T15:18:52.288Z error wcp [opID=5f822198-domain-c8] Error configuring API server on cluster domain-c8 Error configuring cluster NIC on master VM. This operation is part of API server configuration and will be retried.
Supervisor control plane failed: No connectivity to API Master: connectivity Get https://10.155.124.67:6443/healthz?timeout=5s: dial tcp 10.155.124.67:6443: connect: connection refused, config status ERROR
From VC the connection is refused to API
curl -v telnet://10.155.124.67:6443
* Rebuilt URL to: telnet://10.155.124.67:6443/
* Trying 10.155.124.67...
* TCP_NODELAY set
* connect to 10.155.124.67 port 6443 failed: Connection refused
* Failed to connect to 10.155.124.67 port 6443: Connection refused
* Closing connection 0
curl: (7) Failed to connect to 10.155.124.67 port 6443: Connection refused
How do we fix this ?
Bumping up to see if somebody can help me out here