msk05
Enthusiast
Enthusiast

Enabling workload management fails

It's a Nested environment with VC and ESXi 7.0 U1

Using HA proxy as load balancer

Changed Security Promiscuous mode. MAC address changes and Forged transmits setting on nested ESXi Portgroup and HA Proxy Portgroups  to ACCEPT

pastedImage_4.png

pastedImage_5.png

MTU is set to 1600 on both ESXi and workload mgmt vDS

Nested ESXi vDS MTU

pastedImage_2.png

Workload mgmt vDS MTU

pastedImage_3.png

pastedImage_0.png

Log snippet from wcpsvc.log

2020-10-15T00:05:02.674Z info wcp [opID=vCLS] Agency &{0xc002dd5900 {Agency 907a5a3a-e965-4efd-8f4a-3921f7c0cee4}} status is green on cluster domain-c8

2020-10-15T00:05:03.954Z debug wcp [opID=EAMAgent] Ignore non WCP agency vCLS

2020-10-15T00:05:03.999Z error wcp [opID=licenseRefreshMonitor] Supervisor control plane failed: No connectivity to API Master: connectivity Get https://10.155.124.67:6

443/healthz?timeout=5s: context deadline exceeded (Client.Timeout exceeded while awaiting headers), config status ERROR

2020-10-15T00:05:04.007Z debug wcp informer.processLoop() lister.List() returned

2020-10-15T00:05:06.999Z error wcp [opID=licenseRefreshMonitor] Supervisor control plane failed: No connectivity to API Master: connectivity Get https://10.155.124.67:6

443/healthz?timeout=5s: context deadline exceeded (Client.Timeout exceeded while awaiting headers), config status ERROR

2020-10-15T00:05:09.999Z error wcp [opID=licenseRefreshMonitor] Supervisor control plane failed: No connectivity to API Master: connectivity Get https://10.155.124.67:6

443/healthz?timeout=5s: context deadline exceeded (Client.Timeout exceeded while awaiting headers), config status ERROR

2020-10-15T00:05:11.486Z debug wcp [opID=5f816ea2] No notifications. seqNum: 524, Current seqNum: 523

2020-10-15T00:05:12.112Z error wcp Failed to get Kubernetes healthz results on server, 10.155.124.67: Get https://10.155.124.67:6443/healthz?timeout=5s: dial tcp 10.155

.124.67:6443: connect: connection refused

2020-10-15T00:05:12.112Z debug wcp healthz for 10.155.124.67 = "Get https://10.155.124.67:6443/healthz?timeout=5s: dial tcp 10.155.124.67:6443: connect: connection refu

sed"

2020-10-15T00:05:12.112Z debug wcp Publish change event: &cdc.ChangeLogChangeEvent{Resource:std.DynamicID{Type_:"ClusterComputeResource", Id:"domain-c8"}, Kind:"UPDATE"

, Properties:[]string{"kubernetes_status"}, ParentResources:[]std.DynamicID(nil)}

2020-10-15T00:05:12.999Z error wcp [opID=licenseRefreshMonitor] Supervisor control plane failed: No connectivity to API Master: connectivity Get https://10.155.124.67:6

443/healthz?timeout=5s: dial tcp 10.155.124.67:6443: connect: connection refused, config status ERROR

2020-10-15T00:05:14.022Z debug wcp [opID=EAMAgent] Ignore non WCP agency vCLS

2020-10-15T00:05:14.074Z debug wcp informer.processLoop() lister.List() returned

2020-10-15T00:05:15.999Z error wcp [opID=licenseRefreshMonitor] Supervisor control plane failed: No connectivity to API Master: connectivity Get https://10.155.124.67:6

443/healthz?timeout=5s: dial tcp 10.155.124.67:6443: connect: connection refused, config status ERROR

2020-10-15T00:05:18.999Z error wcp [opID=licenseRefreshMonitor] Supervisor control plane failed: No connectivity to API Master: connectivity Get https://10.155.124.67:6

443/healthz?timeout=5s: dial tcp 10.155.124.67:6443: connect: connection refused, config status ERROR

2020-10-15T00:05:21.486Z debug wcp [opID=5f816ea2] seqNum: 524, Current seqNum: 524, CL: &{525 [{{ClusterComputeResource domain-c8} UPDATE [kubernetes_status] []}]}

2020-10-15T00:05:21.486Z debug wcp [opID=5f816ea2] GetChanges result for sequence token: 524, &{525 [{{ClusterComputeResource domain-c8} UPDATE [kubernetes_status] []}]

}

2020-10-15T00:05:21.486Z debug wcp [opID=vapi] Validating output

2020-10-15T00:05:21.486Z debug wcp [opID=vapi] Request processing complete

2020-10-15T00:05:21.487Z debug wcp [opID=vapi] Sending response with output {"output":{"STRUCTURE":{"com.vmware.cdc.change_log.change_list":{"changes":{"OPTIONAL":[{"ST

RUCTURE":{"com.vmware.cdc.change_log.change_event":{"kind":"UPDATE","parent_resources":{"OPTIONAL":null},"properties":{"OPTIONAL":["kubernetes_status"]},"resource":{"ST

RUCTURE":{"com.vmware.vapi.std.dynamic_ID":{"id":"domain-c8","type":"ClusterComputeResource"}}}}}}]},"sequence":"525"}}}}

2020-10-15T00:05:21.999Z error wcp [opID=licenseRefreshMonitor] Supervisor control plane failed: No connectivity to API Master: connectivity Get https://10.155.124.67:6

443/healthz?timeout=5s: dial tcp 10.155.124.67:6443: connect: connection refused, config status ERROR

0 Kudos
3 Replies
msripada
Virtuoso
Virtuoso

10.155.124.67:6443: connect: connection refused"

can you confirm the service is listening on port 6443 or something else is using this port?

thanks,

MS

0 Kudos
msk05
Enthusiast
Enthusiast

2020-10-15T15:18:52.288Z error wcp [opID=5f822198-domain-c8] Failed to create VSphere Distributed Network object. Err Post https://10.155.124.67:6443/apis/netoperator.vmware.com/v1alpha1/vspheredistributednetworks?timeout=2... dial tcp 10.155.124.67:6443: connect: connection refused

2020-10-15T15:18:52.288Z error wcp [opID=5f822198-domain-c8] Error setting up cluster network: Post https://10.155.124.67:6443/apis/netoperator.vmware.com/v1alpha1/vspheredistributednetworks?timeout=2... dial tcp 10.155.124.67:6443: connect: connection refused

2020-10-15T15:18:52.288Z error wcp [opID=5f822198-domain-c8] Error configuring cluster NIC on master VM vm-105: Post https://10.155.124.67:6443/apis/netoperator.vmware.com/v1alpha1/vspheredistributednetworks?timeout=2... dial tcp 10.155.124.67:6443: connect: connection refused

2020-10-15T15:18:52.288Z error wcp [opID=5f822198-domain-c8] Error configuring API server on cluster domain-c8 Error configuring cluster NIC on master VM. This operation is part of API server configuration and will be retried.

Supervisor control plane failed: No connectivity to API Master: connectivity Get https://10.155.124.67:6443/healthz?timeout=5s: dial tcp 10.155.124.67:6443: connect: connection refused, config status ERROR

From VC the connection is refused to API

curl -v telnet://10.155.124.67:6443

* Rebuilt URL to: telnet://10.155.124.67:6443/

*   Trying 10.155.124.67...

* TCP_NODELAY set

* connect to 10.155.124.67 port 6443 failed: Connection refused

* Failed to connect to 10.155.124.67 port 6443: Connection refused

* Closing connection 0

curl: (7) Failed to connect to 10.155.124.67 port 6443: Connection refused

How do we fix this ?

0 Kudos
msk05
Enthusiast
Enthusiast

Bumping up to see if somebody can help me out here

0 Kudos