Workload Management - Error configuring cluster NIC on master VM

[barnette08]

I've seen a few posts on this similar topic but I can't seem to narrow down the issue. Any ideas?

When I ssh into the supervisor node and issue the  kubectl get pods -A command and can see that I'm getting a crashloop for coredns.

NAMESPACE         NAME                                                              
kube-system    coredns-5dc469f6f-tvljf      0/1     CrashLoopBackOff

When I take a look into /var/log/vmware/wcp/wcpsvc.log, I can see that there's a timeout issue.

2021-07-15T19:01:40.58Z error wcp [opID=60f056b8-domain-c144] Error creating NSX resources. Err: Get https://supervisor-cluster-ip:0/apis/vmware.com/v1alpha1/nam
espaces/kube-system/virtualnetworkinterfaces?labelSelector=vm-moid+in+%28vm-1050%29: dial tcp supervisor-cluster-ip:0: i/o timeout
2021-07-15T19:01:40.58Z error wcp [opID=60f056b8-domain-c144] Failed to create cluster network interface for MasterNode: VirtualMachine:vm-1050. Err: Get
https://supervisor-cluster-ip:0/apis/vmware.com/v1alpha1/namespaces/kube-system/virtualnetworkinterfaces?labelSelector=vm-moid+in+%28vm-1050%29: dial tcp 10.234.
146.171:0: i/o timeout
2021-07-15T19:01:40.58Z error wcp [opID=60f056b8-domain-c144] Error configuring cluster NIC on master VM vm-1050: Get https://supervisor-cluster-ip:0/apis/vmware
.com/v1alpha1/namespaces/kube-system/virtualnetworkinterfaces?labelSelector=vm-moid+in+%28vm-1050%29: dial tcp supervisor-cluster-ip:0: i/o timeout
2021-07-15T19:01:40.58Z error wcp [opID=60f056b8-domain-c144] Error configuring API server on cluster domain-c144 Error configuring cluster NIC on master
VM. This operation is part of API server configuration and will be retried.
2021-07-15T19:01:40.629Z error wcp [opID=60f056b8-domain-c144] Error checking if NSX resources exist. Err: Get https://supervisor-cluster-ip:0/apis/vmware.com/v1
alpha1/namespaces/kube-system/virtualnetworkinterfaces?labelSelector=vm-moid+in+%28vm-1049%29: dial tcp supervisor-cluster-ip:0: i/o timeout
2021-07-15T19:01:40.629Z error wcp [opID=60f056b8-domain-c144] Error checking if NSX resources exist for VMs: [vm-1049]. Err: Get https://supervisor-cluster-ip:0
/apis/vmware.com/v1alpha1/namespaces/kube-system/virtualnetworkinterfaces?labelSelector=vm-moid+in+%28vm-1049%29: dial tcp supervisor-cluster-ip:0: i/o timeout
2021-07-15T19:01:40.629Z error wcp [opID=60f056b8-domain-c144] Error creating NSX resources. Err: Get https://supervisor-cluster-ip:0/apis/vmware.com/v1alpha1/na
mespaces/kube-system/virtualnetworkinterfaces?labelSelector=vm-moid+in+%28vm-1049%29: dial tcp supervisor-cluster-ip:0: i/o timeout
2021-07-15T19:01:40.629Z error wcp [opID=60f056b8-domain-c144] Failed to create cluster network interface for MasterNode: VirtualMachine:vm-1049. Err: Get
https://supervisor-cluster-ip:0/apis/vmware.com/v1alpha1/namespaces/kube-system/virtualnetworkinterfaces?labelSelector=vm-moid+in+%28vm-1049%29: dial tcp 10.234
.146.171:0: i/o timeout
2021-07-15T19:01:40.629Z error wcp [opID=60f056b8-domain-c144] Error configuring cluster NIC on master VM vm-1049: Get https://supervisor-cluster-ip:0/apis/vmwar
e.com/v1alpha1/namespaces/kube-system/virtualnetworkinterfaces?labelSelector=vm-moid+in+%28vm-1049%29: dial tcp supervisor-cluster-ip:0: i/o timeout
2021-07-15T19:01:40.629Z error wcp [opID=60f056b8-domain-c144] Error configuring API server on cluster domain-c144 Error configuring cluster NIC on master VM. This operation is part of API server configuration and will be retried.
2021-07-15T19:01:40.629Z warning wcp [opID=60f056b8-domain-c144] Error configuring cluster NIC. Err <nil>
2021-07-15T19:01:40.629Z warning wcp [opID=60f056b8-domain-c144] Error configuring cluster NIC. Err <nil>
2021-07-15T19:01:40.629Z warning wcp [opID=60f056b8-domain-c144] Error configuring cluster NIC. Err <nil>
2021-07-15T19:01:40.629Z info wcp [opID=60f056b8-domain-c144] no single master succeeded - retrying

2021-07-15T19:02:54.238Z error wcp [opID=vapi] Authentication failed.
2021-07-15T19:02:54.238Z error wcp [opID=vapi] Unable to authenticate user
2021-07-15T19:02:54.238Z debug wcp [opID=vapi] Sending response with output {"error":{"ERROR":{"com.vmware.vapi.std.errors.unauthenticated":{"data":{"OPTI
ONAL":null},"error_type":{"OPTIONAL":"UNAUTHENTICATED"},"messages":[{"STRUCTURE":{"com.vmware.vapi.std.localizable_message":{"args":[],"default_message":"
Unable to authenticate user","id":"vapi.security.authentication.invalid","localized":{"OPTIONAL":null},"params":{"OPTIONAL":null}}}}]}}}}