Contributor
Contributor

vSphere with Tanzu - Guest cluster deployment stuck at Control Plane VM

I have master vCenter and 3x nested ESXi with VSAN on which I tried to test Tanzu , VDS switch (not NSX-T), HAproxy as LB.

All MGMT are on existing network, deployed 2x new VLAN, one for Frontend and one for Workload.

Managed to deploy SupervisorCluster up and running, Namespaces. When I try to deploy Guest cluster with simple .yaml I got stuck.

Control Plane VM gets deployed, powered on, HAproxy .cfg populated with frontend/backend IPs, IP set up (unpingable) but then stuck before Wokers deployment.

 

kubectl get events -w
LAST SEEN TYPE REASON OBJECT MESSAGE
0s Warning ReconcileFailure wcpcluster/simple unexpected error while reconciling control plane endpoint for simple: failed to reconcile loadbalanced endpoint for WCPCluster tanzu-ns-01/simple: failed to get control plane endpoint for Cluster tanzu-ns-01/simple: VirtualMachineService LB does not yet have VIP assigned: VirtualMachineService LoadBalancer does not have any Ingresses
0s Normal CreateVMServiceSuccess virtualmachineservice/simple-control-plane-service CreateVMService success
0s Warning ReconcileFailure wcpcluster/simple unexpected error while reconciling control plane endpoint for simple: failed to reconcile loadbalanced endpoint for WCPCluster tanzu-ns-01/simple: failed to get control plane endpoint for Cluster tanzu-ns-01/simple: VirtualMachineService LB does not yet have VIP assigned: VirtualMachineService LoadBalancer does not have any Ingresses
0s Warning ReconcileFailure wcpcluster/simple unexpected error while reconciling control plane endpoint for simple: failed to reconcile loadbalanced endpoint for WCPCluster tanzu-ns-01/simple: failed to get control plane endpoint for Cluster tanzu-ns-01/simple: VirtualMachineService LB does not yet have VIP assigned: VirtualMachineService LoadBalancer does not have any Ingresses
0s Warning ReconcileFailure wcpcluster/simple unexpected error while reconciling control plane endpoint for simple: failed to reconcile loadbalanced endpoint for WCPCluster tanzu-ns-01/simple: failed to get control plane endpoint for Cluster tanzu-ns-01/simple: VirtualMachineService LB does not yet have VIP assigned: VirtualMachineService LoadBalancer does not have any Ingresses
0s Warning ReconcileFailure wcpcluster/simple unexpected error while reconciling control plane endpoint for simple: failed to reconcile loadbalanced endpoint for WCPCluster tanzu-ns-01/simple: failed to get control plane endpoint for Cluster tanzu-ns-01/simple: VirtualMachineService LB does not yet have VIP assigned: VirtualMachineService LoadBalancer does not have any Ingresses
0s Warning ReconcileFailure wcpcluster/simple unexpected error while reconciling control plane endpoint for simple: failed to reconcile loadbalanced endpoint for WCPCluster tanzu-ns-01/simple: failed to get control plane endpoint for Cluster tanzu-ns-01/simple: VirtualMachineService LB does not yet have VIP assigned: VirtualMachineService LoadBalancer does not have any Ingresses
0s Normal Reconcile gateway/simple-control-plane-service Success
0s Normal Reconcile gateway/simple-control-plane-service Success
0s Normal SuccessfulCreate machinedeployment/simple-workers-kszj2 Created MachineSet "simple-workers-kszj2-d4c6b6f49"
0s Normal SuccessfulCreate machineset/simple-workers-kszj2-d4c6b6f49 Created machine "simple-workers-kszj2-d4c6b6f49-m6sg9"
0s Normal SuccessfulCreate machineset/simple-workers-kszj2-d4c6b6f49 Created machine "simple-workers-kszj2-d4c6b6f49-795l5"
0s Normal SuccessfulCreate machineset/simple-workers-kszj2-d4c6b6f49 Created machine "simple-workers-kszj2-d4c6b6f49-5z5zg"
0s Warning ReconcileFailure wcpmachine/simple-control-plane-hfrm7-7twzk vm is not yet created: vmware-system-capw-controller-manager/WCPMachine/infrastructure.cluster.vmware.com/v1alpha3/tanzu-ns-01/simple/simple-control-plane-hfrm7-7twzk
0s Warning ReconcileFailure wcpmachine/simple-control-plane-hfrm7-7twzk vm is not yet created: vmware-system-capw-controller-manager/WCPMachine/infrastructure.cluster.vmware.com/v1alpha3/tanzu-ns-01/simple/simple-control-plane-hfrm7-7twzk
0s Warning ReconcileFailure wcpmachine/simple-control-plane-hfrm7-7twzk vm is not yet created: vmware-system-capw-controller-manager/WCPMachine/infrastructure.cluster.vmware.com/v1alpha3/tanzu-ns-01/simple/simple-control-plane-hfrm7-7twzk
0s Warning ReconcileFailure wcpmachine/simple-control-plane-hfrm7-7twzk vm is not yet created: vmware-system-capw-controller-manager/WCPMachine/infrastructure.cluster.vmware.com/v1alpha3/tanzu-ns-01/simple/simple-control-plane-hfrm7-7twzk
0s Warning ReconcileFailure wcpmachine/simple-control-plane-hfrm7-7twzk vm is not yet created: vmware-system-capw-controller-manager/WCPMachine/infrastructure.cluster.vmware.com/v1alpha3/tanzu-ns-01/simple/simple-control-plane-hfrm7-7twzk
0s Warning ReconcileFailure wcpmachine/simple-control-plane-hfrm7-7twzk vm is not yet created: vmware-system-capw-controller-manager/WCPMachine/infrastructure.cluster.vmware.com/v1alpha3/tanzu-ns-01/simple/simple-control-plane-hfrm7-7twzk
0s Warning ReconcileFailure wcpmachine/simple-control-plane-hfrm7-7twzk vm is not yet created: vmware-system-capw-controller-manager/WCPMachine/infrastructure.cluster.vmware.com/v1alpha3/tanzu-ns-01/simple/simple-control-plane-hfrm7-7twzk
1s Warning ReconcileFailure wcpmachine/simple-control-plane-hfrm7-7twzk vm is not yet created: vmware-system-capw-controller-manager/WCPMachine/infrastructure.cluster.vmware.com/v1alpha3/tanzu-ns-01/simple/simple-control-plane-hfrm7-7twzk
0s Warning ReconcileFailure wcpmachine/simple-control-plane-hfrm7-7twzk vm is not yet created: vmware-system-capw-controller-manager/WCPMachine/infrastructure.cluster.vmware.com/v1alpha3/tanzu-ns-01/simple/simple-control-plane-hfrm7-7twzk
0s Warning ReconcileFailure wcpmachine/simple-control-plane-hfrm7-7twzk vm is not yet created: vmware-system-capw-controller-manager/WCPMachine/infrastructure.cluster.vmware.com/v1alpha3/tanzu-ns-01/simple/simple-control-plane-hfrm7-7twzk
0s Warning ReconcileFailure wcpmachine/simple-control-plane-hfrm7-7twzk vm is not yet created: vmware-system-capw-controller-manager/WCPMachine/infrastructure.cluster.vmware.com/v1alpha3/tanzu-ns-01/simple/simple-control-plane-hfrm7-7twzk
0s Warning ReconcileFailure wcpmachine/simple-control-plane-hfrm7-7twzk vm is not yet created: vmware-system-capw-controller-manager/WCPMachine/infrastructure.cluster.vmware.com/v1alpha3/tanzu-ns-01/simple/simple-control-plane-hfrm7-7twzk
0s Warning ReconcileFailure wcpmachine/simple-control-plane-hfrm7-7twzk vm is not yet created: vmware-system-capw-controller-manager/WCPMachine/infrastructure.cluster.vmware.com/v1alpha3/tanzu-ns-01/simple/simple-control-plane-hfrm7-7twzk
0s Warning ReconcileFailure wcpmachine/simple-control-plane-hfrm7-7twzk vm is not yet created: vmware-system-capw-controller-manager/WCPMachine/infrastructure.cluster.vmware.com/v1alpha3/tanzu-ns-01/simple/simple-control-plane-hfrm7-7twzk
0s Warning ReconcileFailure wcpmachine/simple-control-plane-hfrm7-7twzk vm is not yet created: vmware-system-capw-controller-manager/WCPMachine/infrastructure.cluster.vmware.com/v1alpha3/tanzu-ns-01/simple/simple-control-plane-hfrm7-7twzk
0s Warning ReconcileFailure wcpmachine/simple-control-plane-hfrm7-7twzk vm does not have an IP address: vmware-system-capw-controller-manager/WCPMachine/infrastructure.cluster.vmware.com/v1alpha3/tanzu-ns-01/simple/simple-control-plane-hfrm7-7twzk
0s Warning ReconcileFailure wcpmachine/simple-control-plane-hfrm7-7twzk vm does not have an IP address: vmware-system-capw-controller-manager/WCPMachine/infrastructure.cluster.vmware.com/v1alpha3/tanzu-ns-01/simple/simple-control-plane-hfrm7-7twzk
0s Warning ReconcileFailure wcpmachine/simple-control-plane-hfrm7-7twzk vm does not have an IP address: vmware-system-capw-controller-manager/WCPMachine/infrastructure.cluster.vmware.com/v1alpha3/tanzu-ns-01/simple/simple-control-plane-hfrm7-7twzk
0s Warning ReconcileFailure wcpmachine/simple-control-plane-hfrm7-7twzk vm does not have an IP address: vmware-system-capw-controller-manager/WCPMachine/infrastructure.cluster.vmware.com/v1alpha3/tanzu-ns-01/simple/simple-control-plane-hfrm7-7twzk
0s Normal Reconcile gateway/simple-control-plane-service Success
0s Normal Reconcile gateway/simple-control-plane-service Success
0s Normal Reconcile gateway/simple-control-plane-service Success
0s Normal Reconcile gateway/simple-control-plane-service Success
1s Normal Reconcile gateway/simple-control-plane-service Success

 

Cluster API Status:
API Endpoints:
Host: 172.16.97.194
Port: 6443
Phase: Provisioned
Node Status:
simple-control-plane-rb7r6: pending
simple-workers-kszj2-d4c6b6f49-5z5zg: pending
simple-workers-kszj2-d4c6b6f49-795l5: pending
simple-workers-kszj2-d4c6b6f49-m6sg9: pending
Phase: creating
Vm Status:
simple-control-plane-rb7r6: ready
simple-workers-kszj2-d4c6b6f49-5z5zg: pending
simple-workers-kszj2-d4c6b6f49-795l5: pending
simple-workers-kszj2-d4c6b6f49-m6sg9: pending

 

Any ideas?

 

0 Kudos
3 Replies
Enthusiast
Enthusiast

---

0 Kudos
Enthusiast
Enthusiast

--

0 Kudos
Enthusiast
Enthusiast

Hi, with vSphere w Tanzu using HAProxy Im stucked during TKG cluster deployment. kubectl describe kubeadmcontrolplanes.controlplane.cluster.x-k8s.io tkg-show-01-control-plane is showing me WaitingForKubeadmInit. Anyone? Here is configuration:

ferdis_2-1606634959238.png

HAProxy configuration

 

ferdis_3-1606634970106.png

Frontend CIDR setting in HAProxy

 

Here is Workload Management part

 

ferdis_5-1606635020229.png

 

ferdis_6-1606635029547.png

ferdis_7-1606635049710.png

ferdis_8-1606635060482.png

ferdis_9-1606635068281.png

 

This part is describing state after I initiated TKG guest cluster creation:

 

ferdis_0-1606635449742.png

kubectl describe kubeadmcontrolplanes.controlplane.cluster.x-k8s.io tkg-cluster-1-control-plane

 

ferdis_0-1606636826134.png

ferdis_1-1606637148593.png

kubectl get tanzukubernetescluster,clusters.cluster.x-k8s.io,wcpcluster,machinedeployment.cluster.x-k8s.io,wcpmachinetemplate,machine.cluster.x-k8s.io,wcpmachine,kubeadmconfig,virtualmachinesetresourcepolicies,virtualmachines,virtualmachineservices,configmaps,secrets

ferdis_2-1606635834380.png

from Supervisor VM I can ping Workload Network GW, also can ping HAProxy Workload IP. But cant ping TKG Cluster IP on Workload Network.
SSH to HAProxy IP on Workload Network is refused.

ferdis_0-1606636024126.png

From TKG Control Plane VM I can ping Workload Network GW but cant ping HAProxy Workload IP. Also cant ping Supervisor IP on Workload Network.
SSH to HAProxy IP on Workload Network is not responding at all.

ferdis_0-1606636110202.png

From TKG Control Plane VM on Workload Network I can ping and SSH Supervisor VM IP on Management Network. Also can ping Supervisor Cluster IP On Frontend Network.
Also can ping HAProxy IP on Management Network also can ping HAProxy IP on Frontend Network.

ferdis_0-1606636329881.png

 

 

 

0 Kudos