VMware Cloud Director Container Service Extension Discussions topics

Failed deployments using TKGm on VCD

xeon22 — Mon, 17 Jul 2023 13:46:34 GMT

Detailed Error: [error while bootstrapping the machine [/EPHEMERAL-TEMP-VM]; unable to wait for post customization phase [guestinfo.cloudinit.antrea.manifest.download.status] : [invalid postcustomization phase: [failed] for key [guestinfo.cloudinit.antrea.manifest.download.status] for vm [EPHEMERAL-TEMP-VM] due to :[<13>Jul 17 12:44:36 test1@System/administrator: + wget -O /root/v1.6.1-antrea.yaml.template <13>Jul 17 12:44:36 test1@System/administrator: --2023-07-17 12:44:36-- <13>Jul 17 12:44:36 test1@System/administrator: Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 185.199.109.133, 185.199.111.133, 185.199.108.133, ... <13>Jul 17 12:44:36 test1@System/administrator: Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|185.199.109.133|:443... connected. <13>Jul 17 12:44:36 test1@System/administrator: HTTP request sent, awaiting response...

ScriptExecutionError While deploying tkgm cluster

Cambouch — Fri, 20 Jan 2023 13:53:12 GMT

Hello,

I am facing a ScriptExecution Error while trying to deploy a tkgm cluster.

The deployment fails with a an error "unable to wait for post customization phase ". I tried to access the control plane VM "EPHEMERAL_TEMP_VM" to figure out the problem and i noticed that there are 4 services "Cloud-init,Cloud-init-local,cloud-final,cloud-config" that are not running as attached in the screenshot.

I am using CSE 4.0 for vCloud Director 10.4, TKG 1.4.3 and 1.5.4(Which are supported by cse 4.0) and Ubuntu as TKG OS no FIPS tempates downloaded from official product as required in the documentation.

I tried to modify manually the template and i successfully made all services running. Then, i tried to upload the template to my catalogue to use it as my TKG cluster template but it's not available among templates options.

So, the problem resides especially in the ubuntu template and i am not allowed to modify it to make all services running. Is there any workaround that i can try to solve this problem.

Thanks in advance.

Feature request: Select static IP for load balancer.

ccalvetbeta — Wed, 14 Sep 2022 13:41:17 GMT

Hi,
Now, when exposing a service via load balancer from kubernetes it will pick the first IP address available in the first "IP Allocations" with IP available of the associated edge.
It would help in disaster recovery scenario to be able to define which IP will be used.
Similar to: Use static IP with load balancer - Azure Kubernetes Service | Microsoft Docs

Is is already planned to provide such capability in Tanzu CSE?
If yes, when will it be available?
If not, this is a feature request.

authorize every minute

rickvanvliet — Wed, 14 Sep 2022 08:37:39 GMT

In our VCD event logs we see this message every minute:
User 'clusterauthor' (83137b58-3506-416b-a31d-e68962bce07b) authorize

together with this event:

OAuth token created for client ${oauthToken.clientName}(${oauthToken.clientId})

Event Id:

urn:vcloud:audit:f06957fc-5c8d-4dc6-8747-af8d88181f86

Type:

token

This resulted in the VCD database filling up the disk which resulted in VCD stopping.

Are other having the same issue, or is this a known issue?

It is possible to deploy cluster in a "Routed Data Center Group Network"?

ccalvetbeta — Wed, 31 Aug 2022 14:35:01 GMT

Hi, so far I have only deployed the legacy and beta CSE clusters in "Routed Organization Virtual Data Center Network".
It works.
Is is supported/tested to deploy in a "Routed Data Center Group Network"?
Goal is to see if distributed firewall could be used to isolate clusters deployed in different networks.

Node auto-healing feature of cluster created by CSE 4.0

cchen2 — Fri, 19 Aug 2022 00:31:51 GMT

I am testing CSE 4.0 for my customer.
Does the TKG Cluster created by CSE 4.0 have node auto-healing ability? (the one offered by cluster API, when nodes is done, new node can be automatically deployed and merge into the cluster)

I understand that the cluster is CAPVCD self-managed. As a feature of CAPVCD, the node auto-healing should work.

However, I tried to stop worker node / control plane node, found that nothing happened (No new nodes are created, the nodes stay in "unavailable" status. (I also tried to restart "RDEprojector", and still nothing happened)

I want to know if the node auto-healing feature is supposed to be available for cluster created by CSE 4.0

The default wait time for CreateKindVM and PostCustomizationPhases is too short

steven0301 — Wed, 17 Aug 2022 09:07:31 GMT

The default wait time for CreateKindVM is 180 seconds, if the time taken more than that, the creation of cluster will be blocked or panic. In my env (a cassini testbed), the process of CreateKindVM took about 480 seconds. To make it work, I have to modify the source code to increase the default value and build a temp version to replace the binary in the CSE server VM.

The same issue for ClusterCreatePostCustomizationPhases of "guestinfo.cloudinit.target.cluster.ready.status", the default wait time is 1800 seconds, but in my env, it could take more than that time - it took about 1500 seconds in one run, and took more than 1800 seconds in another run.

Can you increase the default values for these two items or make them configurable?

Node Pool labels for worker node

cchen2 — Mon, 15 Aug 2022 03:05:50 GMT

I've successfully deployed a cluster with 2 worker node pools (worker-pool-1 & worker-pool-2)

From a k8s user perspective, when I want to deploy a pod to a specified worker pool (e.x. worker-pool-2), how should I write the yaml?

For example, in case of GKE, user can use nodeSelector to specify the node pool like below:

  nodeSelector:
    cloud.google.com/gke-nodepool: POOL_NAME

And the gke-nodepool label is automatically added during the creation.

However, when checking the label of TKG cluster created by CSE 4.0, I didn't find any node pool related labels.

Of course users can label by themself, but it may be easier to use if the node pool label can be automatically added.

administrator@client:~$ k describe node cse4-cluster-01-worker-pool-2-5694c75b78-kjr9l
Name: cse4-cluster-01-worker-pool-2-5694c75b78-kjr9l
Roles: <none>
Labels: beta.kubernetes.io/arch=amd64
beta.kubernetes.io/os=linux
kubernetes.io/arch=amd64
kubernetes.io/hostname=cse4-cluster-01-worker-pool-2-5694c75b78-kjr9l
kubernetes.io/os=linux

Deployment failure: unable to wait for post customization phase

jleavers1 — Mon, 08 Aug 2022 19:55:00 GMT

Deployments are failing with the following error:

Aug 08 19:44:20 photon bash[2574]: {"level":"info","ts":1659987860.6753228,"caller":"repair/heartbeat.go:102","msg":"updating component [{errorSet {RecoveryError 0001-01-01 00:00:00 +0000 UTC urn:vcloud:entity:vmware:capvcdCluster:8319ce29-cba5-4f65-8f20-46d0207cb386 map[]} true}] in RDE: [test-1(urn:vcloud:entity:vmware:capvcdCluster:8319ce29-cba5-4f65-8f20-46d0207cb386)]"}

Aug 08 19:44:20 photon bash[2574]: {"level":"error","ts":1659987860.6754072,"caller":"app/main.go:584","msg":"error creating cluster [test-1(urn:vcloud:entity:vmware:capvcdCluster:8319ce29-cba5-4f65-8f20-46d0207cb386)] : [error while bootstrapping the machine [test-1/EPHEMERAL_TEMP_VM]; unable to wait for post customization phase [guestinfo.cloudinit.kind.cluster.creation.status] : [invalid postcustomization phase: [failed] for key [guestinfo.cloudinit.kind.cluster.creation.status] for vm [EPHEMERAL_TEMP_VM]]]","stacktrace":"main.processRDE\n\t/app/main.go:584"}

Deployment failure: "unable to wait for post customization phase"

jleavers1 — Mon, 08 Aug 2022 19:51:35 GMT

Deployment is failing with following shown in journalctl output:

Aug 08 19:43:38 photon bash[2574]: {"level":"info","ts":1659987818.9050834,"caller":"utils/rdeUtils.go:210","msg":"Assigning value [Elpased time waiting for postcustomization status [guestinfo.cloudinit.kind.cluster.creation.status] to complete [0.240771] seconds] to key [HeartbeatString] in [types.VCDKEStatus]"}
Aug 08 19:43:38 photon bash[2574]: {"level":"info","ts":1659987818.939308,"caller":"utils/rdeUtils.go:313","msg":"successfully updated defined entity with ID [urn:vcloud:entity:vmware:capvcdCluster:8319ce29-cba5-4f65-8f20-46d0207cb386]"}
Aug 08 19:43:49 photon bash[2574]: {"level":"error","ts":1659987829.103915,"caller":"cluster/clusterManager.go:238","msg":"error waiting for creation of cluster [test-1(urn:vcloud:entity:vmware:capvcdCluster:8319ce29-cba5-4f65-8f20-46d0207cb386)]: [error while bootstrapping the machine [test-1/EPHEMERAL_TEMP_VM]; unable to wait for post customization phase [guestinfo.cloudinit.kind.cluster.creation.status] : [invalid postcustomization phase: [failed] for key [guestinfo.cloudinit.kind.cluster.creation.status] for vm [EPHEMERAL_TEMP_VM]]]","stacktrace":"gitlab.eng.vmware.com/core-build/vcd-k8s-provider/src/cluster.CreateCluster\n\t/app/src/cluster/clusterManager.go:238\nmain.processRDE\n\t/app/main.go:566"}
Aug 08 19:43:49 photon bash[2574]: {"level":"info","ts":1659987829.1040707,"caller":"repair/heartbeat.go:102","msg":"updating component [{errorSet {ScriptExecutionError 2022-08-08 19:43:49.103954454 +0000 UTC m=+6717.591647529 urn:vcloud:vm:4fd80e53-7307-43fc-b74f-274f33aabb5e map[Detailed Error:[error while bootstrapping the machine [test-1/EPHEMERAL_TEMP_VM]; unable to wait for post customization phase [guestinfo.cloudinit.kind.cluster.creation.status] : [invalid postcustomization phase: [failed] for key [guestinfo.cloudinit.kind.cluster.creation.status] for vm [EPHEMERAL_TEMP_VM]]] during cluster creation]} false}] in RDE: [test-1(urn:vcloud:entity:vmware:capvcdCluster:8319ce29-cba5-4f65-8f20-46d0207cb386)]"}

Is it possible to use cluster api in the beta?

ccalvetbeta — Mon, 08 Aug 2022 14:10:25 GMT

After a successful deployment in the gui, is it possible to do the same with cluster API? Goal would be to automatize all deployment.
Like in Cluster API Provider for VMware Cloud Director - VMware Cloud Provider Blog

For example could cluster API be used to edit number of nodes, create new worker nodes etc.?

CSE with Multisite

marko_uskokovic — Mon, 08 Aug 2022 10:28:06 GMT

I have tried CSE.next on multisite setup, where I have two VCD sites paired on system and organization level.

So, sites are rs-bg-1-ec and rs-bg-2-ec and I can see VDCs and VMs from both sites in VCD.

Also, in CSE UI plugin on rs-bg-2-ec site I can see k8s clusters from both sites:

If I click on cse4-rs-bg-1-test-1 link, it will open rs-bg-1-ec site in new browser tab, but it will not accept my current SSO login and I will need to login again to manage my cluster. Annoyance, but bearable.

The real problem is, if I login to rs-bg-2-ec site, and go to another site by SSO, and then open CSE UI plugin, it will not be able to show any clusters, it will display message "Error: Failed to fetch Kubernetes clusters"

There is 401 Unauthorized HTTP request to rs-bg-1-ec site's /api/session url and there is a "CSE UI: Error fetching sites" message in browser console.

Note: Similar problems were noticed in CSE 3.1 when using multisite setup.

Any plan regarding how to operate multi-node etcd cluster?

ccalvetbeta — Mon, 08 Aug 2022 10:17:21 GMT

The new capability to create multiples nodes for the control plane is good.
However how should they be operated?

I am thinking of etcd
https://kubernetes.io/docs/tasks/administer-cluster/configure-upgrade-etcd/

For example:

What would be the process to replace a failed ectd member?
Removing a failed member doesn't seem possible if they are managed by Tanzu.
I didn't see an option in the gui to delete a specific node.

Will their be an option to backup etcd directly from the GUI or via CLI?

Load balancer rule is not being re-created if needed

marko_uskokovic — Mon, 08 Aug 2022 10:07:58 GMT

I accidentally deleted load balancer rule for 6443 port, and was hoping that CSE will re-create it because it is needed for cluster to work correctly, but that does not happen.

Is that load balancer supposed to be managed by CSE and re-created or modified as needed?

Node count is not correct

marko_uskokovic — Mon, 08 Aug 2022 10:06:01 GMT

Sometimes it happens that node count is not correct even after cluster progresses to Ready state - different values are shown in CSE plugin than it is visible in kubectl command and it stays like that even afer 48h

Storage class not visible in "Kubernetes Storage" screen

marko_uskokovic — Mon, 08 Aug 2022 10:02:51 GMT

After provisioning of CSE.next cluster, I got storage class created:

kubectl --kubeconfig cse4-2 get storageclasses.storage.k8s.io
NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE
default-storage-class-1 (default) named-disk.csi.cloud-director.vmware.com Delete Immediate false 5h1m

But it is not visible in CSE plugin, it just shows N/A

Host SSH key changes after reboot

marko_uskokovic — Mon, 08 Aug 2022 09:59:58 GMT

I have tried to access control plane and worker nodes after provisioning (but I am unable to login because SSH public key I entered during provisioning is not configured on nodes which I described in another thread) so my host SSH key is recorded in my client for future use.

But after rebooting the node, host SSH key is changed:

[me@home ~]$ ssh -p 22 root@172.16.172.16
root@172.16.172.16's password:
Permission denied, please try again.
root@172.16.172.16's password:

reboot happens here

[me@home ~]$ ssh -p 22 root@172.16.172.16
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@ WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that a host key has just been changed.

Is that supposed to happen or is it because cloud-init is failing to configure nodes?

I don't think it is a good thing to change host SSH key every time node is restarted...

Can't login to nodes and cloud-init errors after cluster provisioning

marko_uskokovic — Mon, 08 Aug 2022 09:51:53 GMT

After successful cluster provisioning (State: Ready) I can not login to any node using SSH public key I provided during setup of cluster.

Also, I am not able to login to nodes using password that VCD autogenerated in "Guest OS Customization" (nor with manually specified password).

Since I can not login to nodes I can't see logs of cluster provisioning. Only thing I can see are cloud-init errors after boot:

control-planecontrol-plane-after-rebootworkerworker-after-reboot

Process stuck during upgrading

cchen2 — Mon, 08 Aug 2022 07:49:59 GMT

I've successfully deployed cluster via CSE 4.0 and tried to upgrade from 1.21.8 to 1.22.9.

After submitting the upgrade request via GUI, the upgrade didn't kick off. Then I restarted the rdeprojector pod, and the upgrade process started.

However, after waiting for over 40 minutes, I found that although the version of control nodes was successfully upgraded to 1.22.9, but the worker nodes stuck in 1.21.8 and an additional worker node was added (before upgrade 2 worker nodes, now 3 worker nodes) and stuck in process status (checking in cluster confiug api)

I suppose that the upgrade process stuck, and want to know what may be the trigger and how to fix.

** Some Advice **

1. It will be better if GUI can show the progress of the entire upgrade process, or at least show if the upgrade process is finished or under processing. In current version, it is confusing and hard to tell.

2. I noticed that the rolling update is done with maxSurge > 0, it means that cluster API will create more temp nodes for updating. For those resource sensitive tenant, it may be better to offer an option to config the maxSurge manually.

ScriptExecutionError while deploying a cluster

steven0301 — Mon, 08 Aug 2022 07:22:36 GMT

The cluster events show ScriptExecutionError and the cluster is in "NOT Ready" status for ever.

Found some errors in the logs.

E0808 03:26:47.018187 1 controller.go:188] controller/kubeadmcontrolplane "msg"="Failed to update KubeadmControlPlane Status" "error"="failed to create remote cluster client: failed to retrieve kubeconfig secret for Cluster capvcd-cluster3-ns/capvcd-cluster3: secrets \"capvcd-cluster3-kubeconfig\" not found" "cluster"="capvcd-cluster3" "name"="capvcd-cluster3-control-plane" "namespace"="capvcd-cluster3-ns" "reconciler group"="controlplane.cluster.x-k8s.io" "reconciler kind"="KubeadmControlPlane"

The log bundle is attached.

Could some help to take a look?