marko_uskokovic Tracker

Re: CSE with Multisite

marko_uskokovic — Fri, 26 Aug 2022 15:18:14 GMT

I don't see any specific error, because it opens in new tab without inspect element so I can't track http requests. Will report if I find some error.

When I login to rs-bg-2-ec site and want to go to another site by SSO I just click on site name in "Data Centers" -> "Virtual Data Center" like in picture attached

Re: Node count is not correct

marko_uskokovic — Tue, 09 Aug 2022 12:22:48 GMT

Refreshed it a couple of times during a period of 2 days that cluster existed, always showed wrong number of nodes. Cluster is now gone so I will report if it happens again on another cluster.

CSE with Multisite

marko_uskokovic — Mon, 08 Aug 2022 10:28:06 GMT

I have tried CSE.next on multisite setup, where I have two VCD sites paired on system and organization level.

So, sites are rs-bg-1-ec and rs-bg-2-ec and I can see VDCs and VMs from both sites in VCD.

Also, in CSE UI plugin on rs-bg-2-ec site I can see k8s clusters from both sites:

If I click on cse4-rs-bg-1-test-1 link, it will open rs-bg-1-ec site in new browser tab, but it will not accept my current SSO login and I will need to login again to manage my cluster. Annoyance, but bearable.

The real problem is, if I login to rs-bg-2-ec site, and go to another site by SSO, and then open CSE UI plugin, it will not be able to show any clusters, it will display message "Error: Failed to fetch Kubernetes clusters"

There is 401 Unauthorized HTTP request to rs-bg-1-ec site's /api/session url and there is a "CSE UI: Error fetching sites" message in browser console.

Note: Similar problems were noticed in CSE 3.1 when using multisite setup.

Load balancer rule is not being re-created if needed

marko_uskokovic — Mon, 08 Aug 2022 10:07:58 GMT

I accidentally deleted load balancer rule for 6443 port, and was hoping that CSE will re-create it because it is needed for cluster to work correctly, but that does not happen.

Is that load balancer supposed to be managed by CSE and re-created or modified as needed?

Node count is not correct

marko_uskokovic — Mon, 08 Aug 2022 10:06:01 GMT

Sometimes it happens that node count is not correct even after cluster progresses to Ready state - different values are shown in CSE plugin than it is visible in kubectl command and it stays like that even afer 48h

Storage class not visible in "Kubernetes Storage" screen

marko_uskokovic — Mon, 08 Aug 2022 10:02:51 GMT

After provisioning of CSE.next cluster, I got storage class created:

kubectl --kubeconfig cse4-2 get storageclasses.storage.k8s.io
NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE
default-storage-class-1 (default) named-disk.csi.cloud-director.vmware.com Delete Immediate false 5h1m

But it is not visible in CSE plugin, it just shows N/A

Host SSH key changes after reboot

marko_uskokovic — Mon, 08 Aug 2022 09:59:58 GMT

I have tried to access control plane and worker nodes after provisioning (but I am unable to login because SSH public key I entered during provisioning is not configured on nodes which I described in another thread) so my host SSH key is recorded in my client for future use.

But after rebooting the node, host SSH key is changed:

[me@home ~]$ ssh -p 22 root@172.16.172.16
root@172.16.172.16's password:
Permission denied, please try again.
root@172.16.172.16's password:

reboot happens here

[me@home ~]$ ssh -p 22 root@172.16.172.16
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@ WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that a host key has just been changed.

Is that supposed to happen or is it because cloud-init is failing to configure nodes?

I don't think it is a good thing to change host SSH key every time node is restarted...

Can't login to nodes and cloud-init errors after cluster provisioning

marko_uskokovic — Mon, 08 Aug 2022 09:51:53 GMT

After successful cluster provisioning (State: Ready) I can not login to any node using SSH public key I provided during setup of cluster.

Also, I am not able to login to nodes using password that VCD autogenerated in "Guest OS Customization" (nor with manually specified password).

Since I can not login to nodes I can't see logs of cluster provisioning. Only thing I can see are cloud-init errors after boot:

control-planecontrol-plane-after-rebootworkerworker-after-reboot

Re: CSE 3.1.3 and CSE 4 beta co-existence

marko_uskokovic — Mon, 08 Aug 2022 09:41:30 GMT

At one point, in beta version plugin, I got screen like this (not a real screenshot from that time, this is a mockup):

I was able to proceed dialog with first option which gave me CSE.next cluster, but second option showed no available vdc where to provision CSE 3.1.

I only saw that screen once, all later attempts showed only first option available, so I am wondering what was going on that time.

Or it should work that way, but it doesn't work on my side?

Re: ScriptInitError when creating cluster (during EPHEMERAL_TEMP_VM creating)

marko_uskokovic — Tue, 02 Aug 2022 18:02:46 GMT

Tried with 10.3.3.19780585, same thing happened, which lead me to check logs again and I found

"Missing right to use ExtraConfig guestinfo.userdata.encoding."

Then I figured out that I needed

"Preserve All ExtraConfig Elements During OVF Import and Export" in my "Default rights bundle", not (only) in "Organization admin" role.

EPHEMERAL_TEMP_VM now boots (but cluster is in state Reconciling, so I'm checking it out).

Re: ScriptInitError when creating cluster (during EPHEMERAL_TEMP_VM creating)

marko_uskokovic — Tue, 02 Aug 2022 14:33:13 GMT

Will try with VCD 10.3.3, didn't check requirements before trying

ScriptInitError when creating cluster (during EPHEMERAL_TEMP_VM creating)

marko_uskokovic — Tue, 02 Aug 2022 13:38:05 GMT

This is my first try to provision cse.next k8s cluster, and it fails while creating EPHEMERAL_TEMP_VM with ScriptInitError and later with EphemeralVMError, and It doesn't even start.

VMware Cloud Director version:

10.3.2.19473806

I have attached whole cse log from

journalctl -lu cse