VMware Cloud Community
apatil1
Contributor
Contributor

TKG 1.5.2 - Management cluster deployment failing -

photon-3-kube-v1.22.5+vmware.1
NSX ALB : photon-3-kube-v1.22.5+vmware.1

tanzu mc create --ui 
starts deployment of 1 Management cluster node but it stucks forever.

 

apatil@apatil--MacBookPro16 cli % kubectl get deployment.apps -A --kubeconfig /Users/apatil/.kube-tkg/tmp/config_Z06hUbj9
NAMESPACE                           NAME                                            READY   UP-TO-DATE   AVAILABLE   AGE
capi-kubeadm-bootstrap-system       capi-kubeadm-bootstrap-controller-manager       1/1     1            1           34m
capi-kubeadm-control-plane-system   capi-kubeadm-control-plane-controller-manager   1/1     1            1           34m
capi-system                         capi-controller-manager                         1/1     1            1           34m
capv-system                         capv-controller-manager                         1/1     1            1           34m
cert-manager                        cert-manager                                    1/1     1            1           35m
cert-manager                        cert-manager-cainjector                         1/1     1            1           35m
cert-manager                        cert-manager-webhook                            1/1     1            1           35m
kube-system                         coredns                                         2/2     2            2           35m
local-path-storage                  local-path-provisioner                          1/1     1            1           35m
tkg-system                          ako-operator-controller-manager                 0/1     1            0           34m

 

 
ako-operator-controller-manager deployment failing to come up.

apatil@apatil--MacBookPro16 cli % kubectl logs deployment.apps/ako-operator-controller-manager -n tkg-system --kubeconfig /Users/apatil/.kube-tkg/tmp/config_Z06hUbj9 manager
I0325 05:50:33.680369       1 request.go:665] Waited for 1.000163495s due to client-side throttling, not priority and fairness, request: GET:https://100.64.0.1:443/apis/flowcontrol.apiserver.k8s.io/v1beta1?timeout=32s
2022-03-25T05:50:33.786Z	INFO	controller-runtime.metrics	metrics server is starting to listen	{"addr": "127.0.0.1:8080"}
2022-03-25T05:50:33.786Z	INFO	starting manager
2022-03-25T05:50:33.878Z	INFO	starting metrics server	{"path": "/metrics"}
2022-03-25T05:50:33.878Z	INFO	controller.cluster	Starting EventSource	{"reconciler group": "cluster.x-k8s.io", "reconciler kind": "Cluster", "source": "kind source: /, Kind="}
2022-03-25T05:50:33.878Z	INFO	controller.cluster	Starting EventSource	{"reconciler group": "cluster.x-k8s.io", "reconciler kind": "Cluster", "source": "kind source: /, Kind="}
2022-03-25T05:50:33.878Z	INFO	controller.cluster	Starting Controller	{"reconciler group": "cluster.x-k8s.io", "reconciler kind": "Cluster"}
2022-03-25T05:50:33.880Z	INFO	controller.configmap	Starting EventSource	{"reconciler group": "", "reconciler kind": "ConfigMap", "source": "kind source: /, Kind="}
2022-03-25T05:50:33.880Z	INFO	controller.configmap	Starting Controller	{"reconciler group": "", "reconciler kind": "ConfigMap"}
2022-03-25T05:50:33.879Z	INFO	controller.machine	Starting EventSource	{"reconciler group": "cluster.x-k8s.io", "reconciler kind": "Machine", "source": "kind source: /, Kind="}
2022-03-25T05:50:33.880Z	INFO	controller.machine	Starting EventSource	{"reconciler group": "cluster.x-k8s.io", "reconciler kind": "Machine", "source": "kind source: /, Kind="}
2022-03-25T05:50:33.880Z	INFO	controller.machine	Starting Controller	{"reconciler group": "cluster.x-k8s.io", "reconciler kind": "Machine"}
2022-03-25T05:50:34.082Z	INFO	controller.machine	Starting workers	{"reconciler group": "cluster.x-k8s.io", "reconciler kind": "Machine", "worker count": 1}
2022-03-25T05:50:34.084Z	INFO	controller.configmap	Starting workers	{"reconciler group": "", "reconciler kind": "ConfigMap", "worker count": 1}
2022-03-25T05:50:34.177Z	INFO	controller.cluster	Starting workers	{"reconciler group": "cluster.x-k8s.io", "reconciler kind": "Cluster", "worker count": 1}
2022-03-25T05:50:34.180Z	INFO	controllers.Machine	not a control plane machine, skip	{"Machine": "tkg-system/tkg-mgmt-cluster-md-0-5c8b75dcf8-c4pj7"}
2022-03-25T05:50:34.180Z	INFO	controllers.Cluster	AVI is control plane HA provider	{"Cluster": "tkg-system/tkg-mgmt-cluster"}
2022-03-25T05:50:34.277Z	INFO	controllers.Machine	not a control plane machine, skip	{"Machine": "tkg-system/tkg-mgmt-cluster-md-0-5c8b75dcf8-c4pj7"}
2022-03-25T05:50:34.278Z	INFO	controllers.Machine	not a control plane machine, skip	{"Machine": "tkg-system/tkg-mgmt-cluster-md-0-5c8b75dcf8-c4pj7"}
2022-03-25T05:50:34.379Z	INFO	controllers.Machine	machine is in Endpoints Object, skip	{"Machine": "tkg-system/tkg-mgmt-cluster-md-0-5c8b75dcf8-c4pj7"}
2022-03-25T05:50:34.382Z	INFO	controllers.Machine	not a control plane machine, skip	{"Machine": "tkg-system/tkg-mgmt-cluster-md-0-5c8b75dcf8-c4pj7"}
2022-03-25T05:50:34.389Z	INFO	controllers.ConfigMap	Start reconciling AVI cloud usable network in bootstrap cluster	{"ConfigMap": "tkg-system/avi-k8s-config"}
2022-03-25T05:50:34.842Z	INFO	controllers.ConfigMap	AVI Client initialized successfully	{"ConfigMap": "tkg-system/avi-k8s-config"}
2022-03-25T05:50:35.159Z	DPANIC	controllers.ConfigMap	odd number of arguments passed as key-value pairs for logging	{"ConfigMap": "tkg-system/avi-k8s-config", "ignored key": "d2-117-tkg-workload"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.10.2/pkg/internal/controller/controller.go:114
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.10.2/pkg/internal/controller/controller.go:311
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.10.2/pkg/internal/controller/controller.go:266
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.10.2/pkg/internal/controller/controller.go:227
panic: odd number of arguments passed as key-value pairs for logging

goroutine 266 [running]:
go.uber.org/zap/zapcore.(*CheckedEntry).Write(0xc0005f6240, 0xc0006d8880, 0x1, 0x1)
	/go/pkg/mod/go.uber.org/zap@v1.19.0/zapcore/entry.go:232 +0x532
go.uber.org/zap.(*Logger).DPanic(0xc0002c57a0, 0x1a9db0e, 0x3d, 0xc0006d8880, 0x1, 0x1)
	/go/pkg/mod/go.uber.org/zap@v1.19.0/logger.go:220 +0x85
github.com/go-logr/zapr.handleFields(0xc0002c57a0, 0xc000611f60, 0x1, 0x1, 0x0, 0x0, 0x0, 0x10, 0x1779880, 0xc00064eb01)
	/go/pkg/mod/github.com/go-logr/zapr@v0.4.0/zapr.go:100 +0x5e5
github.com/go-logr/zapr.(*zapLogger).Info(0xc0002174c0, 0x1a8c697, 0x34, 0xc000611f60, 0x1, 0x1)
	/go/pkg/mod/github.com/go-logr/zapr@v0.4.0/zapr.go:127 +0xb0
github.com/vmware-samples/load-balancer-operator-for-kubernetes/controllers/configmap.(*ConfigMapReconciler).Reconcile(0xc0007b0880, 0x1c72118, 0xc00090e120, 0xc0006edf10, 0xa, 0xc0006edf00, 0xe, 0xc00090e120, 0xc00090e090, 0xc0004ffdb0, ...)
	/workspace/controllers/configmap/configmap_controller.go:130 +0x822
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile(0xc0003fcb40, 0x1c72118, 0xc00090e090, 0xc0006edf10, 0xa, 0xc0006edf00, 0xe, 0xc00090e000, 0x0, 0x0, ...)
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.10.2/pkg/internal/controller/controller.go:114 +0x247
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler(0xc0003fcb40, 0x1c72070, 0xc0005e4200, 0x187a020, 0xc0004a6a20)
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.10.2/pkg/internal/controller/controller.go:311 +0x305
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem(0xc0003fcb40, 0x1c72070, 0xc0005e4200, 0x73600000500)
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.10.2/pkg/internal/controller/controller.go:266 +0x205
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2(0xc0001386a0, 0xc0003fcb40, 0x1c72070, 0xc0005e4200)
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.10.2/pkg/internal/controller/controller.go:227 +0x6b
created by sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.10.2/pkg/internal/controller/controller.go:223 +0x425
apatil@apatil--MacBookPro16 cli % kubectl describe configmap tkg-system/avi-k8s-config  --kubeconfig /Users/apatil/.kube-tkg/tmp/config_Z06hUbj9
error: there is no need to specify a resource type as a separate argument when passing arguments in resource/name form (e.g. 'kubectl get resource/<resource_name>' instead of 'kubectl get resource resource/<resource_name>'
apatil@apatil--MacBookPro16 cli % kubectl describe configmap avi-k8s-config -n tkg-system --kubeconfig /Users/apatil/.kube-tkg/tmp/config_Z06hUbj9
Name:         avi-k8s-config
Namespace:    tkg-system
Labels:       <none>
Annotations:  <none>

Data
Reply
0 Kudos
2 Replies
apatil1
Contributor
Contributor

DPANIC controllers.ConfigMap odd number of arguments passed as key-value pairs for logging {"ConfigMap": "tkg-system/avi-k8s-config", "ignored key": "d2-117-tkg-workload"}

Any clue what is this error about?

Reply
0 Kudos
sfloarce
VMware Employee
VMware Employee

Hello apatil1

I had same errors but when downgrading to 1.5.1 the AKO controller was up.

Could you try that?

Reply
0 Kudos