vSphere with Tanzu - failed to configure Master node virtual machine

[aokid]

Hello,

I tried to enable workload management by following the instructions in the vSphere with Tanzu Quick Start Guide.

vSphere with Tanzu Quick Start Guide | VMware

After running the wizard to enable workload management, it has been in the configureing status for a long time.

And it shows that Master node virtual machine configuration is failed.

I checked the wcpsvc.log but did not know what caused the configuration to fail.

--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

2020-10-29T07:49:59.278Z debug wcp Process updates to MasterVM VirtualMachine:vm-2037 extra config property.

2020-10-29T07:49:59.278Z debug wcp Update guest customization status Master VM VirtualMachine:vm-2037 is Started.

2020-10-29T07:49:59.278Z debug wcp Process guest tools running state change on MasterVM VirtualMachine:vm-2037 extra config property.

2020-10-29T07:49:59.278Z debug wcp Desired configuration not set for MasterVM VirtualMachine:vm-2037

2020-10-29T07:49:59.29Z info wcp [opID=5f996427-domain-c2015] Guest customization for VM VirtualMachine:vm-2037 is pending. Waiting...

2020-10-29T07:49:59.29Z error wcp [opID=5f996427-domain-c2015] Error configuring API server on cluster domain-c2015 Customization operations of the guest OS for Master node VM with identifier vm-2037 is pending.

2020-10-29T07:49:59.29Z debug wcp Publish change event: &cdc.ChangeLogChangeEvent{Resource:std.DynamicID{Type_:"ClusterComputeResource", Id:"domain-c2015"}, Kind:"UPDATE", Properties:[]string{"messages"}, ParentResources:[]std.DynamicID(nil)}

2020-10-29T07:49:59.29Z debug wcp [opID=5f996427] [ END ] [kubelifecycle.(*Controller).syncClusterState:285] [34.865807586s] cluster=domain-c2015

2020-10-29T07:49:59.29Z debug wcp [opID=5f997344] Processing cluster sync: "domain-c2015"

2020-10-29T07:49:59.29Z debug wcp [opID=5f997344] Attempt to sync cluster domain-c2015 state with desired state.

2020-10-29T07:49:59.29Z debug wcp [opID=5f997344] [BEGIN] [kubelifecycle.(*Controller).syncClusterState:285] cluster=domain-c2015

2020-10-29T07:49:59.294Z debug wcp [opID=5f997344-domain-c2015] apiServerDNSNames = []

2020-10-29T07:49:59.301Z debug wcp [opID=5f997344-domain-c2015] creating service accounts for cluster domain-c2015

2020-10-29T07:49:59.301Z debug wcp [opID=5f997344] Will create VMOperator service account

2020-10-29T07:49:59.301Z debug wcp [opID=5f997344] Will create AppPlatform service account

2020-10-29T07:49:59.301Z debug wcp [opID=5f997344] Creating the following service accounts: [0 1 2]

2020-10-29T07:49:59.301Z debug wcp [opID=5f997344-domain-c2015] refreshing current agencies before provisioning

2020-10-29T07:49:59.301Z debug wcp [opID=5f997344] refreshing all current agencies

2020-10-29T07:49:59.315Z debug wcp [opID=5f997344] Looking for agencies matching 'vmware-vsc-apiserver' (prefixMatch: true) in set of 4

2020-10-29T07:49:59.319Z debug wcp [opID=5f997344] Inspecting agency 'vCLS'

2020-10-29T07:49:59.324Z debug wcp [opID=5f997344] Inspecting agency 'vmware-vsc-apiserver-skkgc6'

2020-10-29T07:49:59.324Z debug wcp [opID=5f997344] findEamAgencyByCluster found match types.ManagedObjectReference{Type:"Agency", Value:"d4d87d05-1f30-4288-a352-ad5899c19c37"}

2020-10-29T07:49:59.331Z debug wcp [opID=5f997344] Inspecting agency 'vmware-vsc-apiserver-cvmvmj'

2020-10-29T07:49:59.331Z debug wcp [opID=5f997344] findEamAgencyByCluster found match types.ManagedObjectReference{Type:"Agency", Value:"a91cb913-8f21-4de7-90d4-0294218f4979"}

2020-10-29T07:49:59.336Z debug wcp [opID=5f997344] Inspecting agency 'vmware-vsc-apiserver-p7n4bp'

2020-10-29T07:49:59.336Z debug wcp [opID=5f997344] findEamAgencyByCluster found match types.ManagedObjectReference{Type:"Agency", Value:"33ad21e4-a243-4c0d-b45c-1b6627e62885"}

2020-10-29T07:49:59.336Z info wcp [opID=5f997344] EAM: found 3 agencies

2020-10-29T07:49:59.336Z debug wcp [opID=5f997344] EAM: refreshing info for agency: Agency:d4d87d05-1f30-4288-a352-ad5899c19c37

2020-10-29T07:49:59.365Z debug wcp [opID=5f997344] EAM: refreshing info for agency: Agency:a91cb913-8f21-4de7-90d4-0294218f4979

2020-10-29T07:49:59.411Z debug wcp [opID=5f997344] EAM: refreshing info for agency: Agency:33ad21e4-a243-4c0d-b45c-1b6627e62885

2020-10-29T07:49:59.451Z info wcp [opID=5f997344] finished refreshing current agencies: map[Agency:33ad21e4-a243-4c0d-b45c-1b6627e62885:0xc000dfaba0 Agency:a91cb913-8f21-4de7-90d4-0294218f4979:0xc000dfbf40 Agency:d4d87d05-1f30-4288-a352-ad5899c19c37:0xc0009a63a0]

2020-10-29T07:49:59.451Z debug wcp [opID=5f997344-reconcile] Start reconciling Master VMs

2020-10-29T07:49:59.451Z debug wcp [opID=5f997344-reconcile] get vm moref from agent info: VirtualMachine:vm-2038

2020-10-29T07:49:59.451Z debug wcp [opID=5f997344-reconcile] got virtual machine object from agent info &{{{} yellow [] enabled Agent:dd45fe47-4acd-4db5-ba7d-0c38b1681ab1} poweredOff false <nil> VirtualMachine:vm-2038   <nil> <nil> [] [] Agency:d4d87d05-1f30-4288-a352-ad5899c19c37 0xc000f708d0}: VirtualMachine:vm-2038

2020-10-29T07:49:59.453Z debug wcp [opID=5f997344-reconcile] got DNS name for vm VirtualMachine:vm-2038: 4202a5062ce2e690ac16a690e52ad9b9

2020-10-29T07:49:59.453Z debug wcp [opID=5f997344] Request for VM auth for VirtualMachine:vm-2038

2020-10-29T07:49:59.485Z error wcp [opID=5f997344] Kubenode guest command failed. Err ServerFaultCode: The attempted operation cannot be performed in the current state (Powered off).

2020-10-29T07:49:59.485Z debug wcp [opID=5f997344] get vm moref from agent info: VirtualMachine:vm-2038

2020-10-29T07:49:59.485Z debug wcp [opID=5f997344] got virtual machine object from agent info &{{{} yellow [] enabled Agent:dd45fe47-4acd-4db5-ba7d-0c38b1681ab1} poweredOff false <nil> VirtualMachine:vm-2038   <nil> <nil> [] [] Agency:d4d87d05-1f30-4288-a352-ad5899c19c37 0xc000f708d0}: VirtualMachine:vm-2038

2020-10-29T07:49:59.489Z error wcp [opID=5f997344] could not find any net intf on master VM vm-2038 connnected to net dvportgroup-2024

2020-10-29T07:49:59.489Z debug wcp [opID=5f997344] unable to get IPs [] from master vm VirtualMachine:vm-2038: could not find any net intf on master VM vm-2038 connnected to net dvportgroup-2024

2020-10-29T07:49:59.489Z debug wcp [opID=5f997344-reconcile] get vm moref from agent info: VirtualMachine:vm-2037

2020-10-29T07:49:59.489Z debug wcp [opID=5f997344-reconcile] got virtual machine object from agent info &{{{} green [] enabled Agent:77e4426f-66b2-4018-92d8-13f3f8562ae4} poweredOn false <nil> VirtualMachine:vm-2037   <nil> <nil> [] [] Agency:a91cb913-8f21-4de7-90d4-0294218f4979 <nil>}: VirtualMachine:vm-2037

2020-10-29T07:49:59.491Z debug wcp [opID=5f997344-reconcile] got DNS name for vm VirtualMachine:vm-2037: 420211f0fbe11977a7cefa6bfceda5f1

2020-10-29T07:49:59.491Z debug wcp [opID=5f997344] Request for VM auth for VirtualMachine:vm-2037

2020-10-29T07:50:02.049Z error wcp [opID=5f997344] Kubenode guest command failed. Err ServerFaultCode: Failed to authenticate with the guest operating system using the supplied credentials.

2020-10-29T07:50:02.049Z debug wcp [opID=5f997344] get vm moref from agent info: VirtualMachine:vm-2037

2020-10-29T07:50:02.049Z debug wcp [opID=5f997344] got virtual machine object from agent info &{{{} green [] enabled Agent:77e4426f-66b2-4018-92d8-13f3f8562ae4} poweredOn false <nil> VirtualMachine:vm-2037   <nil> <nil> [] [] Agency:a91cb913-8f21-4de7-90d4-0294218f4979 <nil>}: VirtualMachine:vm-2037

2020-10-29T07:50:02.055Z debug wcp [opID=5f997344] Mgmt net device mac address for master VM vm-2037: 00:50:56:82:2c:34

2020-10-29T07:50:02.055Z debug wcp [opID=5f997344] unable to get IPs [] from master vm VirtualMachine:vm-2037: <nil>

2020-10-29T07:50:02.055Z debug wcp [opID=5f997344-reconcile] failed to get hostname for master vm <nil>

2020-10-29T07:50:02.055Z debug wcp [opID=5f997344-reconcile] There is not any etcd member yet, no need to reconcile.

2020-10-29T07:50:02.055Z debug wcp [opID=5f997344] refreshing all current agencies

2020-10-29T07:50:02.067Z debug wcp [opID=5f997344] Looking for agencies matching 'vmware-vsc-apiserver' (prefixMatch: true) in set of 4

2020-10-29T07:50:02.071Z debug wcp [opID=5f997344] Inspecting agency 'vCLS'

2020-10-29T07:50:02.076Z debug wcp [opID=5f997344] Inspecting agency 'vmware-vsc-apiserver-skkgc6'

2020-10-29T07:50:02.076Z debug wcp [opID=5f997344] findEamAgencyByCluster found match types.ManagedObjectReference{Type:"Agency", Value:"d4d87d05-1f30-4288-a352-ad5899c19c37"}

2020-10-29T07:50:02.08Z debug wcp [opID=5f997344] Inspecting agency 'vmware-vsc-apiserver-cvmvmj'

2020-10-29T07:50:02.08Z debug wcp [opID=5f997344] findEamAgencyByCluster found match types.ManagedObjectReference{Type:"Agency", Value:"a91cb913-8f21-4de7-90d4-0294218f4979"}

2020-10-29T07:50:02.087Z debug wcp [opID=5f997344] Inspecting agency 'vmware-vsc-apiserver-p7n4bp'

2020-10-29T07:50:02.087Z debug wcp [opID=5f997344] findEamAgencyByCluster found match types.ManagedObjectReference{Type:"Agency", Value:"33ad21e4-a243-4c0d-b45c-1b6627e62885"}

2020-10-29T07:50:02.087Z info wcp [opID=5f997344] EAM: found 3 agencies

2020-10-29T07:50:02.087Z debug wcp [opID=5f997344] EAM: refreshing info for agency: Agency:d4d87d05-1f30-4288-a352-ad5899c19c37

2020-10-29T07:50:02.108Z debug wcp [opID=5f997344] EAM: refreshing info for agency: Agency:a91cb913-8f21-4de7-90d4-0294218f4979

2020-10-29T07:50:02.122Z debug wcp [opID=5f997344] EAM: refreshing info for agency: Agency:33ad21e4-a243-4c0d-b45c-1b6627e62885

2020-10-29T07:50:02.141Z info wcp [opID=5f997344] finished refreshing current agencies: map[Agency:33ad21e4-a243-4c0d-b45c-1b6627e62885:0xc0015057c0 Agency:a91cb913-8f21-4de7-90d4-0294218f4979:0xc0012e0f60 Agency:d4d87d05-1f30-4288-a352-ad5899c19c37:0xc0003f8ae0]

2020-10-29T07:50:02.141Z debug wcp [opID=5f997344] AllocateIP request for VM: ClusterComputeResource:domain-c2015 and nicID: 0

2020-10-29T07:50:02.141Z debug wcp [opID=5f997344] An IP 192.168.200.64 has already been allocated for ClusterComputeResource:domain-c2015 0.

2020-10-29T07:50:02.141Z info wcp [opID=5f997344-domain-c2015] Management network floating IP is 192.168.200.64

2020-10-29T07:50:02.141Z debug wcp [opID=5f997344] get vm moref from agent info: VirtualMachine:vm-2038

2020-10-29T07:50:02.141Z debug wcp [opID=5f997344] got virtual machine object from agent info &{{{} yellow [] enabled Agent:dd45fe47-4acd-4db5-ba7d-0c38b1681ab1} poweredOff false <nil> VirtualMachine:vm-2038   <nil> <nil> [] [] Agency:d4d87d05-1f30-4288-a352-ad5899c19c37 0xc000f708d0}: VirtualMachine:vm-2038

2020-10-29T07:50:02.143Z debug wcp [opID=5f997344] got DNS name for vm VirtualMachine:vm-2038: 4202a5062ce2e690ac16a690e52ad9b9

2020-10-29T07:50:02.143Z debug wcp [opID=5f997344] get vm moref from agent info: VirtualMachine:vm-2037

2020-10-29T07:50:02.143Z debug wcp [opID=5f997344] got virtual machine object from agent info &{{{} green [] enabled Agent:77e4426f-66b2-4018-92d8-13f3f8562ae4} poweredOn false <nil> VirtualMachine:vm-2037   <nil> <nil> [] [] Agency:a91cb913-8f21-4de7-90d4-0294218f4979 <nil>}: VirtualMachine:vm-2037

2020-10-29T07:50:02.145Z debug wcp [opID=5f997344] got DNS name for vm VirtualMachine:vm-2037: 420211f0fbe11977a7cefa6bfceda5f1

2020-10-29T07:50:02.145Z info wcp [opID=5f997344] Add Master nodes [VirtualMachine:vm-2038] to cluster domain-c2015

2020-10-29T07:50:02.148Z debug wcp Process updates to MasterVM VirtualMachine:vm-2038 extra config property.

2020-10-29T07:50:02.148Z debug wcp Process guest tools running state change on MasterVM VirtualMachine:vm-2038 extra config property.

2020-10-29T07:50:02.15Z info wcp [opID=5f997344-domain-c2015] Configuring 1 master agents on cluster domain-c2015

2020-10-29T07:50:02.15Z debug wcp [opID=5f997344-domain-c2015] Get VC Tag for WCP VM-VM Anti-Affinity

2020-10-29T07:50:02.15Z debug wcp [opID=5f997344-domain-c2015] Rest client for vmodl2 API calls exists, checking session validity

2020-10-29T07:50:02.152Z debug wcp [opID=5f997344-domain-c2015] Rest client for vmodl2 API calls is still valid.

2020-10-29T07:50:02.152Z debug wcp [opID=5f997344-domain-c2015] Get VC Category for WCP VM-VM Anti-Affinity

2020-10-29T07:50:02.152Z debug wcp [opID=5f997344-domain-c2015] Rest client for vmodl2 API calls exists, checking session validity

2020-10-29T07:50:02.216Z debug wcp [opID=5f997344-domain-c2015] Rest client for vmodl2 API calls is still valid.

2020-10-29T07:50:02.231Z debug wcp [opID=5f997344-domain-c2015] Found existing VC Category wp_vmvmaa_category

2020-10-29T07:50:02.239Z debug wcp [opID=5f997344-domain-c2015] Found existing VC Tag wp_vmvmaa_tag

2020-10-29T07:50:02.239Z debug wcp [opID=5f997344-domain-c2015] Rest client for vmodl2 API calls exists, checking session validity

2020-10-29T07:50:02.241Z debug wcp [opID=5f997344-domain-c2015] Rest client for vmodl2 API calls is still valid.

2020-10-29T07:50:02.255Z debug wcp [opID=5f997344-domain-c2015] Add VM VirtualMachine:vm-2037 to ClusterModule 529b8880-655b-2efe-9585-25080acf6e81 for Supervisor Cluster Control Plane VMs

2020-10-29T07:50:02.255Z debug wcp [opID=5f997344-domain-c2015] Rest client for vmodl2 API calls exists, checking session validity

2020-10-29T07:50:02.258Z debug wcp [opID=5f997344-domain-c2015] Rest client for vmodl2 API calls is still valid.

2020-10-29T07:50:02.679Z info wcp [opID=5f997344-domain-c2015] MULTIMASTER configuring agency Agency:a91cb913-8f21-4de7-90d4-0294218f4979

2020-10-29T07:50:02.679Z info wcp [opID=5f997344-domain-c2015] Configuring API server agent Agent:77e4426f-66b2-4018-92d8-13f3f8562ae4 VM on cluster domain-c2015.

2020-10-29T07:50:02.689Z debug wcp [opID=5f997344-domain-c2015] Got cached machine ID: 0f34734e-a3e6-44b6-8416-09472d26826e

2020-10-29T07:50:02.85Z debug wcp [opID=5f997344-domain-c2015] AllocateIP request for VM: ClusterComputeResource:domain-c2015 and nicID: 0

2020-10-29T07:50:02.85Z debug wcp [opID=5f997344-domain-c2015] An IP 192.168.200.64 has already been allocated for ClusterComputeResource:domain-c2015 0.

2020-10-29T07:50:02.857Z debug wcp [opID=5f997344-domain-c2015] Got cached machine ID: 0f34734e-a3e6-44b6-8416-09472d26826e

2020-10-29T07:50:02.933Z debug wcp [opID=5f997344-domain-c2015] Successfully set permissions [{{} <nil> wcp-cluster-user-domain-c2015-0f34734e-a3e6-44b6-8416-09472d26826e false 1005 true}] on entity ClusterComputeResource:domain-c2015

2020-10-29T07:50:03.026Z debug wcp [opID=5f997344-domain-c2015] Successfully set permissions [{{} <nil> wcp-cluster-user-domain-c2015-0f34734e-a3e6-44b6-8416-09472d26826e false 1004 true}] on entity Folder:group-d1

2020-10-29T07:50:03.106Z debug wcp [opID=5f997344-domain-c2015] Successfully set permissions [{{} <nil> wcp-cluster-user-domain-c2015-0f34734e-a3e6-44b6-8416-09472d26826e false 1005 true}] on entity ResourcePool:resgroup-2032

2020-10-29T07:50:03.192Z debug wcp [opID=5f997344-domain-c2015] Successfully set permissions [{{} <nil> wcp-cluster-user-domain-c2015-0f34734e-a3e6-44b6-8416-09472d26826e false 1005 true}] on entity Folder:group-v2033

2020-10-29T07:50:03.192Z debug wcp [opID=5f997344-domain-c2015] Got cached machine ID: 0f34734e-a3e6-44b6-8416-09472d26826e

2020-10-29T07:50:03.301Z debug wcp [opID=5f997344-domain-c2015] Successfully set permissions [{{} <nil> wcp-vmop-user-domain-c2015-0f34734e-a3e6-44b6-8416-09472d26826e@vsphere.lab false 1004 true}] on entity Folder:group-d1

2020-10-29T07:50:03.413Z debug wcp [opID=5f997344-domain-c2015] Successfully set permissions [{{} <nil> wcp-vmop-user-domain-c2015-0f34734e-a3e6-44b6-8416-09472d26826e@vsphere.lab false 1021 true}] on entity Datacenter:datacenter-3

2020-10-29T07:50:03.541Z debug wcp [opID=5f997344-domain-c2015] Successfully set permissions [{{} <nil> wcp-vmop-user-domain-c2015-0f34734e-a3e6-44b6-8416-09472d26826e@vsphere.lab false 1021 true}] on entity ResourcePool:resgroup-2032

2020-10-29T07:50:03.643Z debug wcp [opID=5f997344-domain-c2015] Successfully set permissions [{{} <nil> wcp-vmop-user-domain-c2015-0f34734e-a3e6-44b6-8416-09472d26826e@vsphere.lab false 1021 true}] on entity Folder:group-v2033

2020-10-29T07:50:03.643Z debug wcp [opID=5f997344-domain-c2015] Got cached machine ID: 0f34734e-a3e6-44b6-8416-09472d26826e

2020-10-29T07:50:03.643Z debug wcp [opID=5f997344-domain-c2015] Permission client already exists, reuse it.

2020-10-29T07:50:03.715Z debug wcp [opID=5f997344-domain-c2015] Set global permission ID=username=vsphere.lab\wcp-vmop-user-domain-c2015-0f34734e-a3e6-44b6-8416-09472d26826e;doc=urn:acl:global:permissions on principal wcp-vmop-user-domain-c2015-0f34734e-a3e6-44b6-8416-09472d26826e.

2020-10-29T07:50:03.715Z debug wcp [opID=5f997344-domain-c2015] Got cached machine ID: 0f34734e-a3e6-44b6-8416-09472d26826e

2020-10-29T07:50:03.805Z debug wcp [opID=5f997344-domain-c2015] Successfully set permissions [{{} <nil> wcp-appplatform-user-domain-c2015-0f34734e-a3e6-44b6-8416-09472d26826e@vsphere.lab false 1007 true}] on entity ClusterComputeResource:domain-c2015

2020-10-29T07:50:03.883Z debug wcp [opID=5f997344-domain-c2015] Successfully set permissions [{{} <nil> wcp-appplatform-user-domain-c2015-0f34734e-a3e6-44b6-8416-09472d26826e@vsphere.lab false 1008 true}] on entity Folder:group-d1

2020-10-29T07:50:03.883Z debug wcp [opID=5f997344-domain-c2015] Got cached machine ID: 0f34734e-a3e6-44b6-8416-09472d26826e

2020-10-29T07:50:03.883Z debug wcp [opID=5f997344-domain-c2015] Permission client already exists, reuse it.

2020-10-29T07:50:03.95Z debug wcp [opID=5f997344-domain-c2015] Set global permission ID=username=vsphere.lab\wcp-appplatform-user-domain-c2015-0f34734e-a3e6-44b6-8416-09472d26826e;doc=urn:acl:global:permissions on principal wcp-appplatform-user-domain-c2015-0f34734e-a3e6-44b6-8416-09472d26826e.

2020-10-29T07:50:03.951Z debug wcp [opID=5f997344-domain-c2015] Got cached machine ID: 0f34734e-a3e6-44b6-8416-09472d26826e

2020-10-29T07:50:03.951Z debug wcp [opID=5f997344-domain-c2015] Got cached machine ID: 0f34734e-a3e6-44b6-8416-09472d26826e

2020-10-29T07:50:03.951Z debug wcp [opID=5f997344-domain-c2015] Finding cluster netif for MasterNode: VirtualMachine:vm-2037 in Cluster domain-c2015.

2020-10-29T07:50:03.951Z warning wcp [opID=5f997344-domain-c2015] Did not find any cluster netif for MasterNode: VirtualMachine:vm-2037 in Cluster domain-c2015.

2020-10-29T07:50:03.951Z debug wcp [opID=5f997344-domain-c2015] Got cached machine ID: 0f34734e-a3e6-44b6-8416-09472d26826e

2020-10-29T07:50:03.958Z debug wcp [opID=5f997344-domain-c2015] Mgmt net device mac address for master VM vm-2037: 00:50:56:82:2c:34

2020-10-29T07:50:03.958Z debug wcp [opID=5f997344-domain-c2015] Master 420211f0fbe11977a7cefa6bfceda5f1 does not yet publish a management IP. Deadline is 2020-10-29 07:54:38.839895783 +0000 UTC: <nil>

2020-10-29T07:50:03.96Z info wcp [opID=5f997344-domain-c2015] Deadline for VM VirtualMachine:vm-2037 to publish IP is 2020-10-29 07:54:38.839895783 +0000 UTC.

2020-10-29T07:50:03.96Z info wcp [opID=5f997344-domain-c2015] Wait for guest tools to be ready for VM VirtualMachine:vm-2037

2020-10-29T07:50:03.965Z debug wcp [opID=5f997344-domain-c2015] Got guest.toolsRunningStatus on vm-2037

2020-10-29T07:50:03.966Z info wcp [opID=5f997344-domain-c2015] Guest customization for VM VirtualMachine:vm-2037 is pending. Waiting...

2020-10-29T07:50:03.966Z error wcp [opID=5f997344-domain-c2015] Error configuring API server on cluster domain-c2015 Customization operations of the guest OS for Master node VM with identifier vm-2037 is pending.

2020-10-29T07:50:03.967Z debug wcp Publish change event: &cdc.ChangeLogChangeEvent{Resource:std.DynamicID{Type_:"ClusterComputeResource", Id:"domain-c2015"}, Kind:"UPDATE", Properties:[]string{"messages"}, ParentResources:[]std.DynamicID(nil)}

2020-10-29T07:50:03.967Z debug wcp [opID=5f997344] [ END ] [kubelifecycle.(*Controller).syncClusterState:285] [4.676343254s] cluster=domain-c2015

--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

Does anyone know how can I fix it ?

Thank you,

[kastrosmartis]

Hi

I have the same problem. Wired thing is that first time i managed to enable Workload and all was ok, but next day there was a lot off errors etc. I removed everything and now I am stuck with this error...

So 3 VMs are deployed, first one is powered on and then stuck there...

Any ideas?

[JoWiRgbg]

Same problem here. Are there any troubleshooting solutions in the meantime?

[Lennon-Geng]

Hi, I found a successful way：

Provide "DNS Search Domains" when u enabling WCP if your vCenter is brought-up with FQDN.

[kastrosmartis]

Supervisor cluster got deployed (not sure if "DNS Search Domains" fixed it - made a lot changes).

Now I have similar problem deploying Guest cluster. Control Plane VM gets deployed, powered on, HAproxy .cfg populated with frontend/backend IPs, IP set up (unpingable) but then stuck before Wokers deployment.

kubectl get events -w
LAST SEEN TYPE REASON OBJECT MESSAGE
0s Warning ReconcileFailure wcpcluster/simple unexpected error while reconciling control plane endpoint for simple: failed to reconcile loadbalanced endpoint for WCPCluster tanzu-ns-01/simple: failed to get control plane endpoint for Cluster tanzu-ns-01/simple: VirtualMachineService LB does not yet have VIP assigned: VirtualMachineService LoadBalancer does not have any Ingresses
0s Normal CreateVMServiceSuccess virtualmachineservice/simple-control-plane-service CreateVMService success
0s Warning ReconcileFailure wcpcluster/simple unexpected error while reconciling control plane endpoint for simple: failed to reconcile loadbalanced endpoint for WCPCluster tanzu-ns-01/simple: failed to get control plane endpoint for Cluster tanzu-ns-01/simple: VirtualMachineService LB does not yet have VIP assigned: VirtualMachineService LoadBalancer does not have any Ingresses
0s Warning ReconcileFailure wcpcluster/simple unexpected error while reconciling control plane endpoint for simple: failed to reconcile loadbalanced endpoint for WCPCluster tanzu-ns-01/simple: failed to get control plane endpoint for Cluster tanzu-ns-01/simple: VirtualMachineService LB does not yet have VIP assigned: VirtualMachineService LoadBalancer does not have any Ingresses
0s Warning ReconcileFailure wcpcluster/simple unexpected error while reconciling control plane endpoint for simple: failed to reconcile loadbalanced endpoint for WCPCluster tanzu-ns-01/simple: failed to get control plane endpoint for Cluster tanzu-ns-01/simple: VirtualMachineService LB does not yet have VIP assigned: VirtualMachineService LoadBalancer does not have any Ingresses
0s Warning ReconcileFailure wcpcluster/simple unexpected error while reconciling control plane endpoint for simple: failed to reconcile loadbalanced endpoint for WCPCluster tanzu-ns-01/simple: failed to get control plane endpoint for Cluster tanzu-ns-01/simple: VirtualMachineService LB does not yet have VIP assigned: VirtualMachineService LoadBalancer does not have any Ingresses
0s Warning ReconcileFailure wcpcluster/simple unexpected error while reconciling control plane endpoint for simple: failed to reconcile loadbalanced endpoint for WCPCluster tanzu-ns-01/simple: failed to get control plane endpoint for Cluster tanzu-ns-01/simple: VirtualMachineService LB does not yet have VIP assigned: VirtualMachineService LoadBalancer does not have any Ingresses
0s Normal Reconcile gateway/simple-control-plane-service Success
0s Normal Reconcile gateway/simple-control-plane-service Success
0s Normal SuccessfulCreate machinedeployment/simple-workers-kszj2 Created MachineSet "simple-workers-kszj2-d4c6b6f49"
0s Normal SuccessfulCreate machineset/simple-workers-kszj2-d4c6b6f49 Created machine "simple-workers-kszj2-d4c6b6f49-m6sg9"
0s Normal SuccessfulCreate machineset/simple-workers-kszj2-d4c6b6f49 Created machine "simple-workers-kszj2-d4c6b6f49-795l5"
0s Normal SuccessfulCreate machineset/simple-workers-kszj2-d4c6b6f49 Created machine "simple-workers-kszj2-d4c6b6f49-5z5zg"
0s Warning ReconcileFailure wcpmachine/simple-control-plane-hfrm7-7twzk vm is not yet created: vmware-system-capw-controller-manager/WCPMachine/infrastructure.cluster.vmware.com/v1alpha3/tanzu-ns-01/simple/simple-control-plane-hfrm7-7twzk
0s Warning ReconcileFailure wcpmachine/simple-control-plane-hfrm7-7twzk vm is not yet created: vmware-system-capw-controller-manager/WCPMachine/infrastructure.cluster.vmware.com/v1alpha3/tanzu-ns-01/simple/simple-control-plane-hfrm7-7twzk
0s Warning ReconcileFailure wcpmachine/simple-control-plane-hfrm7-7twzk vm is not yet created: vmware-system-capw-controller-manager/WCPMachine/infrastructure.cluster.vmware.com/v1alpha3/tanzu-ns-01/simple/simple-control-plane-hfrm7-7twzk
0s Warning ReconcileFailure wcpmachine/simple-control-plane-hfrm7-7twzk vm is not yet created: vmware-system-capw-controller-manager/WCPMachine/infrastructure.cluster.vmware.com/v1alpha3/tanzu-ns-01/simple/simple-control-plane-hfrm7-7twzk
0s Warning ReconcileFailure wcpmachine/simple-control-plane-hfrm7-7twzk vm is not yet created: vmware-system-capw-controller-manager/WCPMachine/infrastructure.cluster.vmware.com/v1alpha3/tanzu-ns-01/simple/simple-control-plane-hfrm7-7twzk
0s Warning ReconcileFailure wcpmachine/simple-control-plane-hfrm7-7twzk vm is not yet created: vmware-system-capw-controller-manager/WCPMachine/infrastructure.cluster.vmware.com/v1alpha3/tanzu-ns-01/simple/simple-control-plane-hfrm7-7twzk
0s Warning ReconcileFailure wcpmachine/simple-control-plane-hfrm7-7twzk vm is not yet created: vmware-system-capw-controller-manager/WCPMachine/infrastructure.cluster.vmware.com/v1alpha3/tanzu-ns-01/simple/simple-control-plane-hfrm7-7twzk
1s Warning ReconcileFailure wcpmachine/simple-control-plane-hfrm7-7twzk vm is not yet created: vmware-system-capw-controller-manager/WCPMachine/infrastructure.cluster.vmware.com/v1alpha3/tanzu-ns-01/simple/simple-control-plane-hfrm7-7twzk
0s Warning ReconcileFailure wcpmachine/simple-control-plane-hfrm7-7twzk vm is not yet created: vmware-system-capw-controller-manager/WCPMachine/infrastructure.cluster.vmware.com/v1alpha3/tanzu-ns-01/simple/simple-control-plane-hfrm7-7twzk
0s Warning ReconcileFailure wcpmachine/simple-control-plane-hfrm7-7twzk vm is not yet created: vmware-system-capw-controller-manager/WCPMachine/infrastructure.cluster.vmware.com/v1alpha3/tanzu-ns-01/simple/simple-control-plane-hfrm7-7twzk
0s Warning ReconcileFailure wcpmachine/simple-control-plane-hfrm7-7twzk vm is not yet created: vmware-system-capw-controller-manager/WCPMachine/infrastructure.cluster.vmware.com/v1alpha3/tanzu-ns-01/simple/simple-control-plane-hfrm7-7twzk
0s Warning ReconcileFailure wcpmachine/simple-control-plane-hfrm7-7twzk vm is not yet created: vmware-system-capw-controller-manager/WCPMachine/infrastructure.cluster.vmware.com/v1alpha3/tanzu-ns-01/simple/simple-control-plane-hfrm7-7twzk
0s Warning ReconcileFailure wcpmachine/simple-control-plane-hfrm7-7twzk vm is not yet created: vmware-system-capw-controller-manager/WCPMachine/infrastructure.cluster.vmware.com/v1alpha3/tanzu-ns-01/simple/simple-control-plane-hfrm7-7twzk
0s Warning ReconcileFailure wcpmachine/simple-control-plane-hfrm7-7twzk vm is not yet created: vmware-system-capw-controller-manager/WCPMachine/infrastructure.cluster.vmware.com/v1alpha3/tanzu-ns-01/simple/simple-control-plane-hfrm7-7twzk
0s Warning ReconcileFailure wcpmachine/simple-control-plane-hfrm7-7twzk vm is not yet created: vmware-system-capw-controller-manager/WCPMachine/infrastructure.cluster.vmware.com/v1alpha3/tanzu-ns-01/simple/simple-control-plane-hfrm7-7twzk
0s Warning ReconcileFailure wcpmachine/simple-control-plane-hfrm7-7twzk vm does not have an IP address: vmware-system-capw-controller-manager/WCPMachine/infrastructure.cluster.vmware.com/v1alpha3/tanzu-ns-01/simple/simple-control-plane-hfrm7-7twzk
0s Warning ReconcileFailure wcpmachine/simple-control-plane-hfrm7-7twzk vm does not have an IP address: vmware-system-capw-controller-manager/WCPMachine/infrastructure.cluster.vmware.com/v1alpha3/tanzu-ns-01/simple/simple-control-plane-hfrm7-7twzk
0s Warning ReconcileFailure wcpmachine/simple-control-plane-hfrm7-7twzk vm does not have an IP address: vmware-system-capw-controller-manager/WCPMachine/infrastructure.cluster.vmware.com/v1alpha3/tanzu-ns-01/simple/simple-control-plane-hfrm7-7twzk
0s Warning ReconcileFailure wcpmachine/simple-control-plane-hfrm7-7twzk vm does not have an IP address: vmware-system-capw-controller-manager/WCPMachine/infrastructure.cluster.vmware.com/v1alpha3/tanzu-ns-01/simple/simple-control-plane-hfrm7-7twzk
0s Normal Reconcile gateway/simple-control-plane-service Success
0s Normal Reconcile gateway/simple-control-plane-service Success
0s Normal Reconcile gateway/simple-control-plane-service Success
0s Normal Reconcile gateway/simple-control-plane-service Success
1s Normal Reconcile gateway/simple-control-plane-service Success

Cluster API Status:
API Endpoints:
Host: 172.16.97.194
Port: 6443
Phase: Provisioned
Node Status:
simple-control-plane-rb7r6: pending
simple-workers-kszj2-d4c6b6f49-5z5zg: pending
simple-workers-kszj2-d4c6b6f49-795l5: pending
simple-workers-kszj2-d4c6b6f49-m6sg9: pending
Phase: creating
Vm Status:
simple-control-plane-rb7r6: ready
simple-workers-kszj2-d4c6b6f49-5z5zg: pending
simple-workers-kszj2-d4c6b6f49-795l5: pending
simple-workers-kszj2-d4c6b6f49-m6sg9: pending

[kvtomino]

Did you find a solution? I have the exact same problem

[kastrosmartis]

No, not yet. Tried a few times to deploy, 2-nic Haproxy, 3-nic HAProxy, CIDR address overlapping check,....and still can't get past this last step ;-(

[Lennon-Geng]

Is your frontend network routable to the workload network? Please check it.

[kastrosmartis]

Hi

Yes, it is. Have also tested with temp VM in all portgroups, all routable, pingable,....

What about MTU size? I see on official documentation requirements 1600 size, on some blogs 1500, then 9000 for nested virtualization,...

Will test a few different settings.

[kvtomino]

Yes I also have checked all those things and can confirm it's not a networking issue.

[kastrosmartis]

I have tried on baremetal ESXi servers also (not nested) with same results. Only vCenter is the same in both cases.

First thing after apply for new Guest cluster is error about Ingress...

kubectl get events -w
LAST SEEN TYPE REASON OBJECT MESSAGE
0s Warning ReconcileFailure wcpcluster/simple unexpected error while reconciling control plane endpoint for simple: failed to reconcile loadbalanced endpoint for WCPCluster tanzu-ns-01/simple: failed to get control plane endpoint for Cluster tanzu-ns-01/simple: VirtualMachineService LB does not yet have VIP assigned: VirtualMachineService LoadBalancer does not have any Ingresses
0s Normal CreateVMServiceSuccess virtualmachineservice/simple-control-plane-service CreateVMService success
0s Warning ReconcileFailure wcpcluster/simple unexpected error while reconciling control plane endpoint for simple: failed to reconcile loadbalanced endpoint for WCPCluster tanzu-ns-01/simple: failed to get control plane endpoint for Cluster tanzu-ns-01/simple: VirtualMachineService LB does not yet have VIP assigned: VirtualMachineService LoadBalancer does not have any Ingresses
0s Normal Reconcile gateway/simple-control-plane-service Success
0s Normal SuccessfulCreate machinedeployment/simple-workers-kszj2 Created MachineSet "simple-workers-kszj2-d4c6b6f49"
0s Normal SuccessfulCreate machineset/simple-workers-kszj2-d4c6b6f49 Created machine "simple-workers-kszj2-d4c6b6f49-m6sg9"
0s Normal SuccessfulCreate machineset/simple-workers-kszj2-d4c6b6f49 Created machine "simple-workers-kszj2-d4c6b6f49-795l5"
0s Normal SuccessfulCreate machineset/simple-workers-kszj2-d4c6b6f49 Created machine "simple-workers-kszj2-d4c6b6f49-5z5zg"
0s Warning ReconcileFailure wcpmachine/simple-control-plane-hfrm7-7twzk vm is not yet created: vmware-system-capw-controller-manager/WCPMachine/infrastructure.cluster.vmware.com/v1alpha3/tanzu-ns-01/simple/simple-control-plane-hfrm7-7twzk
0s Warning ReconcileFailure wcpmachine/simple-control-plane-hfrm7-7twzk vm does not have an IP address: vmware-system-capw-controller-manager/WCPMachine/infrastructure.cluster.vmware.com/v1alpha3/tanzu-ns-01/simple/simple-control-plane-hfrm7-7twzk
0s Normal Reconcile gateway/simple-control-plane-service Success
0s Normal Reconcile gateway/simple-control-plane-service Success
0s Normal Reconcile gateway/simple-control-plane-service Success
0s Normal Reconcile gateway/simple-control-plane-service Success
1s Normal Reconcile gateway/simple-control-plane-service Success

Then Control Plane VM gets deployed, IP set up (not pingable), Workers never deployed (pending). HAproxy populated with front/backend settings.

haproxy.cfg
frontend domain-c205698:B5CBA8B0-DCC4-4C81-8407-92E70AE3B8B0-tanzu-ns-01-simple-control-plane-service
mode tcp
bind 172.16.97.193:6443 name domain-c205698:B5CBA8B0-DCC4-4C81-8407-92E70AE3B8B0-tanzu-ns-01-simple-control-plane-service-172.16.97.193:apiserver
log-tag domain-c205698:B5CBA8B0-DCC4-4C81-8407-92E70AE3B8B0-tanzu-ns-01-simple-control-plane-service
option tcplog
use_backend domain-c205698:B5CBA8B0-DCC4-4C81-8407-92E70AE3B8B0-tanzu-ns-01-simple-control-plane-service-apiserver if { dst_port 6443 }

backend domain-c205698:B5CBA8B0-DCC4-4C81-8407-92E70AE3B8B0-tanzu-ns-01-simple-control-plane-service-apiserver
mode tcp
balance roundrobin
option tcp-check
log-tag domain-c205698:B5CBA8B0-DCC4-4C81-8407-92E70AE3B8B0-tanzu-ns-01-simple-control-plane-service-apiserver
server domain-c205698:B5CBA8B0-DCC4-4C81-8407-92E70AE3B8B0-tanzu-ns-01-simple-control-plane-service-172.16.96.23:6443 172.16.96.23:6443 check-ssl weight 100 verify none

[Lennon-Geng]

It works in my lab.

[kastrosmartis]

Are you using virtual router for VLANs (all nested ) or do you have VLANs on physical layer? In documentation there is requirement for MTU 1600 (which I don't have).

Now I have TLS error, something with certificates maybe?

24m Warning ReconcileFailure wcpmachine/tkc-01-control-plane-gr8vc-z4hch vm does not have an IP address: vmware-system-capw-controller-manager/WCPMachine/infrastructure.cluster.vmware.com/v1alpha3/tanzu-ns-01/tkc-01/tkc-01-control-plane-gr8vc-z4hch
26m Normal CreateVMServiceSuccess virtualmachineservice/tkc-01-control-plane-service CreateVMService success
19s Normal Reconcile gateway/tkc-01-control-plane-service Success
21m Warning ControlPlaneUnhealthy kubeadmcontrolplane/tkc-01-control-plane Waiting for control plane to pass control plane health check to continue reconciliation: tanzu-ns-01/tkc-01: Get https://172.16.97.65:6443/api?timeout=30s: dial tcp 172.16.97.65:6443: connect: connection refused
52s Warning ControlPlaneUnhealthy kubeadmcontrolplane/tkc-01-control-plane Waiting for control plane to pass control plane health check to continue reconciliation: tanzu-ns-01/tkc-01: Get https://172.16.97.65:6443/api?timeout=30s: net/http: TLS handshake timeout
26m Normal SuccessfulCreate machineset/tkc-01-workers-tt6xw-669f5696ff Created machine "tkc-01-workers-tt6xw-669f5696ff-mvjqq"
26m Normal SuccessfulCreate machineset/tkc-01-workers-tt6xw-669f5696ff Created machine "tkc-01-workers-tt6xw-669f5696ff-5wlws"
26m Normal SuccessfulCreate machinedeployment/tkc-01-workers-tt6xw Created MachineSet "tkc-01-workers-tt6xw-669f5696ff"
26m Warning ReconcileFailure wcpcluster/tkc-01 unexpected error while reconciling control plane endpoint for tkc-01: failed to reconcile loadbalanced endpoint for WCPCluster tanzu-ns-01/tkc-01: failed to get control plane endpoint for Cluster tanzu-ns-01/tkc-01: VirtualMachineService LB does not yet have VIP assigned: VirtualMachineService LoadBalancer does not have any Ingresses
0s Warning ControlPlaneUnhealthy kubeadmcontrolplane/tkc-01-control-plane Waiting for control plane to pass control plane health check to continue reconciliation: tanzu-ns-01/tkc-01: Get https://172.16.97.65:6443/api?timeout=30s: net/http: TLS handshake timeout
0s Normal Reconcile gateway/tkc-01-control-plane-service Success
0s Normal Reconcile gateway/tkc-01-control-plane-service Success
0s Normal Reconcile gateway/tkc-01-control-plane-service Success
0s Warning ControlPlaneUnhealthy kubeadmcontrolplane/tkc-01-control-plane Waiting for control plane to pass control plane health check to continue reconciliation: tanzu-ns-01/tkc-01: Get https://172.16.97.65:6443/api?timeout=30s: net/http: TLS handshake timeout

curl --insecure -X GET https://172.16.97.65:6443
{
"kind": "Status",
"apiVersion": "v1",
"metadata": {

},
"status": "Failure",
"message": "forbidden: User \"system:anonymous\" cannot get path \"/\"",
"reason": "Forbidden",
"details": {

},
"code": 403

[Lennon-Geng]

I'm using physical switch for all VLAN and routings with default MTU 1500. 1600 is required when u using NSX-T.  

For nested environment, please refer to:

https://www.youtube.com/watch?v=uSGujnlYpVc

[kastrosmartis]

Ok, thanx for confirming that MTU size is not the problem here.

P.S. then there is error in documentation which states that MTU must be 1600...

https://docs.vmware.com/en/VMware-vSphere/7.0/vmware-vsphere-with-tanzu/GUID-C3048E95-6E9D-4AC3-BE96...

Thanx

[dnord]

Did anyone manage to find a solution for the error "Kubenode guest command failed. Err ServerFaultCode: Failed to authenticate with the guest operating system using the supplied credentials."  ??

My installation just loops while trying to configure the Supervisor VMs but failes  due to the error "Kubenode guest command failed. Err ServerFaultCode: Failed to authenticate with the guest operating system using the supplied credentials." and will loop infinitely.

In the wcp logs its just looks like the deployment scripts just

1. download vm template for supervisor vm
2. deploy supervisor vm
3. install agent on supervisor vms
4. try to modify something on the supervisor vms using the vmtools, but gets "Kubenode guest command failed. Err ServerFaultCode: Failed to authenticate with the guest operating system using the supplied credentials."  error
5. wait and the undeploy the supervisor vm
6. go to step one again.

In the /var/log/vxpa.log I have found also something that I think could be related as well.

2021-05-15T18:42:56.792Z info vpxa[3234187] [Originator@6876 sub=vpxLro opID=wcp-60a01325-domain-c3001-vm-42188-88-c0] [VpxLRO] -- BEGIN lro-80464 -- guestOperationsFileManager -- vim.vm.guest.FileManager.initiateFileTransferFromGuest -- 526d512b-858c-5e51-4250-60e4acc82c4b
2021-05-15T18:42:57.372Z info vpxa[2100136] [Originator@6876 sub=vpxLro opID=wcp-60a01325-domain-c3001-vm-42186-9a-2d] [VpxLRO] -- BEGIN lro-80465 -- guestOperationsFileManager -- vim.vm.guest.FileManager.initiateFileTransferFromGuest -- 526d512b-858c-5e51-4250-60e4acc82c4b
2021-05-15T18:42:59.339Z error vpxa[3234187] [Originator@6876 sub=vpxaVmomi opID=wcp-60a01325-domain-c3001-vm-42188-88-c0] [VpxaClientAdapter::InvokeCommon] Re-throwing method-fault 'N3Vim5Fault17InvalidGuestLogin9ExceptionE(Fault cause: vim.fault.InvalidGuestLogin
--> )
--> [context]zKq7AVICAgAAAFgFDQEddnB4YQAAfeQ/bGlidm1hY29yZS5zbwAAsm8cAC3ZGgBSWhYBtijrbGlidmltLXR5cGVzLnNvAIFhggUBgfskBgGB7uMBAQLK7hBsaWJ2bW9taS5zbwACu8MYAmZVGANp0hl2cHhhAAPf1BkCfPoSgVUTEwGBWIASAQNL5SgDuqArA4HKGQPMdioDVqMqA8qyKgMZTSoDVBsrABwxKwCNbCsAUmE8BDt9AGxpYnB0aHJlYWQuc28uMAAFragObGliYy5zby42AA==[/context]' received while invoking initiateFileTransferFromGuest on vim.vm.guest.FileManager:ha-guest-operations-file-manager
2021-05-15T18:42:59.346Z info vpxa[3234187] [Originator@6876 sub=vpxLro opID=wcp-60a01325-domain-c3001-vm-42188-88-c0] [VpxLRO] -- FINISH lro-80464
2021-05-15T18:42:59.346Z info vpxa[3234187] [Originator@6876 sub=Default opID=wcp-60a01325-domain-c3001-vm-42188-88-c0] [VpxLRO] -- ERROR lro-80464 -- guestOperationsFileManager -- vim.vm.guest.FileManager.initiateFileTransferFromGuest: vim.fault.InvalidGuestLogin:
--> Result:
--> (vim.fault.InvalidGuestLogin) {
-->    faultCause = (vmodl.MethodFault) null,
-->    faultMessage = <unset>
-->    msg = "Received SOAP response fault from [<cs p:00000050be9a2180, TCP:localhost:8307>]: initiateFileTransferFromGuest
--> Failed to authenticate with the guest operating system using the supplied credentials."
--> }
--> Args:
-->
--> Arg vm:
--> 'vim.VirtualMachine:440'
--> Arg auth:
--> (vim.vm.guest.NamePasswordAuthentication) {
-->    interactiveSession = false,
-->    username = "root",
-->    password = (not shown
--> }
--> Arg guestFilePath:
--> "/etc/vmware/wcp/.node_configuration_status"

[dnord]

Now I have tried to install Tanzu a whole month! Possibly I had a break through now though.. From the Storage Policy I disabled VM encryption in the Storage Policy and redeployed AGAIN! (must have deployed this a million times now) and suddenly out of the blue I noticed in the wcp-logs [kubelifecycle/master_api_monitor.go:50] healthz for 10.0.50.100 = "ok" and the supervisor cluster in the vcenter UI had suddenly status Running.

[Kelvin_Koh]

disabled VM encryption resolved my issue