tail -f wcpsvc.log:
2020-11-11T07:56:37.167Z debug wcp [opID=5fab7f88] got DNS name for vm VirtualMachine:vm-47: 4211c94e768fe878a40910fb39f8668d
2020-11-11T07:56:37.171Z info wcp [opID=5fab7f88-domain-c9] Configuring 1 master agents on cluster domain-c9
2020-11-11T07:56:37.171Z debug wcp [opID=5fab7f88-domain-c9] Get VC Tag for WCP VM-VM Anti-Affinity
2020-11-11T07:56:37.171Z debug wcp [opID=5fab7f88-domain-c9] Rest client for vmodl2 API calls exists, checking session validity
2020-11-11T07:56:37.176Z debug wcp [opID=5fab7f88-domain-c9] Rest client for vmodl2 API calls is still valid.
2020-11-11T07:56:37.176Z debug wcp [opID=5fab7f88-domain-c9] Get VC Category for WCP VM-VM Anti-Affinity
2020-11-11T07:56:37.176Z debug wcp [opID=5fab7f88-domain-c9] Rest client for vmodl2 API calls exists, checking session validity
2020-11-11T07:56:37.182Z debug wcp [opID=5fab7f88-domain-c9] Rest client for vmodl2 API calls is still valid.
2020-11-11T07:56:37.198Z debug wcp [opID=5fab7f88-domain-c9] Found existing VC Category wp_vmvmaa_category
2020-11-11T07:56:37.206Z debug wcp [opID=5fab7f88-domain-c9] Found existing VC Tag wp_vmvmaa_tag
2020-11-11T07:56:37.207Z debug wcp [opID=5fab7f88-domain-c9] Rest client for vmodl2 API calls exists, checking session validity
2020-11-11T07:56:37.209Z debug wcp [opID=5fab7f88-domain-c9] Rest client for vmodl2 API calls is still valid.
2020-11-11T07:56:37.219Z debug wcp [opID=5fab7f88-domain-c9] Add VM VirtualMachine:vm-47 to ClusterModule 528a79e8-8f2c-6fd5-f6e1-0b03c4e7c899 for Supervisor Cluster Control Plane VMs
2020-11-11T07:56:37.22Z debug wcp [opID=5fab7f88-domain-c9] Rest client for vmodl2 API calls exists, checking session validity
2020-11-11T07:56:37.221Z debug wcp [opID=5fab7f88-domain-c9] Rest client for vmodl2 API calls is still valid.
2020-11-11T07:56:37.235Z info wcp [opID=5fab7f88-domain-c9] MULTIMASTER configuring agency Agency:5789dbed-89d8-4bf7-a877-26a061a471fd
2020-11-11T07:56:37.235Z info wcp [opID=5fab7f88-domain-c9] Configuring API server agent Agent:b0f38ccf-d4f1-47c8-b817-d5521f88c799 VM on cluster domain-c9.
2020-11-11T07:56:37.242Z debug wcp [opID=5fab7f88-domain-c9] Got cached machine ID: f21725ef-917b-4211-9219-00ec2ccdc6a5
2020-11-11T07:56:37.335Z debug wcp [opID=5fab7f88-domain-c9] AllocateIP request for VM: ClusterComputeResource:domain-c9 and nicID: 0
2020-11-11T07:56:37.336Z debug wcp [opID=5fab7f88-domain-c9] An IP 20.12.41.160 has already been allocated for ClusterComputeResource:domain-c9 0.
2020-11-11T07:56:37.339Z debug wcp [opID=5fab7f88-domain-c9] Got cached machine ID: f21725ef-917b-4211-9219-00ec2ccdc6a5
2020-11-11T07:56:37.407Z debug wcp [opID=5fab7f88-domain-c9] Successfully set permissions [{{} <nil> wcp-cluster-user-domain-c9-f21725ef-917b-4211-9219-00ec2ccdc6a5 false 1005 true}] on entity ClusterComputeResource:domain-c9
2020-11-11T07:56:37.46Z debug wcp [opID=5fab7f88-domain-c9] Successfully set permissions [{{} <nil> wcp-cluster-user-domain-c9-f21725ef-917b-4211-9219-00ec2ccdc6a5 false 1004 true}] on entity Folder:group-d1
2020-11-11T07:56:37.516Z debug wcp [opID=5fab7f88-domain-c9] Successfully set permissions [{{} <nil> wcp-cluster-user-domain-c9-f21725ef-917b-4211-9219-00ec2ccdc6a5 false 1005 true}] on entity ResourcePool:resgroup-40
2020-11-11T07:56:37.596Z debug wcp [opID=5fab7f88-domain-c9] Successfully set permissions [{{} <nil> wcp-cluster-user-domain-c9-f21725ef-917b-4211-9219-00ec2ccdc6a5 false 1005 true}] on entity Folder:group-v41
2020-11-11T07:56:37.596Z debug wcp [opID=5fab7f88-domain-c9] Got cached machine ID: f21725ef-917b-4211-9219-00ec2ccdc6a5
2020-11-11T07:56:37.666Z debug wcp [opID=5fab7f88-domain-c9] Successfully set permissions [{{} <nil> wcp-vmop-user-domain-c9-f21725ef-917b-4211-9219-00ec2ccdc6a5@vsphere.local false 1004 true}] on entity Folder:group-d1
2020-11-11T07:56:37.749Z debug wcp [opID=5fab7f88-domain-c9] Successfully set permissions [{{} <nil> wcp-vmop-user-domain-c9-f21725ef-917b-4211-9219-00ec2ccdc6a5@vsphere.local false 1021 true}] on entity Datacenter:datacenter-3
2020-11-11T07:56:37.797Z debug wcp [opID=5fab7f88-domain-c9] Successfully set permissions [{{} <nil> wcp-vmop-user-domain-c9-f21725ef-917b-4211-9219-00ec2ccdc6a5@vsphere.local false 1021 true}] on entity ResourcePool:resgroup-40
2020-11-11T07:56:37.863Z debug wcp [opID=5fab7f88-domain-c9] Successfully set permissions [{{} <nil> wcp-vmop-user-domain-c9-f21725ef-917b-4211-9219-00ec2ccdc6a5@vsphere.local false 1021 true}] on entity Folder:group-v41
2020-11-11T07:56:37.864Z debug wcp [opID=5fab7f88-domain-c9] Got cached machine ID: f21725ef-917b-4211-9219-00ec2ccdc6a5
2020-11-11T07:56:37.864Z debug wcp [opID=5fab7f88-domain-c9] Permission client already exists, reuse it.
2020-11-11T07:56:37.949Z debug wcp [opID=5fab7f88-domain-c9] Set global permission ID=username=vsphere.local\wcp-vmop-user-domain-c9-f21725ef-917b-4211-9219-00ec2ccdc6a5;doc=urn:acl:global:permissions on principal wcp-vmop-user-domain-c9-f21725ef-917b-4211-9219-00ec2ccdc6a5.
2020-11-11T07:56:37.949Z debug wcp [opID=5fab7f88-domain-c9] Got cached machine ID: f21725ef-917b-4211-9219-00ec2ccdc6a5
2020-11-11T07:56:37.993Z debug wcp [opID=5fab7f88-domain-c9] Successfully set permissions [{{} <nil> wcp-appplatform-user-domain-c9-f21725ef-917b-4211-9219-00ec2ccdc6a5@vsphere.local false 1007 true}] on entity ClusterComputeResource:domain-c9
2020-11-11T07:56:38.041Z debug wcp [opID=5fab7f88-domain-c9] Successfully set permissions [{{} <nil> wcp-appplatform-user-domain-c9-f21725ef-917b-4211-9219-00ec2ccdc6a5@vsphere.local false 1008 true}] on entity Folder:group-d1
2020-11-11T07:56:38.041Z debug wcp [opID=5fab7f88-domain-c9] Got cached machine ID: f21725ef-917b-4211-9219-00ec2ccdc6a5
2020-11-11T07:56:38.041Z debug wcp [opID=5fab7f88-domain-c9] Permission client already exists, reuse it.
2020-11-11T07:56:38.075Z debug wcp [opID=5fab7f88-domain-c9] Set global permission ID=username=vsphere.local\wcp-appplatform-user-domain-c9-f21725ef-917b-4211-9219-00ec2ccdc6a5;doc=urn:acl:global:permissions on principal wcp-appplatform-user-domain-c9-f21725ef-917b-4211-9219-00ec2ccdc6a5.
2020-11-11T07:56:38.076Z debug wcp [opID=5fab7f88-domain-c9] Got cached machine ID: f21725ef-917b-4211-9219-00ec2ccdc6a5
2020-11-11T07:56:38.076Z debug wcp [opID=5fab7f88-domain-c9] Got cached machine ID: f21725ef-917b-4211-9219-00ec2ccdc6a5
2020-11-11T07:56:38.076Z debug wcp [opID=5fab7f88-domain-c9] Finding cluster netif for MasterNode: VirtualMachine:vm-47 in Cluster domain-c9.
2020-11-11T07:56:38.076Z warning wcp [opID=5fab7f88-domain-c9] Did not find any cluster netif for MasterNode: VirtualMachine:vm-47 in Cluster domain-c9.
2020-11-11T07:56:38.076Z debug wcp [opID=5fab7f88-domain-c9] Got cached machine ID: f21725ef-917b-4211-9219-00ec2ccdc6a5
2020-11-11T07:56:38.081Z debug wcp [opID=5fab7f88-domain-c9] Mgmt net device mac address for master VM vm-47: 00:50:56:91:c7:b2
2020-11-11T07:56:38.092Z debug wcp [opID=5fab7f88-domain-c9] VM VirtualMachine:vm-47 current Annotation: This Virtual Machine is a VMware agent implementing support for vSphere Workloads. Its lifecycle operations are managed by VMware vCenter Server.
EAM Agency: vmware-vsc-apiserver-cqvzl5
2020-11-11T07:56:38.092Z debug wcp [opID=5fab7f88-domain-c9] Found corresponding EAM Agency name for VM VirtualMachine:vm-47: vmware-vsc-apiserver-cqvzl5
2020-11-11T07:56:38.092Z debug wcp [opID=5fab7f88-domain-c9] VM VirtualMachine:vm-47 Annotation already contains EAM Agency Name, no need to reconfigure.
2020-11-11T07:56:38.17Z debug wcp [opID=5fab7f88-domain-c9] Failed to delete file from /dev/shm/secret.tmp: ServerFaultCode: File /dev/shm/secret.tmp was not found
2020-11-11T07:56:38.45Z info wcp [opID=5fab7f88-domain-c9] Certificate already exists on node.
2020-11-11T07:56:38.542Z error wcp [opID=5fab7f88-domain-c9] Failed to install API server. Err: Configure operation for the Master node VM with identifier vm-47 failed.
2020-11-11T07:56:38.542Z error wcp [opID=5fab7f88-domain-c9] Error configuring API server on cluster domain-c9 Configure operation for the Master node VM with identifier vm-47 failed.
2020-11-11T07:56:38.542Z debug wcp [opID=5fab7f88-domain-c9] Refreshing APIServerEndpoint and trustbundle.
2020-11-11T07:56:38.542Z debug wcp [opID=5fab7f88] get vm moref from agent info: VirtualMachine:vm-45
2020-11-11T07:56:38.542Z debug wcp [opID=5fab7f88] checking if vm-45 has been successfully configured
2020-11-11T07:56:38.542Z debug wcp [opID=5fab7f88] Request for VM auth for VirtualMachine:vm-45
2020-11-11T07:56:38.56Z error wcp [opID=5fab7f88] Failed to get kubenode VM VirtualMachine:vm-45 configuration status. Err ServerFaultCode: The attempted operation cannot be performed in the current state (Powered off).
2020-11-11T07:56:38.56Z error wcp [opID=5fab7f88] failed to get guest customization status. err API call to VMware vCenter Server (vpxd) failed. Details 'ServerFaultCode: The attempted operation cannot be performed in the current state (Powered off).'
2020-11-11T07:56:38.56Z debug wcp [opID=5fab7f88] vm for agency &{Agency:980f694b-ffd4-4e0b-bf70-2c5e1c9a5067 Agent:bc9b6963-3b25-4068-af49-fd0245d7f912 0xc0003b4780 <nil>} is not fully configured
2020-11-11T07:56:38.56Z debug wcp [opID=5fab7f88] get vm moref from agent info: VirtualMachine:vm-46
2020-11-11T07:56:38.56Z debug wcp [opID=5fab7f88] checking if vm-46 has been successfully configured
2020-11-11T07:56:38.56Z debug wcp [opID=5fab7f88] Request for VM auth for VirtualMachine:vm-46
2020-11-11T07:56:38.583Z error wcp [opID=5fab7f88] Failed to get kubenode VM VirtualMachine:vm-46 configuration status. Err ServerFaultCode: The attempted operation cannot be performed in the current state (Powered off).
2020-11-11T07:56:38.583Z error wcp [opID=5fab7f88] failed to get guest customization status. err API call to VMware vCenter Server (vpxd) failed. Details 'ServerFaultCode: The attempted operation cannot be performed in the current state (Powered off).'
2020-11-11T07:56:38.583Z debug wcp [opID=5fab7f88] vm for agency &{Agency:0aa3122d-63c2-4468-8b3e-8998ae27ee69 Agent:9d49bb66-d1f7-43ea-bfb2-4ece180b3b5b 0xc0001bc780 <nil>} is not fully configured
2020-11-11T07:56:38.583Z debug wcp [opID=5fab7f88] get vm moref from agent info: VirtualMachine:vm-47
2020-11-11T07:56:38.583Z debug wcp [opID=5fab7f88] checking if vm-47 has been successfully configured
2020-11-11T07:56:38.583Z debug wcp [opID=5fab7f88] Request for VM auth for VirtualMachine:vm-47
2020-11-11T07:56:38.786Z debug wcp [opID=5fab7f88] vm vm-47 guest configuration is not DONE: status Failed
2020-11-11T07:56:38.786Z debug wcp [opID=5fab7f88] vm for agency &{Agency:5789dbed-89d8-4bf7-a877-26a061a471fd Agent:b0f38ccf-d4f1-47c8-b817-d5521f88c799 0xc0001bc870 0xc001120000} is not fully configured
2020-11-11T07:56:38.786Z error wcp [opID=5fab7f88-domain-c9] there should be at least one master vm available but no associated moref
2020-11-11T07:56:38.786Z debug wcp Publish change event: &cdc.ChangeLogChangeEvent{Resource:std.DynamicID{Type_:"ClusterComputeResource", Id:"domain-c9"}, Kind:"UPDATE", Properties:[]string{"messages"}, ParentResources:[]std.DynamicID(nil)}
2020-11-11T07:56:38.786Z debug wcp [opID=5fab7f88] [ END ] [kubelifecycle.(*Controller).syncClusterState:285] [3.16022908s] cluster=domain-c9
2020-11-11T07:56:44.189Z debug wcp [opID=EAMAgent] Ignore non WCP agency vCLS
2020-11-11T07:56:44.218Z debug wcp informer.processLoop() lister.List() returned
2020-11-11T07:56:54.231Z debug wcp [opID=EAMAgent] Ignore non WCP agency vCLS
2020-11-11T07:56:54.272Z debug wcp informer.processLoop() lister.List() returned
2020-11-11T07:56:54.723Z info wcp [opID=vCLS] Getting all clusters in the inventory
2020-11-11T07:56:54.725Z info wcp [opID=vCLS] Returning the list of clusters: [ClusterComputeResource:domain-c9]
2020-11-11T07:56:54.748Z info wcp [opID=vCLS] Going to read service config
2020-11-11T07:56:54.748Z info wcp [opID=vCLS] Found OVF locator file:///storage/lifecycle/vmware-hdcs
2020-11-11T07:56:54.748Z info wcp [opID=vCLS] Found OVF directory as /storage/lifecycle/vmware-hdcs
2020-11-11T07:56:54.751Z info wcp [opID=vCLS] Agency &{0xc001615180 {Agency b27a2a00-d3e8-43cc-8ceb-0a9a0d1b133f}} status is green on cluster domain-c9
2020-11-11T07:57:04.285Z debug wcp [opID=EAMAgent] Ignore non WCP agency vCLS
2020-11-11T07:57:04.316Z debug wcp informer.processLoop() lister.List() returned
