1 Reply Latest reply on Jun 29, 2020 7:11 AM by daphnissov

    Enable workload management hangs on configuring

    mashio Lurker

      i'm trying to configure TKG cluster on vsphere 7 for the first time.

      NSX-T 3.0 configured and running.

      when i enable workload management with all required info it's never finished configuring.

      i can see in wcp log many messages in loop.

       

      attaching error messages that repeatedly showing in the log:

       

      020-06-29T12:52:51.438Z debug wcp informer.processLoop() lister.List() returned

      2020-06-29T12:52:54.612Z error wcp [opID=5ef9ca68-domain-c8] Unexpected object: &Status{ListMeta:ListMeta{SelfLink:,ResourceVersion:,Continue:,RemainingItemCount:nil,},Status:Failure,Message:an error on the server ("unable to decode an event from the watch stream: net/http: request canceled (Client.Timeout exceeded while reading body)") has prevented the request from succeeding,Reason:InternalError,Details:&StatusDetails{Name:,Group:,Kind:,Causes:[]StatusCause{StatusCause{Type:UnexpectedServerResponse,Message:unable to decode an event from the watch stream: net/http: request canceled (Client.Timeout exceeded while reading body),Field:,},StatusCause{Type:ClientWatchDecoding,Message:unable to decode an event from the watch stream: net/http: request canceled (Client.Timeout exceeded while reading body),Field:,},},RetryAfterSeconds:0,UID:,},Code:500,}

      2020-06-29T12:52:54.612Z error wcp [opID=5ef9ca68-domain-c8] Error watching NSX CRD resources.

      2020-06-29T12:52:54.612Z error wcp [opID=5ef9ca68-domain-c8] Error creating NSX resources. Err: Kubernetes API call failed. Details Error watching NSX CRD resources.

      2020-06-29T12:52:54.612Z error wcp [opID=5ef9ca68-domain-c8] Failed to create cluster network interface for MasterNode: VirtualMachine:vm-1008. Err: Kubernetes API call failed. Details Error watching NSX CRD resources.

      2020-06-29T12:52:54.612Z error wcp [opID=5ef9ca68-domain-c8] Error configuring API server on cluster domain-c8 An error occurred. This operation will be retried.

      2020-06-29T12:52:54.832Z error wcp [opID=5ef9ca68-domain-c8] Unexpected object: &Status{ListMeta:ListMeta{SelfLink:,ResourceVersion:,Continue:,RemainingItemCount:nil,},Status:Failure,Message:an error on the server ("unable to decode an event from the watch stream: net/http: request canceled (Client.Timeout exceeded while reading body)") has prevented the request from succeeding,Reason:InternalError,Details:&StatusDetails{Name:,Group:,Kind:,Causes:[]StatusCause{StatusCause{Type:UnexpectedServerResponse,Message:unable to decode an event from the watch stream: net/http: request canceled (Client.Timeout exceeded while reading body),Field:,},StatusCause{Type:ClientWatchDecoding,Message:unable to decode an event from the watch stream: net/http: request canceled (Client.Timeout exceeded while reading body),Field:,},},RetryAfterSeconds:0,UID:,},Code:500,}

      2020-06-29T12:52:54.832Z error wcp [opID=5ef9ca68-domain-c8] Error watching NSX CRD resources.

      2020-06-29T12:52:54.832Z error wcp [opID=5ef9ca68-domain-c8] Error creating NSX resources. Err: Kubernetes API call failed. Details Error watching NSX CRD resources.

      2020-06-29T12:52:54.832Z error wcp [opID=5ef9ca68-domain-c8] Failed to create cluster network interface for MasterNode: VirtualMachine:vm-1007. Err: Kubernetes API call failed. Details Error watching NSX CRD resources.

      2020-06-29T12:52:54.832Z error wcp [opID=5ef9ca68-domain-c8] Error configuring API server on cluster domain-c8 An error occurred. This operation will be retried.

      2020-06-29T12:52:54.957Z error wcp [opID=5ef9ca68-domain-c8] Unexpected object: &Status{ListMeta:ListMeta{SelfLink:,ResourceVersion:,Continue:,RemainingItemCount:nil,},Status:Failure,Message:an error on the server ("unable to decode an event from the watch stream: net/http: request canceled (Client.Timeout exceeded while reading body)") has prevented the request from succeeding,Reason:InternalError,Details:&StatusDetails{Name:,Group:,Kind:,Causes:[]StatusCause{StatusCause{Type:UnexpectedServerResponse,Message:unable to decode an event from the watch stream: net/http: request canceled (Client.Timeout exceeded while reading body),Field:,},StatusCause{Type:ClientWatchDecoding,Message:unable to decode an event from the watch stream: net/http: request canceled (Client.Timeout exceeded while reading body),Field:,},},RetryAfterSeconds:0,UID:,},Code:500,}

      2020-06-29T12:52:54.957Z error wcp [opID=5ef9ca68-domain-c8] Error watching NSX CRD resources.

      2020-06-29T12:52:54.957Z error wcp [opID=5ef9ca68-domain-c8] Error creating NSX resources. Err: Kubernetes API call failed. Details Error watching NSX CRD resources.

      2020-06-29T12:52:54.957Z error wcp [opID=5ef9ca68-domain-c8] Failed to create cluster network interface for MasterNode: VirtualMachine:vm-1006. Err: Kubernetes API call failed. Details Error watching NSX CRD resources.

      2020-06-29T12:52:54.957Z error wcp [opID=5ef9ca68-domain-c8] Error configuring API server on cluster domain-c8 An error occurred. This operation will be retried.

      2020-06-29T12:52:54.957Z info wcp [opID=5ef9ca68-domain-c8] no single master succeeded - retrying

      2020-06-29T12:52:54.957Z debug wcp Publish change event: &cdc.ChangeLogChangeEvent{Resource:std.DynamicID{Type_:"ClusterComputeResource", Id:"domain-c8"}, Kind:"UPDATE", Properties:[]string{"messages"}, ParentResources:[]std.DynamicID(nil)}

       

       

      does anyone had a similar issue to this?