mashio
Contributor
Contributor

Enable workload management hangs on configuring

i'm trying to configure TKG cluster on vsphere 7 for the first time.

NSX-T 3.0 configured and running.

when i enable workload management with all required info it's never finished configuring.

i can see in wcp log many messages in loop.

attaching error messages that repeatedly showing in the log:

020-06-29T12:52:51.438Z debug wcp informer.processLoop() lister.List() returned

2020-06-29T12:52:54.612Z error wcp [opID=5ef9ca68-domain-c8] Unexpected object: &Status{ListMeta:ListMeta{SelfLink:,ResourceVersion:,Continue:,RemainingItemCount:nil,},Status:Failure,Message:an error on the server ("unable to decode an event from the watch stream: net/http: request canceled (Client.Timeout exceeded while reading body)") has prevented the request from succeeding,Reason:InternalError,Details:&StatusDetails{Name:,Group:,Kind:,Causes:[]StatusCause{StatusCause{Type:UnexpectedServerResponse,Message:unable to decode an event from the watch stream: net/http: request canceled (Client.Timeout exceeded while reading body),Field:,},StatusCause{Type:ClientWatchDecoding,Message:unable to decode an event from the watch stream: net/http: request canceled (Client.Timeout exceeded while reading body),Field:,},},RetryAfterSeconds:0,UID:,},Code:500,}

2020-06-29T12:52:54.612Z error wcp [opID=5ef9ca68-domain-c8] Error watching NSX CRD resources.

2020-06-29T12:52:54.612Z error wcp [opID=5ef9ca68-domain-c8] Error creating NSX resources. Err: Kubernetes API call failed. Details Error watching NSX CRD resources.

2020-06-29T12:52:54.612Z error wcp [opID=5ef9ca68-domain-c8] Failed to create cluster network interface for MasterNode: VirtualMachine:vm-1008. Err: Kubernetes API call failed. Details Error watching NSX CRD resources.

2020-06-29T12:52:54.612Z error wcp [opID=5ef9ca68-domain-c8] Error configuring API server on cluster domain-c8 An error occurred. This operation will be retried.

2020-06-29T12:52:54.832Z error wcp [opID=5ef9ca68-domain-c8] Unexpected object: &Status{ListMeta:ListMeta{SelfLink:,ResourceVersion:,Continue:,RemainingItemCount:nil,},Status:Failure,Message:an error on the server ("unable to decode an event from the watch stream: net/http: request canceled (Client.Timeout exceeded while reading body)") has prevented the request from succeeding,Reason:InternalError,Details:&StatusDetails{Name:,Group:,Kind:,Causes:[]StatusCause{StatusCause{Type:UnexpectedServerResponse,Message:unable to decode an event from the watch stream: net/http: request canceled (Client.Timeout exceeded while reading body),Field:,},StatusCause{Type:ClientWatchDecoding,Message:unable to decode an event from the watch stream: net/http: request canceled (Client.Timeout exceeded while reading body),Field:,},},RetryAfterSeconds:0,UID:,},Code:500,}

2020-06-29T12:52:54.832Z error wcp [opID=5ef9ca68-domain-c8] Error watching NSX CRD resources.

2020-06-29T12:52:54.832Z error wcp [opID=5ef9ca68-domain-c8] Error creating NSX resources. Err: Kubernetes API call failed. Details Error watching NSX CRD resources.

2020-06-29T12:52:54.832Z error wcp [opID=5ef9ca68-domain-c8] Failed to create cluster network interface for MasterNode: VirtualMachine:vm-1007. Err: Kubernetes API call failed. Details Error watching NSX CRD resources.

2020-06-29T12:52:54.832Z error wcp [opID=5ef9ca68-domain-c8] Error configuring API server on cluster domain-c8 An error occurred. This operation will be retried.

2020-06-29T12:52:54.957Z error wcp [opID=5ef9ca68-domain-c8] Unexpected object: &Status{ListMeta:ListMeta{SelfLink:,ResourceVersion:,Continue:,RemainingItemCount:nil,},Status:Failure,Message:an error on the server ("unable to decode an event from the watch stream: net/http: request canceled (Client.Timeout exceeded while reading body)") has prevented the request from succeeding,Reason:InternalError,Details:&StatusDetails{Name:,Group:,Kind:,Causes:[]StatusCause{StatusCause{Type:UnexpectedServerResponse,Message:unable to decode an event from the watch stream: net/http: request canceled (Client.Timeout exceeded while reading body),Field:,},StatusCause{Type:ClientWatchDecoding,Message:unable to decode an event from the watch stream: net/http: request canceled (Client.Timeout exceeded while reading body),Field:,},},RetryAfterSeconds:0,UID:,},Code:500,}

2020-06-29T12:52:54.957Z error wcp [opID=5ef9ca68-domain-c8] Error watching NSX CRD resources.

2020-06-29T12:52:54.957Z error wcp [opID=5ef9ca68-domain-c8] Error creating NSX resources. Err: Kubernetes API call failed. Details Error watching NSX CRD resources.

2020-06-29T12:52:54.957Z error wcp [opID=5ef9ca68-domain-c8] Failed to create cluster network interface for MasterNode: VirtualMachine:vm-1006. Err: Kubernetes API call failed. Details Error watching NSX CRD resources.

2020-06-29T12:52:54.957Z error wcp [opID=5ef9ca68-domain-c8] Error configuring API server on cluster domain-c8 An error occurred. This operation will be retried.

2020-06-29T12:52:54.957Z info wcp [opID=5ef9ca68-domain-c8] no single master succeeded - retrying

2020-06-29T12:52:54.957Z debug wcp Publish change event: &cdc.ChangeLogChangeEvent{Resource:std.DynamicID{Type_:"ClusterComputeResource", Id:"domain-c8"}, Kind:"UPDATE", Properties:[]string{"messages"}, ParentResources:[]std.DynamicID(nil)}

does anyone had a similar issue to this?

0 Kudos
20 Replies
amdjfk
Contributor
Contributor

As it turns out, I hadd all MTU set to 9000 and K8S didn't work. It deployed (once I set up BGP on the switch), but I couldn't get to the Cluster Ingress IP. ONce I set the Overlay mtu to 1600 and the edge overlay mtu to 1600, everything worked. 

0 Kudos