Hi all -
Been noticing these rather frequent errors in the logs of a lab that was upgraded to vRA 7.5. I've dug around, restarted IaaS nodes, etc. but can't seem to shake this error:
"
Exception has been thrown by the target of an invocation.
Inner exception: One or more errors occurred.
Inner exception: One or more errors occurred.
Inner exception: An error occurred while sending the request.
Inner exception: The underlying connection was closed: An unexpected error occurred on a send.
Inner exception: Authentication failed because the remote party has closed the transport stream."
The full stack trace is here:
System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation. ---> System.AggregateException: One or more errors occurred. ---> System.AggregateException: One or more errors occurred. ---> System.Net.Http.HttpRequestException: An error occurred while sending the request. ---> System.Net.WebException: The underlying connection was closed: An unexpected error occurred on a send. ---> System.IO.IOException: Authentication failed because the remote party has closed the transport stream.
at System.Net.Security.SslState.StartReadFrame(Byte[] buffer, Int32 readBytes, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.StartReceiveBlob(Byte[] buffer, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.CheckCompletionBeforeNextReceive(ProtocolToken message, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.ForceAuthentication(Boolean receiveFirst, Byte[] buffer, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.ProcessAuthentication(LazyAsyncResult lazyResult)
at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
at System.Net.TlsStream.ProcessAuthentication(LazyAsyncResult result)
at System.Net.TlsStream.BeginWrite(Byte[] buffer, Int32 offset, Int32 size, AsyncCallback asyncCallback, Object asyncState)
at System.Net.ConnectStream.WriteHeaders(Boolean async)
--- End of inner exception stack trace ---
at System.Net.HttpWebRequest.EndGetResponse(IAsyncResult asyncResult)
at System.Net.Http.HttpClientHandler.GetResponseCallback(IAsyncResult ar)
--- End of inner exception stack trace ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at VMware.Cafe.SignRestRequestMessageHandler.<SendAsync>d__0.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at DynamicOps.Common.Client.RestClient.<>c__DisplayClasse`1.<<GetResourceOrDefault>b__d>d__10.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at DynamicOps.Common.Client.RestClient.<RetryWebRequestWrapper>d__ad.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at DynamicOps.Common.Client.RestClient.<GetResourceOrDefault>d__14`1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at VMware.Cafe.ComponentRegistryClient.<FindDefaultEndpointForEndpointType>d__4d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at VMware.Cafe.ComponentRegistryClientFactory.<GetEndpointAsync>d__c.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at VMware.Cafe.ComponentRegistryClientFactory.<CreateAsyncHelper>d__33`1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at VMware.Cafe.ComponentRegistryClientFactory.<CreateAsync>d__19`1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at VMware.Cafe.ComponentRegistryClientFactory.<CreateAsync>d__16`1.MoveNext()
--- End of inner exception stack trace ---
at System.Threading.Tasks.Task`1.GetResultCore(Boolean waitCompletionNotification)
at VMware.Cafe.ComponentRegistryClientFactory.CreateAsync[T](SecurityContext user, CreateOptions opts)
at VMware.Cafe.ComponentRegistryClientFactory.<CreateAsync>d__19`1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at VMware.Cafe.ComponentRegistryClientFactory.<CreateAsync>d__16`1.MoveNext()
--- End of inner exception stack trace ---
at System.Threading.Tasks.Task`1.GetResultCore(Boolean waitCompletionNotification)
at DynamicOps.Repository.Runtime.SecurityModel.CafeSecurityProvider.LoadSecurityInformation(UserIdentity userIdentity)
at DynamicOps.Repository.Runtime.SecurityModel.SecurityModelContext.GetIdentityTasksFromCache(UserIdentity userIdentity)
at DynamicOps.Repository.Runtime.SecurityModel.SecurityModelContext.get_IdentityTasks()
at DynamicOps.Repository.Runtime.ServiceModel.Data.RepositoryDataService`2.CalculateReadPermissionScopes(Int32 entityId)
at DynamicOps.Repository.Runtime.ServiceModel.Data.RepositoryDataService`2.InternalOnQueryEntity[TEntity](Int32 entityId)
--- End of inner exception stack trace ---
at System.RuntimeMethodHandle.InvokeMethod(Object target, Object[] arguments, Signature sig, Boolean constructor)
at System.Reflection.RuntimeMethodInfo.UnsafeInvokeInternal(Object obj, Object[] parameters, Object[] arguments)
at System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
at System.Data.Services.DataServiceConfiguration.ComposeResourceContainer(IDataService service, ResourceSetWrapper container, Expression queryExpression)
at System.Data.Services.RequestUriProcessor.ComposeExpressionForEntitySet(SegmentInfo segment, IDataService service, Boolean isLastSegment, Boolean checkRights)
at System.Data.Services.RequestUriProcessor.ComposeExpressionForSegments(IList`1 segments, IDataService service, Boolean isCrossReferencingUri)
at System.Data.Services.RequestUriProcessor.ProcessRequestUri(Uri absoluteRequestUri, IDataService service, Boolean internalQuery)
at System.Data.Services.DataService`1.HandleRequest()
Any help is appreciated!