vRA 7 Errors on Manager Service and Repository

[wllp]

I have lately been seeing errors in the vRA Logs (Infrastructure > Monitoring > Log)

Has anyone encountered with any of the below errors? Please kindly advise.

Error	10/29/2016 2:16 PM UTC+08:00	Repository	Repository	IAAS01\Administrator	IAAS01	-1 Inner exception: The underlying provider failed on Open. Inner exception: A transport-level error has occurred when receiving results from the server. (provider: Session Provider, error: 19 - Physical connection is not usable)
Error	10/29/2016 2:16 PM UTC+08:00	Repository	Repository	IAAS01\Administrator	IAAS01	-1 Inner exception: An error occurred while updating the entries. See the inner exception for details. Inner exception: A transport-level error has occurred when receiving results from the server. (provider: Session Provider, error: 19 - Physical connection is not usable)
Error	10/29/2016 2:01 PM UTC+08:00	Manager Service	Manager Service	IAAS01\Administrator	IAAS01	Error processing workflow creation Error executing query usp_SearchInitializingRequestVirtualMachines Inner Exception: A transport-level error has occurred when receiving results from the server. (provider: Session Provider, error: 19 - Physical connection is not usable)
rror	10/29/2016 1:37 PM UTC+08:00	Manager Service	Manager Service	IAAS01\Administrator	IAAS01	Error processing bulk machine request Error executing query usp_SelectRequest Inner Exception: A transport-level error has occurred when receiving results from the server. (provider: Session Provider, error: 19 - Physical connection is not usable)
Error	10/29/2016 1:37 PM UTC+08:00	Manager Service	Manager Service	IAAS01\Administrator	IAAS01	Error processing machine request Error executing query usp_SearchRequestedVirtualMachines Inner Exception: A transport-level error has occurred when receiving results from the server. (provider: Session Provider, error: 19 - Physical connection is not usable)
Error	10/29/2016 1:37 PM UTC+08:00	Manager Service	Manager Service	IAAS01\Administrator	IAAS01	Error processing workflow creation Error executing query usp_SearchInitializingRequestVirtualMachines Inner Exception: A transport-level error has occurred when receiving results from the server. (provider: Session Provider, error: 19 - Physical connection is not usable)
Error	10/29/2016 1:37 PM UTC+08:00	DEM-Worker	DEM	IAAS01\Administrator	IAAS01	Error occurred while pinging repository An error occurred while processing this request. Inner Exception: <?xml version="1.0" encoding="utf-8"?><m:error xmlns:m="http://schemas.microsoft.com/ado/2007/08/dataservices/metadata"><m:code /><m:message xml:lang="en-PH">An error occurred while processing this request.</m:message><m:innererror><m:message>-1</m:message><m:type>DynamicOps.Repository.Runtime.Common.RepositorySqlException</m:type><m:stacktrace></m:stacktrace><m:internalexception><m:message>The underlying provider failed on Open.</m:message><m:type>System.Data.EntityException</m:type><m:stacktrace> at System.Data.EntityClient.EntityConnection.OpenStoreConnectionIf(Boolean openCondition, DbConnection storeConnectionToOpen, DbConnection originalConnection, String exceptionCode, String attemptedOperation, Boolean&amp; closeStoreConnectionOnFailure)&#xD; at System.Data.EntityClient.EntityConnection.Open()&#xD; at System.Data.Objects.ObjectContext.EnsureConnection()&#xD; at System.Data.Objects.ObjectQuery`1.GetResults(Nullable`1 forMergeOption)&#xD; at System.Data.Objects.ObjectQuery`1.System.Collections.Generic.IEnumerable&lt;T&gt;.GetEnumerator()&#xD; at System.Data.Objects.ObjectQuery`1.GetEnumeratorInternal()&#xD; at System.Data.Services.WebUtil.GetRequestEnumerator(IEnumerable enumerable)&#xD; at System.Data.Services.QueryResultInfo.MoveNext()&#xD; at System.Data.Services.DataService`1.SerializeResponseBody(RequestDescription description, IDataService dataService, IODataResponseMessage responseMessage)&#xD; at System.Data.Services.DataService`1.HandleRequest()</m:stacktrace><m:internalexception><m:message>A transport-level error has occurred when receiving results from the server. (provider: Session Provider, error: 19 - Physical connection is not usable)</m:message><m:type>System.Data.SqlClient.SqlException</m:type><m:stacktrace> at System.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection, Action`1 wrapCloseInAction)&#xD; at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj, Boolean callerHasConnectionLock, Boolean asyncClose)&#xD; at System.Data.SqlClient.TdsParserStateObject.ReadSniError(TdsParserStateObject stateObj, UInt32 error)&#xD; at System.Data.SqlClient.TdsParserStateObject.ReadSniSyncOverAsync()&#xD; at System.Data.SqlClient.TdsParserStateObject.TryReadNetworkPacket()&#xD; at System.Data.SqlClient.TdsParserStateObject.TryPrepareBuffer()&#xD; at System.Data.SqlClient.TdsParserStateObject.TryReadByte(Byte&amp; value)&#xD; at System.Data.SqlClient.TdsParser.TryRun(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj, Boolean&amp; dataReady)&#xD; at System.Data.SqlClient.TdsParser.Run(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj)&#xD; at System.Data.SqlClient.TdsParser.TdsExecuteTransactionManagerRequest(Byte[] buffer, TransactionManagerRequestType request, String transactionName, TransactionManagerIsolationLevel isoLevel, Int32 timeout, SqlInternalTransaction transaction, TdsParserStateObject stateObj, Boolean isDelegateControlRequest)&#xD; at System.Data.SqlClient.SqlInternalConnectionTds.ExecuteTransactionYukon(TransactionRequest transactionRequest, String transactionName, IsolationLevel iso, SqlInternalTransaction internalTransaction, Boolean isDelegateControlRequest)&#xD; at System.Data.SqlClient.SqlDelegatedTransaction.Initialize()&#xD; at System.Transactions.TransactionStatePSPEOperation.PSPEInitialize(InternalTransaction tx, IPromotableSinglePhaseNotification promotableSinglePhaseNotification)&#xD; at System.Transactions.TransactionStateActive.EnlistPromotableSinglePhase(InternalTransaction tx, IPromotableSinglePhaseNotification promotableSinglePhaseNotification, Transaction atomicTransaction)&#xD; at System.Transactions.Transaction.EnlistPromotableSinglePhase(IPromotableSinglePhaseNotification promotableSinglePhaseNotification)&#xD; at System.Data.SqlClient.SqlInternalConnection.EnlistNonNull(Transaction tx)&#xD; at System.Data.ProviderBase.DbConnectionPool.PrepareConnection(DbConnection owningObject, DbConnectionInternal obj, Transaction transaction)&#xD; at System.Data.ProviderBase.DbConnectionPool.TryGetConnection(DbConnection owningObject, UInt32 waitForMultipleObjectsTimeout, Boolean allowCreate, Boolean onlyOneCheckConnection, DbConnectionOptions userOptions, DbConnectionInternal&amp; connection)&#xD; at System.Data.ProviderBase.DbConnectionPool.TryGetConnection(DbConnection owningObject, TaskCompletionSource`1 retry, DbConnectionOptions userOptions, DbConnectionInternal&amp; connection)&#xD; at System.Data.ProviderBase.DbConnectionFactory.TryGetConnection(DbConnection owningConnection, TaskCompletionSource`1 retry, DbConnectionOptions userOptions, DbConnectionInternal oldConnection, DbConnectionInternal&amp; connection)&#xD; at System.Data.ProviderBase.DbConnectionInternal.TryOpenConnectionInternal(DbConnection outerConnection, DbConnectionFactory connectionFactory, TaskCompletionSource`1 retry, DbConnectionOptions userOptions)&#xD; at System.Data.SqlClient.SqlConnection.TryOpenInner(TaskCompletionSource`1 retry)&#xD; at System.Data.SqlClient.SqlConnection.TryOpen(TaskCompletionSource`1 retry)&#xD; at System.Data.SqlClient.SqlConnection.Open()&#xD; at System.Data.EntityClient.EntityConnection.OpenStoreConnectionIf(Boolean openCondition, DbConnection storeConnectionToOpen, DbConnection originalConnection, String exceptionCode, String attemptedOperation, Boolean&amp; closeStoreConnectionOnFailure)</m:stacktrace></m:internalexception></m:internalexception></m:innererror></m:error>
Warning	10/29/2016 1:37 PM UTC+08:00	DEM-Orchestrator	iaas01DEO	IAAS01\Administrator	IAAS01	Error on Orchestrator Task Schedule Workflows An error occurred while processing this request. Inner Exception: <?xml version="1.0" encoding="utf-8"?><m:error xmlns:m="http://schemas.microsoft.com/ado/2007/08/dataservices/metadata"><m:code /><m:message xml:lang="en-PH">An error occurred while processing this request.</m:message><m:innererror><m:message>-1</m:message><m:type>DynamicOps.Repository.Runtime.Common.RepositorySqlException</m:type><m:stacktrace></m:stacktrace><m:internalexception><m:message>The underlying provider failed on Open.</m:message><m:type>System.Data.EntityException</m:type><m:stacktrace> at System.Data.EntityClient.EntityConnection.OpenStoreConnectionIf(Boolean openCondition, DbConnection storeConnectionToOpen, DbConnection originalConnection, String exceptionCode, String attemptedOperation, Boolean&amp; closeStoreConnectionOnFailure)&#xD; at System.Data.EntityClient.EntityConnection.Open()&#xD; at System.Data.Objects.ObjectContext.EnsureConnection()&#xD; at System.Data.Objects.ObjectQuery`1.GetResults(Nullable`1 forMergeOption)&#xD; at System.Data.Objects.ObjectQuery`1.System.Collections.Generic.IEnumerable&lt;T&gt;.GetEnumerator()&#xD; at System.Data.Objects.ObjectQuery`1.GetEnumeratorInternal()&#xD; at System.Data.Services.WebUtil.GetRequestEnumerator(IEnumerable enumerable)&#xD; at System.Data.Services.QueryResultInfo.MoveNext()&#xD; at System.Data.Services.DataService`1.SerializeResponseBody(RequestDescription description, IDataService dataService, IODataResponseMessage responseMessage)&#xD; at System.Data.Services.DataService`1.HandleRequest()</m:stacktrace><m:internalexception><m:message>A transport-level error has occurred when receiving results from the server. (provider: Session Provider, error: 19 - Physical connection is not usable)</m:message><m:type>System.Data.SqlClient.SqlException</m:type><m:stacktrace> at System.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection, Action`1 wrapCloseInAction)&#xD; at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj, Boolean callerHasConnectionLock, Boolean asyncClose)&#xD; at System.Data.SqlClient.TdsParserStateObject.ReadSniError(TdsParserStateObject stateObj, UInt32 error)&#xD; at System.Data.SqlClient.TdsParserStateObject.ReadSniSyncOverAsync()&#xD; at System.Data.SqlClient.TdsParserStateObject.TryReadNetworkPacket()&#xD; at System.Data.SqlClient.TdsParserStateObject.TryPrepareBuffer()&#xD; at System.Data.SqlClient.TdsParserStateObject.TryReadByte(Byte&amp; value)&#xD; at System.Data.SqlClient.TdsParser.TryRun(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj, Boolean&amp; dataReady)&#xD; at System.Data.SqlClient.TdsParser.Run(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj)&#xD; at System.Data.SqlClient.TdsParser.TdsExecuteTransactionManagerRequest(Byte[] buffer, TransactionManagerRequestType request, String transactionName, TransactionManagerIsolationLevel isoLevel, Int32 timeout, SqlInternalTransaction transaction, TdsParserStateObject stateObj, Boolean isDelegateControlRequest)&#xD; at System.Data.SqlClient.SqlInternalConnectionTds.ExecuteTransactionYukon(TransactionRequest transactionRequest, String transactionName, IsolationLevel iso, SqlInternalTransaction internalTransaction, Boolean isDelegateControlRequest)&#xD; at System.Data.SqlClient.SqlDelegatedTransaction.Initialize()&#xD; at System.Transactions.TransactionStatePSPEOperation.PSPEInitialize(InternalTransaction tx, IPromotableSinglePhaseNotification promotableSinglePhaseNotification)&#xD; at System.Transactions.TransactionStateActive.EnlistPromotableSinglePhase(InternalTransaction tx, IPromotableSinglePhaseNotification promotableSinglePhaseNotification, Transaction atomicTransaction)&#xD; at System.Transactions.Transaction.EnlistPromotableSinglePhase(IPromotableSinglePhaseNotification promotableSinglePhaseNotification)&#xD; at System.Data.SqlClient.SqlInternalConnection.EnlistNonNull(Transaction tx)&#xD; at System.Data.ProviderBase.DbConnectionPool.PrepareConnection(DbConnection owningObject, DbConnectionInternal obj, Transaction transaction)&#xD; at System.Data.ProviderBase.DbConnectionPool.TryGetConnection(DbConnection owningObject, UInt32 waitForMultipleObjectsTimeout, Boolean allowCreate, Boolean onlyOneCheckConnection, DbConnectionOptions userOptions, DbConnectionInternal&amp; connection)&#xD; at System.Data.ProviderBase.DbConnectionPool.TryGetConnection(DbConnection owningObject, TaskCompletionSource`1 retry, DbConnectionOptions userOptions, DbConnectionInternal&amp; connection)&#xD; at System.Data.ProviderBase.DbConnectionFactory.TryGetConnection(DbConnection owningConnection, TaskCompletionSource`1 retry, DbConnectionOptions userOptions, DbConnectionInternal oldConnection, DbConnectionInternal&amp; connection)&#xD; at System.Data.ProviderBase.DbConnectionInternal.TryOpenConnectionInternal(DbConnection outerConnection, DbConnectionFactory connectionFactory, TaskCompletionSource`1 retry, DbConnectionOptions userOptions)&#xD; at System.Data.SqlClient.SqlConnection.TryOpenInner(TaskCompletionSource`1 retry)&#xD; at System.Data.SqlClient.SqlConnection.TryOpen(TaskCompletionSource`1 retry)&#xD; at System.Data.SqlClient.SqlConnection.Open()&#xD; at System.Data.EntityClient.EntityConnection.OpenStoreConnectionIf(Boolean openCondition, DbConnection storeConnectionToOpen, DbConnection originalConnection, String exceptionCode, String attemptedOperation, Boolean&amp; closeStoreConnectionOnFailure)</m:stacktrace></m:internalexception></m:internalexception></m:innererror></m:error>
Warning	10/29/2016 1:37 PM UTC+08:00	DEM-Orchestrator	iaas01DEO	IAAS01\Administrator	IAAS01	Error on Orchestrator Task Check DEMs An error occurred while processing this request. Inner Exception: <?xml version="1.0" encoding="utf-8"?><m:error xmlns:m="http://schemas.microsoft.com/ado/2007/08/dataservices/metadata"><m:code /><m:message xml:lang="en-PH">An error occurred while processing this request.</m:message><m:innererror><m:message>-1</m:message><m:type>DynamicOps.Repository.Runtime.Common.RepositorySqlException</m:type><m:stacktrace></m:stacktrace><m:internalexception><m:message>The underlying provider failed on Open.</m:message><m:type>System.Data.EntityException</m:type><m:stacktrace> at System.Data.EntityClient.EntityConnection.OpenStoreConnectionIf(Boolean openCondition, DbConnection storeConnectionToOpen, DbConnection originalConnection, String exceptionCode, String attemptedOperation, Boolean&amp; closeStoreConnectionOnFailure)&#xD; at System.Data.EntityClient.EntityConnection.Open()&#xD; at System.Data.Objects.ObjectContext.EnsureConnection()&#xD; at System.Data.Objects.ObjectQuery`1.GetResults(Nullable`1 forMergeOption)&#xD; at System.Data.Objects.ObjectQuery`1.System.Collections.Generic.IEnumerable&lt;T&gt;.GetEnumerator()&#xD; at System.Data.Objects.ObjectQuery`1.GetEnumeratorInternal()&#xD; at System.Data.Services.WebUtil.GetRequestEnumerator(IEnumerable enumerable)&#xD; at System.Data.Services.QueryResultInfo.MoveNext()&#xD; at System.Data.Services.DataService`1.SerializeResponseBody(RequestDescription description, IDataService dataService, IODataResponseMessage responseMessage)&#xD; at System.Data.Services.DataService`1.HandleRequest()</m:stacktrace><m:internalexception><m:message>A transport-level error has occurred when receiving results from the server. (provider: Session Provider, error: 19 - Physical connection is not usable)</m:message><m:type>System.Data.SqlClient.SqlException</m:type><m:stacktrace> at System.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection, Action`1 wrapCloseInAction)&#xD; at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj, Boolean callerHasConnectionLock, Boolean asyncClose)&#xD; at System.Data.SqlClient.TdsParserStateObject.ReadSniError(TdsParserStateObject stateObj, UInt32 error)&#xD; at System.Data.SqlClient.TdsParserStateObject.ReadSniSyncOverAsync()&#xD; at System.Data.SqlClient.TdsParserStateObject.TryReadNetworkPacket()&#xD; at System.Data.SqlClient.TdsParserStateObject.TryPrepareBuffer()&#xD; at System.Data.SqlClient.TdsParserStateObject.TryReadByte(Byte&amp; value)&#xD; at System.Data.SqlClient.TdsParser.TryRun(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj, Boolean&amp; dataReady)&#xD; at System.Data.SqlClient.TdsParser.Run(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj)&#xD; at System.Data.SqlClient.TdsParser.TdsExecuteTransactionManagerRequest(Byte[] buffer, TransactionManagerRequestType request, String transactionName, TransactionManagerIsolationLevel isoLevel, Int32 timeout, SqlInternalTransaction transaction, TdsParserStateObject stateObj, Boolean isDelegateControlRequest)&#xD; at System.Data.SqlClient.SqlInternalConnectionTds.ExecuteTransactionYukon(TransactionRequest transactionRequest, String transactionName, IsolationLevel iso, SqlInternalTransaction internalTransaction, Boolean isDelegateControlRequest)&#xD; at System.Data.SqlClient.SqlDelegatedTransaction.Initialize()&#xD; at System.Transactions.TransactionStatePSPEOperation.PSPEInitialize(InternalTransaction tx, IPromotableSinglePhaseNotification promotableSinglePhaseNotification)&#xD; at System.Transactions.TransactionStateActive.EnlistPromotableSinglePhase(InternalTransaction tx, IPromotableSinglePhaseNotification promotableSinglePhaseNotification, Transaction atomicTransaction)&#xD; at System.Transactions.Transaction.EnlistPromotableSinglePhase(IPromotableSinglePhaseNotification promotableSinglePhaseNotification)&#xD; at System.Data.SqlClient.SqlInternalConnection.EnlistNonNull(Transaction tx)&#xD; at System.Data.ProviderBase.DbConnectionPool.PrepareConnection(DbConnection owningObject, DbConnectionInternal obj, Transaction transaction)&#xD; at System.Data.ProviderBase.DbConnectionPool.TryGetConnection(DbConnection owningObject, UInt32 waitForMultipleObjectsTimeout, Boolean allowCreate, Boolean onlyOneCheckConnection, DbConnectionOptions userOptions, DbConnectionInternal&amp; connection)&#xD; at System.Data.ProviderBase.DbConnectionPool.TryGetConnection(DbConnection owningObject, TaskCompletionSource`1 retry, DbConnectionOptions userOptions, DbConnectionInternal&amp; connection)&#xD; at System.Data.ProviderBase.DbConnectionFactory.TryGetConnection(DbConnection owningConnection, TaskCompletionSource`1 retry, DbConnectionOptions userOptions, DbConnectionInternal oldConnection, DbConnectionInternal&amp; connection)&#xD; at System.Data.ProviderBase.DbConnectionInternal.TryOpenConnectionInternal(DbConnection outerConnection, DbConnectionFactory connectionFactory, TaskCompletionSource`1 retry, DbConnectionOptions userOptions)&#xD; at System.Data.SqlClient.SqlConnection.TryOpenInner(TaskCompletionSource`1 retry)&#xD; at System.Data.SqlClient.SqlConnection.TryOpen(TaskCompletionSource`1 retry)&#xD; at System.Data.SqlClient.SqlConnection.Open()&#xD; at System.Data.EntityClient.EntityConnection.OpenStoreConnectionIf(Boolean openCondition, DbConnection storeConnectionToOpen, DbConnection originalConnection, String exceptionCode, String attemptedOperation, Boolean&amp; closeStoreConnectionOnFailure)</m:stacktrace></m:internalexception></m:internalexception></m:innererror></m:error>

[rcporto]

Your IAAS SQL database is running on a different server than your IAAS server? Take a look here: vRealize Automation Data collection doesn’t work » VCDX56

[wllp]

Hi Richardson,

Yes, the IaaS and IaaS SQL are running on 2 separate servers. I have already checked the DTC on both servers.

The errors that I am receiving are different from the post that you have provided. Any other ideas?

Thanks!