Hot Shot
Hot Shot

[vRA 7.5] - Distributed Execution Manager : Error 503 & 401

Hi everyone,

we have a strange error on Distributed Execution Manager :

2019-12-31T10:47:10.096Z INS550 vcac: [component="iaas:DynamicOps.DEM.exe" priority="Error" thread="5036"] [sub-thread-Id="16"  context=""  token=""] Error occurred writing to the repository tracking log

System.InvalidOperationException: Failed to save changes due to HTTP exception exceeding the maximum number of save change attempts to fail.  Retry query later. ---> System.Net.WebException: The remote server returned an error: (503) Server Unavailable.

   at System.Data.Services.Client.BatchSaveResult.BatchRequest()

   at System.Data.Services.Client.DataServiceContext.SaveChanges(SaveChangesOptions options)

   at DynamicOps.Repository.RepositoryServiceContext.SaveChanges(SaveChangesOptions options)

   --- End of inner exception stack trace ---

   at DynamicOps.Repository.RepositoryServiceContext.ValidateRetryCount(Int32& retryCount, Int32& cycleCount, String message, Exception innerException)

   at DynamicOps.Repository.RepositoryServiceContext.SaveChanges(SaveChangesOptions options)

   at DynamicOps.Repository.Tracking.RepoLoggingSingleton.WriteExceptionToLogs(String message, Exception exceptionObject, Boolean writeAsWarning)

INNER EXCEPTION: System.Net.WebException: The remote server returned an error: (503) Server Unavailable.

   at System.Data.Services.Client.BatchSaveResult.BatchRequest()

   at System.Data.Services.Client.DataServiceContext.SaveChanges(SaveChangesOptions options)

   at DynamicOps.Repository.RepositoryServiceContext.SaveChanges(SaveChangesOptions options)

2019-12-31T10:47:10.628Z INS550 vcac: [component="iaas:DynamicOps.DEM.exe" priority="Error" thread="9032"] [sub-thread-Id="12"  context=""  token=""] Error occurred writing to the repository tracking log

System.InvalidOperationException: Failed to save changes due to HTTP exception exceeding the maximum number of save change attempts to fail.  Retry query later. ---> System.Net.WebException: The remote server returned an error: (503) Server Unavailable.

   at System.Data.Services.Client.BatchSaveResult.BatchRequest()

   at System.Data.Services.Client.DataServiceContext.SaveChanges(SaveChangesOptions options)

   at DynamicOps.Repository.RepositoryServiceContext.SaveChanges(SaveChangesOptions options)

   --- End of inner exception stack trace ---

   at DynamicOps.Repository.RepositoryServiceContext.ValidateRetryCount(Int32& retryCount, Int32& cycleCount, String message, Exception innerException)

   at DynamicOps.Repository.RepositoryServiceContext.SaveChanges(SaveChangesOptions options)

   at DynamicOps.Repository.Tracking.RepoLoggingSingleton.WriteExceptionToLogs(String message, Exception exceptionObject, Boolean writeAsWarning)

INNER EXCEPTION: System.Net.WebException: The remote server returned an error: (503) Server Unavailable.

   at System.Data.Services.Client.BatchSaveResult.BatchRequest()

   at System.Data.Services.Client.DataServiceContext.SaveChanges(SaveChangesOptions options)

   at DynamicOps.Repository.RepositoryServiceContext.SaveChanges(SaveChangesOptions options)

2019-12-31T10:47:48.694Z INS550 vcac: [component="iaas:DynamicOps.DEM.exe" priority="Error" thread="740"] [sub-thread-Id="9"  context=""  token=""] Error occurred writing to the repository tracking log

System.Net.WebException: The remote server returned an error: (404) Not Found.

   at System.Data.Services.Client.BatchSaveResult.BatchRequest()

   at System.Data.Services.Client.DataServiceContext.SaveChanges(SaveChangesOptions options)

   at DynamicOps.Repository.RepositoryServiceContext.SaveChanges(SaveChangesOptions options)

   at DynamicOps.Repository.Tracking.RepoLoggingSingleton.WriteExceptionToLogs(String message, Exception exceptionObject, Boolean writeAsWarning)

2019-12-31T10:47:50.976Z INS550 vcac: [component="iaas:DynamicOps.DEM.exe" priority="Error" thread="9000"] [sub-thread-Id="11"  context=""  token=""] Error occurred writing to the repository tracking log

System.Net.WebException: The remote server returned an error: (404) Not Found.

   at System.Data.Services.Client.BatchSaveResult.BatchRequest()

   at System.Data.Services.Client.DataServiceContext.SaveChanges(SaveChangesOptions options)

   at DynamicOps.Repository.RepositoryServiceContext.SaveChanges(SaveChangesOptions options)

   at DynamicOps.Repository.Tracking.RepoLoggingSingleton.WriteExceptionToLogs(String message, Exception exceptionObject, Boolean writeAsWarning)

We have check the "IIS application Pools" :

- DefaultAppPool

- RepositoryAppPool

- vCACAppPool

- WapiAppPool

all are started & the identity used is formatted like this : Domain\IaasUser

This account can "Log on as a service" on IaaS Server.

All Windows services for vRealize Automation are started.

Do you have an idea for a workaround please ?

Thx for your help.

0 Kudos
2 Replies
VMware Employee
VMware Employee

Can you check if you are able to ping and nslookup to the vRA appliances and the Manager servers from the DEM machines in question ?

0 Kudos
Hot Shot
Hot Shot

Hi,

NslookUp and Ping on hostname and FQDN vRA appliance it's ok.

0 Kudos