VMware Cloud Community
ymichalak
Hot Shot
Hot Shot

[vRA 7.5] - Distributed Execution Manager : Error 503 & 401

Hi everyone,

we have a strange error on Distributed Execution Manager :

2019-12-31T10:47:10.096Z INS550 vcac: [component="iaas:DynamicOps.DEM.exe" priority="Error" thread="5036"] [sub-thread-Id="16"  context=""  token=""] Error occurred writing to the repository tracking log

System.InvalidOperationException: Failed to save changes due to HTTP exception exceeding the maximum number of save change attempts to fail.  Retry query later. ---> System.Net.WebException: The remote server returned an error: (503) Server Unavailable.

   at System.Data.Services.Client.BatchSaveResult.BatchRequest()

   at System.Data.Services.Client.DataServiceContext.SaveChanges(SaveChangesOptions options)

   at DynamicOps.Repository.RepositoryServiceContext.SaveChanges(SaveChangesOptions options)

   --- End of inner exception stack trace ---

   at DynamicOps.Repository.RepositoryServiceContext.ValidateRetryCount(Int32& retryCount, Int32& cycleCount, String message, Exception innerException)

   at DynamicOps.Repository.RepositoryServiceContext.SaveChanges(SaveChangesOptions options)

   at DynamicOps.Repository.Tracking.RepoLoggingSingleton.WriteExceptionToLogs(String message, Exception exceptionObject, Boolean writeAsWarning)

INNER EXCEPTION: System.Net.WebException: The remote server returned an error: (503) Server Unavailable.

   at System.Data.Services.Client.BatchSaveResult.BatchRequest()

   at System.Data.Services.Client.DataServiceContext.SaveChanges(SaveChangesOptions options)

   at DynamicOps.Repository.RepositoryServiceContext.SaveChanges(SaveChangesOptions options)

2019-12-31T10:47:10.628Z INS550 vcac: [component="iaas:DynamicOps.DEM.exe" priority="Error" thread="9032"] [sub-thread-Id="12"  context=""  token=""] Error occurred writing to the repository tracking log

System.InvalidOperationException: Failed to save changes due to HTTP exception exceeding the maximum number of save change attempts to fail.  Retry query later. ---> System.Net.WebException: The remote server returned an error: (503) Server Unavailable.

   at System.Data.Services.Client.BatchSaveResult.BatchRequest()

   at System.Data.Services.Client.DataServiceContext.SaveChanges(SaveChangesOptions options)

   at DynamicOps.Repository.RepositoryServiceContext.SaveChanges(SaveChangesOptions options)

   --- End of inner exception stack trace ---

   at DynamicOps.Repository.RepositoryServiceContext.ValidateRetryCount(Int32& retryCount, Int32& cycleCount, String message, Exception innerException)

   at DynamicOps.Repository.RepositoryServiceContext.SaveChanges(SaveChangesOptions options)

   at DynamicOps.Repository.Tracking.RepoLoggingSingleton.WriteExceptionToLogs(String message, Exception exceptionObject, Boolean writeAsWarning)

INNER EXCEPTION: System.Net.WebException: The remote server returned an error: (503) Server Unavailable.

   at System.Data.Services.Client.BatchSaveResult.BatchRequest()

   at System.Data.Services.Client.DataServiceContext.SaveChanges(SaveChangesOptions options)

   at DynamicOps.Repository.RepositoryServiceContext.SaveChanges(SaveChangesOptions options)

2019-12-31T10:47:48.694Z INS550 vcac: [component="iaas:DynamicOps.DEM.exe" priority="Error" thread="740"] [sub-thread-Id="9"  context=""  token=""] Error occurred writing to the repository tracking log

System.Net.WebException: The remote server returned an error: (404) Not Found.

   at System.Data.Services.Client.BatchSaveResult.BatchRequest()

   at System.Data.Services.Client.DataServiceContext.SaveChanges(SaveChangesOptions options)

   at DynamicOps.Repository.RepositoryServiceContext.SaveChanges(SaveChangesOptions options)

   at DynamicOps.Repository.Tracking.RepoLoggingSingleton.WriteExceptionToLogs(String message, Exception exceptionObject, Boolean writeAsWarning)

2019-12-31T10:47:50.976Z INS550 vcac: [component="iaas:DynamicOps.DEM.exe" priority="Error" thread="9000"] [sub-thread-Id="11"  context=""  token=""] Error occurred writing to the repository tracking log

System.Net.WebException: The remote server returned an error: (404) Not Found.

   at System.Data.Services.Client.BatchSaveResult.BatchRequest()

   at System.Data.Services.Client.DataServiceContext.SaveChanges(SaveChangesOptions options)

   at DynamicOps.Repository.RepositoryServiceContext.SaveChanges(SaveChangesOptions options)

   at DynamicOps.Repository.Tracking.RepoLoggingSingleton.WriteExceptionToLogs(String message, Exception exceptionObject, Boolean writeAsWarning)

We have check the "IIS application Pools" :

- DefaultAppPool

- RepositoryAppPool

- vCACAppPool

- WapiAppPool

all are started & the identity used is formatted like this : Domain\IaasUser

This account can "Log on as a service" on IaaS Server.

All Windows services for vRealize Automation are started.

Do you have an idea for a workaround please ?

Thx for your help.

Reply
0 Kudos
3 Replies
bnagabhushan
VMware Employee
VMware Employee

Can you check if you are able to ping and nslookup to the vRA appliances and the Manager servers from the DEM machines in question ?

Reply
0 Kudos
ymichalak
Hot Shot
Hot Shot

Hi,

NslookUp and Ping on hostname and FQDN vRA appliance it's ok.

Reply
0 Kudos
marlounb
Contributor
Contributor

Was there ever an answer to this question?

Reply
0 Kudos