Long processing results in Data Adapter timeout

[erikDoesStorage]

In our plugin we have encountered a situation where when we have a large configuration of our objects it takes a long time to process resulting in hitting a time out situation.  I couldn't find anything in the forum that addresses this (although the search tools are rather poor and it seems they can only be searched for a word that occurs in the subject of the forum entries).


In the virgo log file I see:

The data service execution timed out java.util.concurrent.TimeoutException: Tasks didn't complete within the allotted timeout: 120000 MILLISECONDS. The tasks that timed out are:


Further down in the virgo log I also see this related exception/error:

com.vmware.vise.data.query.DataServiceException: The data service timed out because a back-end task took more than 120 seconds. See the vsphere-client-virgo.log for more details.

 


In the case were we encounter this situation, we are determining the related information between datastores and our objects.  


We are doing a $.getJSON() call in our client side javascript to get the data with a URL parameter along the lines of + +"/vsphere-client/vcenterWebClient-ui/rest/data/properties/urn:vmomi:Datastore:datastore-14:bcfd5614-c4f9-4e5c-9d15-ac754e33c2f8?properties=System.fullInfo"


What can I do about this situation where we have to do a lot of processing/data gathering that exceeds this 120ms timeout?  


 


 

[laurentsd]

We added this timeout mechanism in 6.0 to single out data adapters that are too slow because they impact the user experience.

When you know there is a chance your adapter can take a long time my recommendation is to add logic to avoid the timeout and return either partial results or a code to the UI so it knows what to display.

If you have too many results to process you could add pagination and require the user to request more data.

If you don't have that many objects but it's taking a long time to process you could build a temporary cache and make the UI issue the query again to complete the operation.

In general there are not easy solutions other than making things fast in the first place 🙂

Note: the timeout of 2 minutes can be configured in webclient.properties if you want to see how it works with a different value.

# Data Service timeout in seconds.  An error will be displayed in the UI if a data adapter is too slow to respond

# to a query and the logs will provide more details.

dataservice.timeoutSeconds = 120

[CathyBr]

Hi Laurent,

we are also having problems with requests which take a long time.  Everything worked properly (but slowly) in 5.5, but in V6

things seem to be broken. We are working on speeding up the execution -but in

some cases it is just not possible.  We had this problem in the past as well, so in this one particular (Custom Summary View), we split up

the requests. We first make a request for objects (like vCenters, and some other customer objects), and display them.   Then

we make property requests to find more details about each. - Some of these property requests take long, because we need

info from our 3rd Party WebService, and also ask Hosts about their statuses etc.  (you now how long it can take to get somethings over CIM)

Although we make the Property requests separately for each object, we have found that they are now being grouped into

1 request before it is delivered to the property provider - and then when the data is retrieved and sent back, the one request

is split back into the number of requests which were orginially dispatched.

In this case we often get responses where the error is null, and the result is null (although the provider logged that the data was sent correctly).

In the log, you can see the error

"The property provider took too long...."

I changed the value in the webclient.properties as you suggested, but that only seems to repress the error in the gui.  It does

not have any impact on the fact that we get result=null.

We understand, that it would make sense to redesign this view somehow, but in the meantime is there anything we can do

to prevent this behavior?  Is there any way to make a property request, and say that it should be executed separately?

thanks

Cathy

[laurentsd]

You can use a value of -1 to disable this timeout completely for testing.

If one query takes such a long time because of the back-end service is slow  you could return a temporary result that makes your UI code trigger the query again at a later time while you are caching the result on the java side.

I don't think it's possible to prevent the grouping of multiple requests into one, I'll have to check.

[CathyBr]

Hi Laurent,

we are still really struggling with the Long processing - I have found that this problem happens with many of our

views (which worked in 5.5 - and the past beta versions of 6.0)

Even when we send a single property request - it quite often returns with null result.

I see in the log:

A PropertyProviderAdapter execution took too long: 2943 milliseconds (....)

but I also see in the log, that the request never even reached the provider - because it logs when a request is recieved.  In the gui

I ask the user if he would like to try again, if he confirms I resend the request - and on the second time  it works.

Cathy

[laurentsd]

You need to debug your property provider adapter to see why it is taking so long to respond.  If it doesn't get called something else is going on.

Sorry, I can't help you more without a reproducible test case using an SDK sample for instance, or else if you can send me your plugin Eclipse project ready to compile and run.

[CathyBr]

Hi Laurent,

we will start working on producing the problem using the samples - but I don't know how long that will take.

I tried it again, and when I could reproduce that the provider was not called for the request, I saw in the log:

data-service-pool-757   70000134 100003 200001 c.v.vsphere.client.cm.util.ServiceInfoPropertyValueExtractor     Unknown Service property:  id

could that be relavant to this case - or is that something else?  Should I change the logging level for something, to see more info?

Cathy

[laurentsd]

This warning is not related to your problem. I have seen it in other environments.

You can try to use the logging level to DEBUG in server/configuration/serviceability.xml to see if there is more info:

<logger level="DEBUG" additivity="false" name="com.vmware">

  <appender-ref ref="SIFTED_LOG_FILE" />

  <appender-ref ref="LOG_FILE" />

</logger>

[dscottimw]

Has anyone found a resolution to this issue? I'm running into the same thing.

Thanks

[Peter_Ivanov]

There is a resolution, but it is quite complex.

In brief the idea is to implement a mechanism similar to the tasks:

1) data adapter should return data object like this:

pubic class MyResultWrapper {
   public String requestId;
   public int progress;  // if you can report one
   public MyResult result;  // the real result object; will be null, until progress = 100
}

2) the first time you query for data, the data adapter start asynchronous processing and returns request id immediately

You should use Spring executor/thread pool like org.springframework.scheduling.concurrent.ThreadPoolTaskExecutor and set WaitForTasksToCompleteOnShutdown=false

It is important to make sure that you use small number of threads; to enqueue the requests and to shutdown the threads on server shutdown to prevent leaks.

Note that using thread pools in the plugins is strongly discouraged. It is preferred to move this logic to your own backend server.

3) the UI starts polling over some period of time passing the same request id in the QuerySpec (either in the "name" property of in the "options" map)

4) Once the result is ready, send it using MyResultWrapper#result

[abhishekdubey]

Hi Peter,

I have also faced some similar issue.

Based on your suggestion as per below comment

>>>>>>>>3) the UI starts polling over some period of time passing the same request id in the QuerySpec (either in the "name" property of in the "options" map)

I have one curious question

• How can UI columsetcontainer [Grid View] can poll the information from Adapter .

In our case we followed the chassisRackVSphere-service and chassisRackVSphere-ui samples to develop our plugin.

Can you Please provide some detail how can we poll information on Grid View[Columsetcontainer ]

thanks!!!

[laurentsd]

You cannot initiate your own polling from a grid view using a columnsetcontainer, so your only choice is to only display data that is fast enough to fetch so that your adapter doesn't make the whole grid hang.

[wilber8222]

FYI.

Web client services crash when I changed value of dataservice.timeoutSeconds to -1.