vcloud vsphere inventory discovery task in progress for 2 hours 30 minutes and counting

[glanzmann]

Hello, we started a inventory discovery task for a vcenter. This task finished once according to the log file before. The first run took 20 minutes. Today I started the discovery task again but thought it took to long so I restarted the agent. That resulted in the discovery task never finishing. According to ManagerService.Exe.config configuration file inventory discovery timeout should be 2 hours. So I waited two hours but the discovery task is still 'in progress'. I assume that I caused this problem, by acting to quickly instead of waiting. However we restarted the whole system multiple times, tried to change the inventory discovery timeout and also moved away the state Workitems.bin file in tmp directory of the agent. Please advice how to recover from the problem. Cheers, Thomas

[glanzmann · Accepted solution]

Hello, after leaving the system alone for two hours, the system recovered. My analysis is: I should have not interfered with the data collection and should have waited. Why it takes 22 mintues to collect the data is not clear however after letting system sitting there for two hours it recovered and started the data collection. Now everything is up and running again. Cheers,     Thomas

[admin]

for what it's worth, in most cases where I've seen performance issues they've been network related between the proxy and the host.  There can be several causes (some can be related to tuning in the vCAC collection code), but in one case I saw that DNS was slow to resolve @ around 11 seconds which works fine for vSphere client but the API for whatever reason it didn't cache the result, we worked with network engineers and put in a record in the hosts file as a temporary workaround until it could be permanently resolved.  So I'd make sure connection to and from the vSphere endpoint is optimal.

[glanzmann]

Hallo, so we still have the problem that the inventory collection takes 30 minutes while it should take no more than 2 minutes. The inventory consists of one Cluster, two ESX servers and 20 VMs. We did a man in the middle attack between the vCenter and the agent and looked at the traffic. What we can see that the inventory collection consts of one long TCP connection, the agent issues soap calls and the calls are immediatly ansswer, but sometimes the agent waits upto _1 minute_ to issue the next soap call. I assume that this is the symptom. Also when I compare the throughput flow graph of a installation that works with the one that does not, I can see that the one that does not work bursts and than sleeps while the one that works has a constant flow of throughput. We tried to reinstall the vcac server with the same image as the environment where it works, but still have the problem. We also added the hosts entries to the vcac server, no change. We tried to use sysinternals process explorer to monitor the agent, but the only thing we can see here that the delta between writing to the .config file and thread exit is 6 seconds instead of a fraction of a second. So next we try to install vcac without joining the active directory and reinstall a new vcenter and see if the poblem persists. The debug log shows nothing because there is only logged when the inventory location starts, finishes and is propagated to the vcac server. Cheers,    Thomas

We opened a SR: 13330063105