Large number of idle and active sessions

[cafesito]

Hi,

I have a C# application that uses the vSphere SDK to communicate with various virtual centers (VCs). This application periodically queries the virtual centers and populates a local cache. Most of the time, the application works perfectly well, but every once in a while (like every 2 or 3 months), we get a huge number of idle and active sessions in the VC (please see attached image), and don't quite know what triggers this. Whenever we get this spike in the number of idle or active sessions, I see that the application attempted to connect to the VC only once - yet somehow, a large number of idle and active sessions are recorded in the victimized VC.

This is the code we're using to connect to the VCs:

var readonlyClient = new SDK.VimClient();
readonlyClient.Connect(GetVimApiUrl(_vmmserver));
readonlyClient.Login(_readonlyUser, _readonlyPassword);

We're using the code below to re-connect if sessions are not active. This if statement gets called right before we attempt to make a call out to the VC.

if (!HasActiveSession(_readonlyClient))
{
     try
     {
          _readonlyClient.Logout();
          _readonlyClient.Disconnect();
          logger.DebugFormat("Readonly session is not active for {0} and we have disconnected", _vmmserver);
     }
     catch (Exception e)
     {
          logger.ErrorFormat("Readonly session is not active for {0} and there was an error when trying to disconnect. Error details: {1}", _vmmserver, e);
     }

     Stopwatch sw = Stopwatch.StartNew();
     _readonlyClient = null; // explicitly release resources
     _readonlyClient = new SDK.VimClient(); // re-initialize to try to get around this exception: VMware.Vim.VimException: The session is not authenticated. ---> System.Web.Services.Protocols.SoapException: The session is not authenticated.
     _readonlyClient.Connect(GetVimApiUrl(_vmmserver));
     _readonlyClient.Login(_readonlyUser, _readonlyPassword);
     logger.DebugFormat("Reconnected Readonly Vim Client in {0} ms", sw.ElapsedMilliseconds);
}

I've noticed that whenever we see the spike in idle and active sessions, the statement "Reconnected Readonly Vim Client in X ms" appears exactly once for the victimized VC.

Initially, the number of idle and active sessions may be around 120 or more. Then it may dwindle to 80 Active, 30 Idle, then 50 Active, 10 Idle, then spike up again (as seen in the attached image) to over 175 or more. Eventually the problem gets so bad, that we need to disable the account used by the C# application from connecting to the VC.

Searching the internet for an answer to this, I ran into this article: http://communities.vmware.com/thread/148275, but this is for ESX 3.5, and we're using version 4.1.

Any insight as to what could be causing this spike in idle and active sessions?

Thanks,

Juan

[cafesito]

Hi,

Here is more info that may help bottom out on this issue:

Subsequent calls to the if statement (to check if connections are active) are made to retrieve info from the VC. Those calls show this exception printed in the logs:

2012-03-02 11:17:58,397 ERROR VMWareSDKHelper: Readonly session is not active for mytestvc.domain.com and there was an error when trying to disconnect. Error details: System.InvalidOperationException: ServiceContent is not initialized.
   at VMware.Vim.VimClient.Logout()

and points to the Logout() line in the try block:

try
{
     _readonlyClient.Logout();
     _readonlyClient.Disconnect();
     logger.DebugFormat("Readonly session is not active for {0} and we have disconnected", _vmmserver);
}

However, I would assume that the code that immediately follows it would re-initialize the ServiceContent:

Stopwatch sw = Stopwatch.StartNew();
_readonlyClient = null; // explicitly release resources
_readonlyClient = new SDK.VimClient(); // re-initialize to try to get  around this exception: VMware.Vim.VimException: The session is not  authenticated. ---> System.Web.Services.Protocols.SoapException: The  session is not authenticated.
_readonlyClient.Connect(GetVimApiUrl(_vmmserver));
_readonlyClient.Login(_readonlyUser, _readonlyPassword);
logger.DebugFormat("Reconnected Readonly Vim Client in {0} ms", sw.ElapsedMilliseconds);

But since ServiceContent is not initialized, other subsequent calls try to LogOut() and re-connect. However the call to LogOut() throws the exception above, and the re-connect part does not re-initialize the ServiceContent. I think its all of these attempts to re-connect that cause the spikes in idle and active sessions.

Is there another way to re-initialize the ServiceContent?

Any guidance is appreciated.

Thanks,

Juan