Perfomance degradation with a shared connection/se...

KeyboardSlappin · ‎09-16-2013

Apologies for the cross post from vSphere Management SDK forum, but i think my problem is applicable in the context of this forum as well. Note all the following code is in C#, but as there's no C# fourm so here goes...

I am trying to implement a REST service via a C#, ASP.NET WebAPI application which retrieves information from vSphere via the vSphere Webservice API.

When the WebAPI application is started a connection with the vSphere Webservice is established.

Most of the connection code has been taken from the sample code supplied with the Webservice documentation.

private VSphereWebServiceConnection()

{

AppConfigSettings config = new AppConfigSettings();

Hashtable settings = config.Get(new string[] { "vcIPAddress", "vcUserName", "vcPassword" });

string vcIPAddress = settings["vcIPAddress"] as string;

string vcUserName = settings["vcUserName"] as string;

string vcPassword = settings["vcPassword"] as string;

string url = String.Format("https://{0}/sdk", vcIPAddress);

ServiceReference sRef = new ServiceReference();

sRef.CreateServiceRef("ServiceInstance");

VimService _service = new VimService();

_service.Url = url;

_service.Timeout = 600000;

_service.CookieContainer = new CookieContainer();

// Create a managed object of the root level object

ManagedObjectReference _svcRef = new ManagedObjectReference();

_svcRef.type = "ServiceInstance";

_svcRef.Value = "ServiceInstance";

ServiceContent _sic = _service.RetrieveServiceContent(_svcRef);

VSphereWebServiceSessionManager sm = new VSphereWebServiceSessionManager();

this.vimService = sm.StartSessionWithLogin(_sic, _service, vcUserName, vcPassword);

this.connectionCreationTime = DateTime.Now;

CookieManager cm = new CookieManager();

Cookie c = cm.GetCookieFromSession(this.vimService);

Console.WriteLine("Cookie {0}={1}", c.Name, c.Value);

log.Info("vSphere connection created at " + this.connectionCreationTime);

}

This is coded as a singleton so that whenever i want a VimService object i just perform a static call to VSphereWebServiceConnection.Get which returns me VimService object.

Subsequently a timer will retrieve the current time from the connection in order to keep the connection alive.

ManagedObjectReference _svcRef = new ManagedObjectReference();

_svcRef.type = "ServiceInstance";

_svcRef.Value = "ServiceInstance";

VimService v = GetConnection.VimService;

Console.WriteLine("Polling the vSphere Web Service connection!!! {0}", v.CurrentTime(_svcRef));

Information on VMs within a folder is retrieved by making GET requests to the WebAPI. Multiple connections can be made to the WebAPI and the frequency of the calls are high as polling takes place. Each connection will try to use the same connection established with the vSphere WebService during the start of the application .

When the GET request is received i first retrieve the Datacenter and it's vmFolder property.

public Hashtable Get(VMware.Vim.VimClient vClient, string vSpherefolderName, List<string> vmPropertyPaths)

{

AppConfigSettings config = new AppConfigSettings();

Hashtable settings = config.Get(new string[] { "vcIPAddress", "vcUserName", "vcPassword", "DataCentreName" });

string vcIPAddress = settings["vcIPAddress"] as string;

string vcUserName = settings["vcUserName"] as string;

string vcPassword = settings["vcPassword"] as string;

string dataCentreName = settings["DataCentreName"] as string;

string url = string.Format("https://{0}/sdk", vcIPAddress);

VimService _service = VSphereWebServiceConnection.GetConnection.VimService;

// Create a managed object of the root level object

_svcRef = new ManagedObjectReference();

_svcRef.type = "ServiceInstance";

_svcRef.Value = "ServiceInstance";

// Retrieve the data from the ServiceContent < this will fail in debug mode because of certificate not handled

_sic = _service.RetrieveServiceContent(_svcRef);

_propCol = _sic.propertyCollector;

_rootFolder = _sic.rootFolder;

VSphereWebServiceUtil vSphereWebServiceUtil = new VSphereWebServiceUtil();

// First get the Datacenter MoRef

ManagedObjectReference dataCentreMoref = vSphereWebServiceUtil.GetDatacenter(_service, _svcRef, _sic, _propCol, _rootFolder, dataCentreName);

Console.WriteLine("DatacenterMoref - {0}-{1}", dataCentreMoref.type, dataCentreMoref.Value);

// Then get the customer's VM folder MoRef

ManagedObjectReference vSphereVmFolderMoRef = vSphereWebServiceUtil.GetVSphereCustomerVmFolder(_service, _svcRef, _sic, _propCol, _rootFolder, dataCentreMoref, vSpherefolderName);

Console.WriteLine("vSphere Customer VM Folder MoRef - {0}-{1}", vSphereVmFolderMoRef.type, vSphereVmFolderMoRef.Value);

List<RetrieveResult> retrievedResultsList = vSphereWebServiceUtil.GetResults(_service, _svcRef, _sic, _propCol, _rootFolder, vSphereVmFolderMoRef, vmPropertyPaths);

Hashtable viewTable = vSphereWebServiceUtil.OutputDataset(retrievedResultsList);

return viewTable;

}

public ManagedObjectReference GetDatacenter(VimService _service, ManagedObjectReference _svcRef, ServiceContent _sic, ManagedObjectReference _propCol, ManagedObjectReference _rootFolder, string dataCentreName)

{

this._service = _service;

this._sic = _sic;

this._svcRef = _svcRef;

this._propCol = _propCol;

this._rootFolder = _rootFolder;

// Get references to the ViewManager and PropertyCollector

ManagedObjectReference viewMgrRef = _sic.viewManager;

ManagedObjectReference propColl = _sic.propertyCollector;

List<string> vmList = new List<string>();

vmList.Add("Datacenter");

ManagedObjectReference cViewRef = _service.CreateContainerView(viewMgrRef, _sic.rootFolder, vmList.ToArray(), true);

// Start the spec

ObjectSpec oSpec = new ObjectSpec();

oSpec.obj = cViewRef;

oSpec.skip = true;

TraversalSpec tSpec = new TraversalSpec();

tSpec.name = "traverseEntities";

tSpec.path = "view";

tSpec.skip = false;

tSpec.type = "ContainerView";

List<SelectionSpec> selectionSpecList = new List<SelectionSpec>();

selectionSpecList.Add(tSpec);

oSpec.selectSet = selectionSpecList.ToArray();

PropertySpec pSpec = new PropertySpec();

pSpec.type = "Datacenter";

List<string> pathList = new List<string>();

pathList.Add("name");

pSpec.pathSet = pathList.ToArray();

PropertyFilterSpec fSpec = new PropertyFilterSpec();

List<ObjectSpec> objectSpecList = new List<ObjectSpec>();

objectSpecList.Add(oSpec);

fSpec.objectSet = objectSpecList.ToArray();

List<PropertySpec> propertySpecList = new List<PropertySpec>();

propertySpecList.Add(pSpec);

fSpec.propSet = propertySpecList.ToArray();

List<PropertyFilterSpec> fSpecList = new List<PropertyFilterSpec>();

fSpecList.Add(fSpec);

RetrieveOptions ro = new RetrieveOptions();

RetrieveResult props = _service.RetrievePropertiesEx(propColl, fSpecList.ToArray(), ro);

ManagedObjectReference dataCenterMoref = null;

if (props != null)

{

foreach (ObjectContent oc in props.objects)

{

ManagedObjectReference thisFolder = new ManagedObjectReference() { type = oc.obj.type, Value = oc.obj.Value };

Console.WriteLine(thisFolder);

List<DynamicProperty> dps = oc.propSet.ToList();

if (dps != null)

{

foreach (DynamicProperty dp in dps)

{

string name = (string)dp.val;

string path = dp.name;

if (name == dataCentreName)

{

dataCenterMoref = new ManagedObjectReference();

dataCenterMoref.type = oc.obj.type;

dataCenterMoref.Value = oc.obj.Value;

break;

}

return dataCenterMoref;

}

From there i find the relevant folder

public ManagedObjectReference GetVSphereCustomerVmFolder(VimService _service, ManagedObjectReference _svcRef, ServiceContent _sic, ManagedObjectReference _propCol, ManagedObjectReference _rootFolder, ManagedObjectReference dcMoRef, string vSpherefolderName)

{

this._service = _service;

this._sic = _sic;

this._svcRef = _svcRef;

this._propCol = _propCol;

this._rootFolder = _rootFolder;

// Get references to the ViewManager and PropertyCollector

ManagedObjectReference viewMgrRef = _sic.viewManager;

ManagedObjectReference propColl = _sic.propertyCollector;

List<string> vmList = new List<string>();

vmList.Add("Folder");

// The PropertySpec object specifies what properties

// retrieve from what type of Managed Object

PropertySpec pSpec = new PropertySpec();

pSpec.type = "Folder";

pSpec.pathSet = new string[] { "name" };

SelectionSpec recurseFolders = new SelectionSpec();

recurseFolders.name = "folder2childEntity";

TraversalSpec folder2childEntity = new TraversalSpec();

folder2childEntity.type = "Folder";

folder2childEntity.path = "childEntity";

folder2childEntity.name = recurseFolders.name;

folder2childEntity.selectSet = new SelectionSpec[] { recurseFolders };

// Traverse from a Datacenter through the 'vmFolder' property

TraversalSpec dc2vmFolder = new TraversalSpec();

dc2vmFolder.type = "Datacenter";

dc2vmFolder.path = "vmFolder";

dc2vmFolder.selectSet = new SelectionSpec[] { folder2childEntity };

ObjectSpec oSpec = new ObjectSpec();

oSpec.obj = dcMoRef;

oSpec.skip = true;

oSpec.selectSet = new SelectionSpec[] { dc2vmFolder };

PropertyFilterSpec pfSpec = new PropertyFilterSpec();

pfSpec.propSet = new PropertySpec[] { pSpec };

pfSpec.objectSet = new ObjectSpec[] { oSpec };

RetrieveOptions ro = new RetrieveOptions();

RetrieveResult props = _service.RetrievePropertiesEx(propColl, new PropertyFilterSpec[] { pfSpec }, ro);

ManagedObjectReference vSphereCustomerVmFolderMoRef = null;

if (props != null)

{

foreach (ObjectContent oc in props.objects)

{

ManagedObjectReference thisFolder = new ManagedObjectReference() { type = oc.obj.type, Value = oc.obj.Value };

Console.WriteLine(thisFolder);

List<DynamicProperty> dps = oc.propSet.ToList();

if (dps != null)

{

foreach (DynamicProperty dp in dps)

{

string name = (string)dp.val;

string path = dp.name;

Console.WriteLine("{0} = {1} MOREF = {2}-{3}", path, name, oc.obj.type, oc.obj.Value);

if (name == vSpherefolderName)

{

vSphereCustomerVmFolderMoRef = new ManagedObjectReference();

vSphereCustomerVmFolderMoRef.Value = oc.obj.Value;

vSphereCustomerVmFolderMoRef.type = oc.obj.type;

Console.WriteLine("{0} found : {1}-{2}", vSpherefolderName, oc.obj.type, oc.obj.Value);

break;

}

return vSphereCustomerVmFolderMoRef;

}

and retrieve all the VMs

public List<RetrieveResult> GetResults(VimService _service, ManagedObjectReference _svcRef, ServiceContent _sic, ManagedObjectReference _propCol, ManagedObjectReference _rootFolder, ManagedObjectReference cfolder, List<string> vmPropertyPaths)

{

this._service = _service;

this._sic = _sic;

this._svcRef = _svcRef;

this._propCol = _propCol;

this._rootFolder = _rootFolder;

// Get references to the ViewManager and PropertyCollector

ManagedObjectReference viewMgrRef = _sic.viewManager;

ManagedObjectReference propColl = _sic.propertyCollector;

// use a container view for virtual machines to define the traversal

// - invoke the VimPortType method createContainerView (corresponds

// to the ViewManager method) - pass the ViewManager MOR and

// the other parameters required for the method invocation

// - createContainerView takes a string[] for the type parameter;

// declare an arraylist and add the type string to it

List<string> vmList = new List<string>();

vmList.Add("VirtualMachine");

//ManagedObjectReference cViewRef = _service.CreateContainerView(viewMgrRef, _sic.rootFolder, vmList.ToArray(), true);

//ManagedObjectReference cfolder = new ManagedObjectReference() { type = "Folder", Value = "group-v7599" };

ManagedObjectReference cViewRef = _service.CreateContainerView(viewMgrRef, cfolder, vmList.ToArray(), true);

// create an object spec to define the beginning of the traversal;

// container view is the root object for this traversal

ObjectSpec oSpec = new ObjectSpec();

oSpec.obj = cViewRef;

oSpec.skip = true;

// create a traversal spec to select all objects in the view

TraversalSpec tSpec = new TraversalSpec();

tSpec.name = "traverseEntities";

tSpec.path = "view";

tSpec.skip = false;

tSpec.type = "ContainerView";

// add the traversal spec to the object spec;

// the accessor method (getSelectSet) returns a reference

// to the mapped XML representation of the list; using this

// reference to add the spec will update the list

List<SelectionSpec> selectionSpecList = new List<SelectionSpec>();

selectionSpecList.Add(tSpec);

oSpec.selectSet = selectionSpecList.ToArray();

// specify the property for retrieval (virtual machine name)

PropertySpec pSpec = new PropertySpec();

pSpec.type = "VirtualMachine";

pSpec.pathSet = vmPropertyPaths.ToArray();

// create a PropertyFilterSpec and add the object and

// property specs to it; use the getter method to reference

// the mapped XML representation of the lists and add the specs

// directly to the list

PropertyFilterSpec fSpec = new PropertyFilterSpec();

List<ObjectSpec> objectSpecList = new List<ObjectSpec>();

objectSpecList.Add(oSpec);

fSpec.objectSet = objectSpecList.ToArray();

List<PropertySpec> propertySpecList = new List<PropertySpec>();

propertySpecList.Add(pSpec);

fSpec.propSet = propertySpecList.ToArray();

// Create a list for the filters and add the spec to it

List<PropertyFilterSpec> fSpecList = new List<PropertyFilterSpec>();

fSpecList.Add(fSpec);

// get the data from the server

List<RetrieveResult> retrievedResultsList = new List<RetrieveResult>();

RetrieveOptions ro = new RetrieveOptions();

RetrieveResult props = _service.RetrievePropertiesEx(propColl, fSpecList.ToArray(), ro);

if (props != null)

{

retrievedResultsList.Add(props);

while (props.token != null)

{

props = _service.ContinueRetrievePropertiesEx(propColl, props.token);

retrievedResultsList.Add(props);

}

return retrievedResultsList;

}

i.e. the structure is like this:

Rootfolder -> Datacenter.vmFolder -> Folder1

| VM_1

| VM_2

| VM_3

| ...

|

-> Folder2

| VM_a

| VM_b

| vm_c

| ...

The issue i am seeing as that whilst the application seems to work initially, it gradually slows down and becomes unresponsive and ultimately i start seeing timeouts trying to retrieve the information from vSphere.

So the question here is what is going on? What resource am i running out of?

Looking at the vSphere server i see that vpxd.exe climbs and plateaus around 250-300mb but can climb higher. The java.exe sometimes goes as high as 1GB+ whilst the tomcat server floats around 700mb.

When it gets to this stage the performance starts to degrade badly to the point of being unresponsive and the application is no longer able to retrieve information from vSphere via the Webservice API.

Do i understand the use of the connection and sessions correctly?

As far as i am aware session can essentially be retrieved by using the same cookie that is handed to the application during login and reloading into a new connection OR the session can be maintained by keeping it alive by retrieving the current time from the Webservice (is that correct?)

Is my understanding of a connection correct? i.e. by retrieving the same VimService i am maintaining the single connection OR do i end up establishing a new connection each time i call

ServiceContent _sic = _service.RetrieveServiceContent(_svcRef);

And if so how do i avoid that.

Note i have also tried to call the Destroy methods on various ManagedObject such as the PropertyCollector etc.

Wherever possible i have tried to create instances like ServiceContent, VimServer, RootFolder, PropertyCollector up front and then passed them to methods that require them in the hope that this will help the resource management.

Any ideas?

KeyboardSlappin · ‎09-26-2013

Just a quick bump on this as we've not found a solution to this. Anybody with any ideas?

All

Perfomance degradation with a shared connection/session to WebService and use of PropertyFilter, PropertyCollector