Unable to retrieving more than 100 VMs using PropertyCollector and WaitForUpdatesEx method

[KeyboardSlappin]

Using vSphere SDK for .NET. Sorry, couldn't find a .NET forum for the vSphere SDK but the Java forum should be close enough.

I am having problems retrieving a list of VMs from a Resource Pool. Specifically if there are more than 100 VMs (104 in my case) in the Folder or Resource Pool the returned UpdateSet will only ever have 100 items in it.

The offending code is as follows:

UpdateSet uSet = pc.WaitForUpdatesEx(string.Empty, new WaitOptions() { MaxWaitSeconds = 0 });
Console.Writeline(uSet.FilterSet[0].ObjectSet.Count); // will always contain no more than 100 items

There seems to be a hard cut off of 100 items.
WaitForUpdatesEx's WaitOptions contains a MaxObjectUpdates property, from the documentation

"The maximum number of ObjectUpdate entries that should be returned in a single result from WaitForUpdatesEx. See truncated

An unset value indicates that there is no maximum. In this case PropertyCollector policy may still limit the number of objects that appear in an UpdateSet."

But what is the "PropertyCollector Policy"  that it is referring to? I've been trawling the web and can find absolutely no reference to this at all other than in the documentation.

I also tried just putting in a a MaxObjectUpdates = 150 just to see what would happen. Again only 100 items returned.

For reference here is a more complete section of code:

EntityViewBase folderMngObj = GetVsphereVmFolderManagedObjectRefId(vClient, vSpherefolderName);

if (folderMngObj != null)
     {
          ManagedObjectReference svcRef = new ManagedObjectReference() { Type = "ServiceInstance", Value = "ServiceInstance" };
          ServiceInstance srvInst = new ServiceInstance(vClient, svcRef);

          ServiceContent sContent = srvInst.RetrieveServiceContent();
          ViewManager viewManager = new ViewManager(vClient, sContent.ViewManager);
          PropertyCollector pc = new PropertyCollector(vClient, sContent.PropertyCollector);

          ManagedObjectReference cvMngObjRef = viewManager.CreateContainerView(folderMngObj.MoRef, new string[] { "VirtualMachine" }, false);
          ContainerView cv = new ContainerView(vClient, cvMngObjRef);
          List<PropertySpec> propertySpecList = new List<PropertySpec>();

          foreach (string propertyPath in vmPropertyPaths)
          {
               PropertySpec propSpec = new PropertySpec();
               propSpec.PathSet = new string[] { propertyPath };
               propSpec.Type = "VirtualMachine";
               propertySpecList.Add(propSpec);
          }

          PropertySpec[] propertySpecs = propertySpecList.ToArray();
          PropertyFilterSpec pfs = new PropertyFilterSpec();
          pfs.ObjectSet = new ObjectSpec[] { CreateObjSpec(cv) };
          pfs.PropSet = propertySpecs;

          // Create a Property with partialUpdate is true
          PropertyFilter pf = new PropertyFilter(vClient, pc.CreateFilter(pfs, true));

          // Wait for initial udpate with empty version string
          UpdateSet uSet = pc.WaitForUpdatesEx(string.Empty, new WaitOptions() { MaxWaitSeconds = 0 });

          // Here uSet.FilterSet[0].ObjectSet will always contain no more than 100 items

          viewTable = OutputDataset(uSet);
          pf.DestroyPropertyFilter();
}

Any help would be appreciated.

[BenN · Accepted solution]

What is the value of the 'truncated' flag in the returned UpdateSet?

     http://www.vmware.com/support/developer/vc-sdk/visdk41pubs/ApiReference/vmodl.query.PropertyCollecto...

If true, you must call back into waitForUpdatesEx (with the version from the UpdateSet) to get the rest of the updates.

[stumpr]

Exactly, no guarantee it isn't limited by the server.

The maximum number of ObjectUpdate entries that should be returned in a single result from WaitForUpdatesEx. See truncated An unset value indicates that there is no maximum. In this case PropertyCollector policy may still limit the number of objects that appear in an UpdateSet. A positive value causes WaitForUpdatesEx to suspend the update calculation when the total count of ObjectUpdate entries ready to return reaches the specified maximum.PropertyCollector policy may still limit the total count to something less thanmaxObjectUpdates. A value less than or equal to 0 is illegal.

Check truncated for WaitForUpdatesEx or token for RetrievePropertiesEx/ContinueRetrievePropertiesEx.

[KeyboardSlappin]

Yup, sure enough Truncated was set to true. Thanks for the help. Your answer was marked as correct.

[atoka]

I am also facing the same issue on vSphere SDK. Even after making maxObjects in RetrieveOptions to value more than 100 for RetrievePropertiesEx call, I still get only 100 values.

I am using RetrievePropertiesEx call to get the list of VMs and its properties. How do i set the Truncated flag to false. I was not able to find this filed for any of the parameter to RetrievePropertiesEx call ?

Thanks.

[stumpr]

You could try using the deprecated method RetrieveProperties() instead of RetrievePropertiesEx().  RetrieveProperties doesn't use the truncated option, but of course, as it's deprecated it might not follow through in the future.  But that will do it in one return vs chunks.  Bear in mind a large filter can be pretty significant in larger environments, so there are practical reasons to using truncated updates.

[atoka]

I do want to truncate but I want to specify custom maxObjects instead of 100. I am testing a setup with 3000 VMs and instead of making 30 network calls(as value count is 100), I want to set maxObjects to some value like 500 and reduce the network calls.

Why is setting maxObjects parameter to a value bigger than 100 not working. Note that if I set the maxObjects parameter value below 100, it works fine and returns specified value count but if I specify more than 100, it just returns only 100.

Thanks.

[stumpr]

The API has a max limit as well, 100.  You can go lower, but not higher I believe.

[atoka]

Thanks for quick response.

Is max limit 100! I didn't know about it. If possible can you point me to VMware doc which mentions this limitation? I am not able to find it in any of VMware's vSphere SDK doc. Thanks.

So, then in my case with 3k VMs what would be the best approach to get the list of all VMs and its properties. If i use RetrieveProperties the call hanging forever and with RetrievePropertiesEx call, the max limit is 100 which will increase my network calls.

Thanks.

[stumpr]

Not sure it's documented in the API docs clearly, just know from testing.  If I get some cycles, I'll see if there is a VPX configuration value to change it in the vCenter server.