VMware Cloud Community
marsherian
Enthusiast
Enthusiast

vRO ActiveDirectory Plugin does not return all objects in an orgnizationalUnit

Hello all,

I am on-site working with a customer that has approximately 35,000 objects in a single OU. Unfortunately, I need to present a selection/search criteria to the users to select security groups to add an object (AD:ComputerAD). However, in the AD Browse-Tree there are only 898 elements returned from this OU.

I have invoked the "Configure Active Directory Plug-in Options" workflow, for all configured AD End-Points with the following:

searchSizeLimitnumber50000.0
searchSizeLImitPerSerernumber100000.0

I have updated to the latest AD-Plugin, details below:

vRO:

Version: 7.2.0

Build number: 4629841

Build date: Nov 10, 2016

Database version: 1.77

Plugins:

AD 3.0.6.5491166

AMQP 1.0.4.4512213

Azure 1.0.0.4635872

Cli 2.0.0.3036461

Configurator 7.2.0.4629841

DataManagement 1.0.0.2504108

DynamicTypes 1.2.0.4542195

Enums 7.0.1.4542193

GEF 2.0.0.4278511

Library 7.0.1.4542193

Mail 7.0.1.4542193

NSX 1.0.4.4233593

Net 7.0.1.4542193

PowerShell 1.0.10.4512207

REST 2.2.1.4512209

RPProvisionVCAC 7.2.0-SNAPSHOT.4644246

RPTestJenkins 7.2.0-SNAPSHOT.4644246

SNMP 1.0.3.4538184

SOAP 2.0.0.4552786

SQL 1.1.4.4009493

SSH 7.0.1.4445953

TeamFoundationServer 7.2.0-SNAPSHOT.4644246

VAPI 7.2.0.4542196

VC 6.5.0.4512205

VCACDevopsRPEngine 7.2.0-SNAPSHOT.4644246

VCO 7.2.0.4629841

VRAServer 7.2.0-SNAPSHOT.4644246

Workflow documentation 7.0.1.4542193

XML 7.0.1.4542193

vCAC 7.2.0.4601797

vCACCAFE 7.2.0.4602407

I've set the AD Plugin to debug mode and am seeing the following log entries:

==> /storage/log/vmware/vco/app-server/integration-server.log <==

2017-05-10 10:23:08.280+0000 vco: [component="MSPluginFactory" priority="DEBUG" thread="http-nio-127.0.0.1-8280-exec-2" user="" context="" token="" wfid="" wfname="" anctoken="" wfstack="" instanceid="a7609700-d9ff-4e97-a1d7-c8d27bfb65f0"] Find item. TypeName: OrganizationalUnit

2017-05-10 10:23:08.287+0000 vco: [component="MSPluginFactory" priority="DEBUG" thread="http-nio-127.0.0.1-8280-exec-4" user="" context="" token="" wfid="" wfname="" anctoken="" wfstack="" instanceid="a7609700-d9ff-4e97-a1d7-c8d27bfb65f0"] Find item. TypeName: OrganizationalUnit

2017-05-10 10:23:08.810+0000 vco: [component="Container" priority="WARN" thread="http-nio-127.0.0.1-8280-exec-3" user="" context="" token="" wfid="" wfname="" anctoken="" wfstack="" instanceid="a7609700-d9ff-4e97-a1d7-c8d27bfb65f0"] Error while adding Item to a list in the container [REDACTED_OU_NAME]

2017-05-10 10:23:09.363+0000 vco: [component="Container" priority="WARN" thread="http-nio-127.0.0.1-8280-exec-3" user="" context="" token="" wfid="" wfname="" anctoken="" wfstack="" instanceid="a7609700-d9ff-4e97-a1d7-c8d27bfb65f0"] Error while adding Item to a list in the container [REDACTED_OU_NAME]

2017-05-10 10:23:09.896+0000 vco: [component="Container" priority="WARN" thread="http-nio-127.0.0.1-8280-exec-3" user="" context="" token="" wfid="" wfname="" anctoken="" wfstack="" instanceid="a7609700-d9ff-4e97-a1d7-c8d27bfb65f0"] Error while adding Item to a list in the container [REDACTED_OU_NAME]

2017-05-10 10:23:10.459+0000 vco: [component="Container" priority="WARN" thread="http-nio-127.0.0.1-8280-exec-3" user="" context="" token="" wfid="" wfname="" anctoken="" wfstack="" instanceid="a7609700-d9ff-4e97-a1d7-c8d27bfb65f0"] Error while adding Item to a list in the container [REDACTED_OU_NAME]

2017-05-10 10:23:11.011+0000 vco: [component="Container" priority="WARN" thread="http-nio-127.0.0.1-8280-exec-3" user="" context="" token="" wfid="" wfname="" anctoken="" wfstack="" instanceid="a7609700-d9ff-4e97-a1d7-c8d27bfb65f0"] Error while adding Item to a list in the container [REDACTED_OU_NAME]

If anyone has a work-around or a potential fix for this it would be really welcomed.

8 Replies
iiliev
VMware Employee
VMware Employee

Hi,

How did you set AD plug-in in debug mode? Could you also check the content of the file server.log (in the same folder as integration-server.log)?

For each such error "Error while adding Item to a list in the container" in debug log mode there should be two log messages - one short on WARN level (as shown in your log excerpt) and one longer on DEBUG level, which should also log the whole Java exception stack trace. This stack trace should provide additional details on what exactly is the underlying LDAP exception.

Reply
0 Kudos
igaydajiev
VMware Employee
VMware Employee

There is additional limitation implied directly from Microsoft Active Directory Server.

Which limits results set to 1000 entities per page. Which means that you will not be able to list more than 1000 elements in vRO inventory tree (without modifying Active Directory server settings which is not recommended ).

Since vRO inventory does not apply any filtering when you try to expand the OU containing it will request all child elements for corresponding OU from AD server. But AD server will send only first 1000.

If you really need to work with large result sets (bigger than 1000 elements ) than only option is to use new feature added in (AD plug-in version 3.x) allowing for performing paged queries. You can check "Added support for SimplePagedResultsControl." example here  Active Directory plugin 3.0.0 - New and Noteworthy .

AD plugin 3.x adds support for generic LDAP client that can be used for executing arbitrary LDAP queries and recieve paged results.

In this case you can create a custom vRO action accepting required filtering parameters and perform an LDAP query based on those parameters. Result's  from this action can be presented to the user for selecting required entity.

igaydajiev
VMware Employee
VMware Employee

2017-05-10 10:23:09.363+0000 vco: [component="Container" priority="WARN" thread="http-nio-127.0.0.1-8280-exec-3" user="" context="" token="" wfid="" wfname="" anctoken="" wfstack="" instanceid="a7609700-d9ff-4e97-a1d7-c8d27bfb65f0"] Error while adding Item to a list in the container [REDACTED_OU_NAME]

Above messages are also interesting. It looks like some of the OU child elements are not present in the inventory since we are not able to create corresponding vRO objects fro them.

Providing the complete server.log file will help to understand the root cause for this. But have in mind that even then you will be able to see maximum of 1000 elements,

Reply
0 Kudos
marsherian
Enthusiast
Enthusiast

I set the plug-in to debug mode via the configuration UI drop down.

The logs entries from the server.log for the same time frame:

server.log

2017-05-10 10:20:35.398+0000 [http-nio-127.0.0.1-8280-exec-5] DEBUG {} [MSPluginFactory] Create MSPluginFactory for username [REDACTED], sessionId [ee0775c4-bad1-4ddc-810c-f4de1264a63b]

2017-05-10 10:20:35.400+0000 [http-nio-127.0.0.1-8280-exec-5] DEBUG {} [MSPluginFactory] Find item. TypeName: AdHost

2017-05-10 10:20:35.484+0000 [http-nio-127.0.0.1-8280-exec-1] DEBUG {} [MSPluginFactory] Find item. TypeName: AdHost

2017-05-10 10:21:42.419+0000 [http-nio-127.0.0.1-8280-exec-1] WARN  {} [ADObjectFactory] Unsupported entry type! Found ldap entry CN=NTDS Quotas,DC=bsci,DC=bossci,DC=com object classes null

2017-05-10 10:21:53.925+0000 [http-nio-127.0.0.1-8280-exec-4] WARN  {} [Container] Error while adding Item to a list in the container Staging-DirSync-CRM

2017-05-10 10:21:54.441+0000 [http-nio-127.0.0.1-8280-exec-4] WARN  {} [Container] Error while adding Item to a list in the container Staging-DirSync-CRM

2017-05-10 10:21:54.959+0000 [http-nio-127.0.0.1-8280-exec-4] WARN  {} [Container] Error while adding Item to a list in the container Staging-DirSync-CRM

2017-05-10 10:21:55.470+0000 [http-nio-127.0.0.1-8280-exec-4] WARN  {} [Container] Error while adding Item to a list in the container Staging-DirSync-CRM

2017-05-10 10:21:56.017+0000 [http-nio-127.0.0.1-8280-exec-4] WARN  {} [Container] Error while adding Item to a list in the container Staging-DirSync-CRM

2017-05-10 10:21:57.556+0000 [http-nio-127.0.0.1-8280-exec-1] INFO  {} [Service_linux] Starting: [/usr/bin/sudo, /etc/init.d/vco-server, status]

2017-05-10 10:21:57.626+0000 [http-nio-127.0.0.1-8280-exec-6] INFO  {} [Service_linux] Starting: [/usr/bin/sudo, /etc/init.d/vco-server, status]

2017-05-10 10:22:01.553+0000 [http-nio-127.0.0.1-8280-exec-4] WARN  {} [Container] Error while adding Item to a list in the container Staging-DirSync-CRM

2017-05-10 10:23:08.280+0000 [http-nio-127.0.0.1-8280-exec-2] DEBUG {} [MSPluginFactory] Find item. TypeName: OrganizationalUnit

2017-05-10 10:23:08.287+0000 [http-nio-127.0.0.1-8280-exec-4] DEBUG {} [MSPluginFactory] Find item. TypeName: OrganizationalUnit

2017-05-10 10:23:08.810+0000 [http-nio-127.0.0.1-8280-exec-3] WARN  {} [Container] Error while adding Item to a list in the container Staging-DirSync-CRM

2017-05-10 10:23:09.363+0000 [http-nio-127.0.0.1-8280-exec-3] WARN  {} [Container] Error while adding Item to a list in the container Staging-DirSync-CRM

2017-05-10 10:23:09.896+0000 [http-nio-127.0.0.1-8280-exec-3] WARN  {} [Container] Error while adding Item to a list in the container Staging-DirSync-CRM

2017-05-10 10:23:10.459+0000 [http-nio-127.0.0.1-8280-exec-3] WARN  {} [Container] Error while adding Item to a list in the container Staging-DirSync-CRM

2017-05-10 10:23:11.011+0000 [http-nio-127.0.0.1-8280-exec-3] WARN  {} [Container] Error while adding Item to a list in the container Staging-DirSync-CRM

2017-05-10 10:26:54.929+0000 [tokenLifetimeMonitorScheduler-1] INFO  {} [OAuthTokenLifetimeService] Renewing of security tokens activated for 0 tokens expiring between 2017-05-10 10:25:54.928 and 2017-05-10 10:42:54.928.

2017-05-10 10:29:19.165+0000 [http-nio-127.0.0.1-8280-exec-3] DEBUG {} [MSPluginFactory] Find item. TypeName: OrganizationalUnit

2017-05-10 10:29:19.170+0000 [http-nio-127.0.0.1-8280-exec-8] DEBUG {} [MSPluginFactory] Find item. TypeName: OrganizationalUnit

2017-05-10 10:29:19.688+0000 [http-nio-127.0.0.1-8280-exec-9] WARN  {} [Container] Error while adding Item to a list in the container Staging-DirSync-CRM

2017-05-10 10:29:20.199+0000 [http-nio-127.0.0.1-8280-exec-9] WARN  {} [Container] Error while adding Item to a list in the container Staging-DirSync-CRM

2017-05-10 10:29:20.716+0000 [http-nio-127.0.0.1-8280-exec-9] WARN  {} [Container] Error while adding Item to a list in the container Staging-DirSync-CRM

2017-05-10 10:29:21.282+0000 [http-nio-127.0.0.1-8280-exec-9] WARN  {} [Container] Error while adding Item to a list in the container Staging-DirSync-CRM

2017-05-10 10:29:21.799+0000 [http-nio-127.0.0.1-8280-exec-9] WARN  {} [Container] Error while adding Item to a list in the container Staging-DirSync-CRM

Reply
0 Kudos
marsherian
Enthusiast
Enthusiast

Thanks very much for the information, I'll work on it in my lab, but will change the form to use a search field for the time being,

Reply
0 Kudos
igaydajiev
VMware Employee
VMware Employee

Thanks for the information.

Using search list instead is viable option in case you don't need to work with the whole list .

To enable additional logging that will reveal the reason for WARN message you can edint

/etc/vco/app-server/log4j.xml file and replace

    <!-- Default root category -->

with

  <category additivity="true" name="ch.dunes.ad.object">

        <priority value="DEBUG"/>

        <appender-ref ref="FILE"/>

    </category>

    <!-- Default root category -->

Reply
0 Kudos
bsti
Enthusiast
Enthusiast

I ran into this issue trying to return AD Computers using the Plugin.  Using the code you linked to above, here is how I fixed it:

/*  getAdComputer.js

This script retrieves a list of AD Computers.  It differs from getComputerAd() in that it does not have the 1000 record limit.

REFERENCES

  Action - getAdComputer

INPUTS

  name              (string)   - Specifies a regex filter.  Only computers with names matching the pattern will be returned. This is required.  AD only returns 1000 records by default.

  distinguishedName (string)   - Specifies a regex filter.  Only computers with DNs matching the pattern will be returned.

OUTPUTS

  ComputerAD[]

 

*/

//  Get the default AD Host configured in VRO:

var adServer = AD_HostManager.findAllHosts()[0];

var rootBaseDn = adServer.hostConfiguration.ldapBase;

var ldapClient = adServer.getLdapClient();

//  Return only name and DN from the LDAP lookup (so we can filter on them):

var propertyList = [

  "name",

  "distinguishedName"

];

var searchRequest = LdapSearchRequest.createRequest( 

  rootBaseDn,

  "(&(objectCategory=computer)(objectClass=computer))", 

  LdapSearchScope.SUB, 

  propertyList,

  LdapDereferencePolicy.ALWAYS); 

 

var resumeCookie = null;

var allResults = [];

var iterations = 0;

while ( true && iterations < 1000 )

{

  //  Don't let the loop go over 1000 iterations to avoid infinite loop.

  iterations ++;

  pagedSearchControl = new LdapSimplePagedResultsControl(999999, resumeCookie, true);

  // Make sure there is no another control registered with same OID 

  searchRequest.removeControlByOid(pagedSearchControl.getOID()); 

  // add SimplePagedResultsControl to current search control 

  searchRequest.addControl(pagedSearchControl);

  var searchResult = ldapClient.searchBySearchRequest(searchRequest); 

  var ldapEntries = searchResult.getSearchEntries();

  if ( ldapEntries )

  {

    ldapEntries.map(function (elem) { allResults.push(elem); });

  }

  var responseControl = LdapSimplePagedResultsControl.get(searchResult);

  if ( responseControl.moreResultsToReturn() )

  {

    resumeCookie = responseControl.getCookieBytes();

  }

  else

  {

    break;

  }

}

if ( allResults )

{

  //  Filter by name, if provided:

  if ( name )

  {

    var regx = new RegExp(name,"gi");

    allResults = allResults.filter(function (elem) { return elem.getAttributeValue("name").match(regx) } );

  }

  //  Filter by distinguishedName, if provided:

  if ( distinguishedName )

  {

    var regx = new RegExp(distinguishedName,"gi");

    allResults = allResults.filter(function (elem) { return elem.getAttributeValue("distinguishedName").match(regx) } );

  }

  //  Convert the LdapEntry results to ComputerAD objects:

  var adComputers = allResults.map(function (elem) { return ActiveDirectory.getComputerAD(elem.getAttributeValue("name", null)); });

  //  Sort the collection by name:

  adComputers = adComputers.sort(function (itm1,itm2) { if ( itm1.name.toLowerCase() >= itm2.name.toLowerCase() ) { return 1; } else { return -1; } });

   

  System.debug("Computer objects found: " + adComputers.length);

  return adComputers;

}

else

{

  throw "No LDAP entries found for AD Computers in " + rootBaseDn + "!";

}

I implemented this as an action that returns Computer objects, but you can modify the code to return other objects as you like.

samwalker1380
VMware Employee
VMware Employee

I realise it's been a while since the original post, but I have hit the same issue recently so did a write up about it... In short, it's an AD limitation, although you can change it using ntdsutil...

https://www.6cd.co.uk/2019/01/issues-with-vro-and-active-directory.html