Hello all,
I am on-site working with a customer that has approximately 35,000 objects in a single OU. Unfortunately, I need to present a selection/search criteria to the users to select security groups to add an object (AD:ComputerAD). However, in the AD Browse-Tree there are only 898 elements returned from this OU.
I have invoked the "Configure Active Directory Plug-in Options" workflow, for all configured AD End-Points with the following:
searchSizeLimit | number | 50000.0 | |||
searchSizeLImitPerSerer | number | 100000.0 |
I have updated to the latest AD-Plugin, details below:
vRO:
Version: 7.2.0
Build number: 4629841
Build date: Nov 10, 2016
Database version: 1.77
Plugins:
AD 3.0.6.5491166
AMQP 1.0.4.4512213
Azure 1.0.0.4635872
Cli 2.0.0.3036461
Configurator 7.2.0.4629841
DataManagement 1.0.0.2504108
DynamicTypes 1.2.0.4542195
Enums 7.0.1.4542193
GEF 2.0.0.4278511
Library 7.0.1.4542193
Mail 7.0.1.4542193
NSX 1.0.4.4233593
Net 7.0.1.4542193
PowerShell 1.0.10.4512207
REST 2.2.1.4512209
RPProvisionVCAC 7.2.0-SNAPSHOT.4644246
RPTestJenkins 7.2.0-SNAPSHOT.4644246
SNMP 1.0.3.4538184
SOAP 2.0.0.4552786
SQL 1.1.4.4009493
SSH 7.0.1.4445953
TeamFoundationServer 7.2.0-SNAPSHOT.4644246
VAPI 7.2.0.4542196
VC 6.5.0.4512205
VCACDevopsRPEngine 7.2.0-SNAPSHOT.4644246
VCO 7.2.0.4629841
VRAServer 7.2.0-SNAPSHOT.4644246
Workflow documentation 7.0.1.4542193
XML 7.0.1.4542193
vCAC 7.2.0.4601797
vCACCAFE 7.2.0.4602407
I've set the AD Plugin to debug mode and am seeing the following log entries:
==> /storage/log/vmware/vco/app-server/integration-server.log <==
2017-05-10 10:23:08.280+0000 vco: [component="MSPluginFactory" priority="DEBUG" thread="http-nio-127.0.0.1-8280-exec-2" user="" context="" token="" wfid="" wfname="" anctoken="" wfstack="" instanceid="a7609700-d9ff-4e97-a1d7-c8d27bfb65f0"] Find item. TypeName: OrganizationalUnit
2017-05-10 10:23:08.287+0000 vco: [component="MSPluginFactory" priority="DEBUG" thread="http-nio-127.0.0.1-8280-exec-4" user="" context="" token="" wfid="" wfname="" anctoken="" wfstack="" instanceid="a7609700-d9ff-4e97-a1d7-c8d27bfb65f0"] Find item. TypeName: OrganizationalUnit
2017-05-10 10:23:08.810+0000 vco: [component="Container" priority="WARN" thread="http-nio-127.0.0.1-8280-exec-3" user="" context="" token="" wfid="" wfname="" anctoken="" wfstack="" instanceid="a7609700-d9ff-4e97-a1d7-c8d27bfb65f0"] Error while adding Item to a list in the container [REDACTED_OU_NAME]
2017-05-10 10:23:09.363+0000 vco: [component="Container" priority="WARN" thread="http-nio-127.0.0.1-8280-exec-3" user="" context="" token="" wfid="" wfname="" anctoken="" wfstack="" instanceid="a7609700-d9ff-4e97-a1d7-c8d27bfb65f0"] Error while adding Item to a list in the container [REDACTED_OU_NAME]
2017-05-10 10:23:09.896+0000 vco: [component="Container" priority="WARN" thread="http-nio-127.0.0.1-8280-exec-3" user="" context="" token="" wfid="" wfname="" anctoken="" wfstack="" instanceid="a7609700-d9ff-4e97-a1d7-c8d27bfb65f0"] Error while adding Item to a list in the container [REDACTED_OU_NAME]
2017-05-10 10:23:10.459+0000 vco: [component="Container" priority="WARN" thread="http-nio-127.0.0.1-8280-exec-3" user="" context="" token="" wfid="" wfname="" anctoken="" wfstack="" instanceid="a7609700-d9ff-4e97-a1d7-c8d27bfb65f0"] Error while adding Item to a list in the container [REDACTED_OU_NAME]
2017-05-10 10:23:11.011+0000 vco: [component="Container" priority="WARN" thread="http-nio-127.0.0.1-8280-exec-3" user="" context="" token="" wfid="" wfname="" anctoken="" wfstack="" instanceid="a7609700-d9ff-4e97-a1d7-c8d27bfb65f0"] Error while adding Item to a list in the container [REDACTED_OU_NAME]
If anyone has a work-around or a potential fix for this it would be really welcomed.
Hi,
How did you set AD plug-in in debug mode? Could you also check the content of the file server.log (in the same folder as integration-server.log)?
For each such error "Error while adding Item to a list in the container" in debug log mode there should be two log messages - one short on WARN level (as shown in your log excerpt) and one longer on DEBUG level, which should also log the whole Java exception stack trace. This stack trace should provide additional details on what exactly is the underlying LDAP exception.
There is additional limitation implied directly from Microsoft Active Directory Server.
Which limits results set to 1000 entities per page. Which means that you will not be able to list more than 1000 elements in vRO inventory tree (without modifying Active Directory server settings which is not recommended ).
Since vRO inventory does not apply any filtering when you try to expand the OU containing it will request all child elements for corresponding OU from AD server. But AD server will send only first 1000.
If you really need to work with large result sets (bigger than 1000 elements ) than only option is to use new feature added in (AD plug-in version 3.x) allowing for performing paged queries. You can check "Added support for SimplePagedResultsControl." example here Active Directory plugin 3.0.0 - New and Noteworthy .
AD plugin 3.x adds support for generic LDAP client that can be used for executing arbitrary LDAP queries and recieve paged results.
In this case you can create a custom vRO action accepting required filtering parameters and perform an LDAP query based on those parameters. Result's from this action can be presented to the user for selecting required entity.
2017-05-10 10:23:09.363+0000 vco: [component="Container" priority="WARN" thread="http-nio-127.0.0.1-8280-exec-3" user="" context="" token="" wfid="" wfname="" anctoken="" wfstack="" instanceid="a7609700-d9ff-4e97-a1d7-c8d27bfb65f0"] Error while adding Item to a list in the container [REDACTED_OU_NAME]
Above messages are also interesting. It looks like some of the OU child elements are not present in the inventory since we are not able to create corresponding vRO objects fro them.
Providing the complete server.log file will help to understand the root cause for this. But have in mind that even then you will be able to see maximum of 1000 elements,
I set the plug-in to debug mode via the configuration UI drop down.
The logs entries from the server.log for the same time frame:
server.log |
---|
2017-05-10 10:20:35.398+0000 [http-nio-127.0.0.1-8280-exec-5] DEBUG {} [MSPluginFactory] Create MSPluginFactory for username [REDACTED], sessionId [ee0775c4-bad1-4ddc-810c-f4de1264a63b] 2017-05-10 10:20:35.400+0000 [http-nio-127.0.0.1-8280-exec-5] DEBUG {} [MSPluginFactory] Find item. TypeName: AdHost 2017-05-10 10:20:35.484+0000 [http-nio-127.0.0.1-8280-exec-1] DEBUG {} [MSPluginFactory] Find item. TypeName: AdHost 2017-05-10 10:21:42.419+0000 [http-nio-127.0.0.1-8280-exec-1] WARN {} [ADObjectFactory] Unsupported entry type! Found ldap entry CN=NTDS Quotas,DC=bsci,DC=bossci,DC=com object classes null 2017-05-10 10:21:53.925+0000 [http-nio-127.0.0.1-8280-exec-4] WARN {} [Container] Error while adding Item to a list in the container Staging-DirSync-CRM 2017-05-10 10:21:54.441+0000 [http-nio-127.0.0.1-8280-exec-4] WARN {} [Container] Error while adding Item to a list in the container Staging-DirSync-CRM 2017-05-10 10:21:54.959+0000 [http-nio-127.0.0.1-8280-exec-4] WARN {} [Container] Error while adding Item to a list in the container Staging-DirSync-CRM 2017-05-10 10:21:55.470+0000 [http-nio-127.0.0.1-8280-exec-4] WARN {} [Container] Error while adding Item to a list in the container Staging-DirSync-CRM 2017-05-10 10:21:56.017+0000 [http-nio-127.0.0.1-8280-exec-4] WARN {} [Container] Error while adding Item to a list in the container Staging-DirSync-CRM 2017-05-10 10:21:57.556+0000 [http-nio-127.0.0.1-8280-exec-1] INFO {} [Service_linux] Starting: [/usr/bin/sudo, /etc/init.d/vco-server, status] 2017-05-10 10:21:57.626+0000 [http-nio-127.0.0.1-8280-exec-6] INFO {} [Service_linux] Starting: [/usr/bin/sudo, /etc/init.d/vco-server, status] 2017-05-10 10:22:01.553+0000 [http-nio-127.0.0.1-8280-exec-4] WARN {} [Container] Error while adding Item to a list in the container Staging-DirSync-CRM 2017-05-10 10:23:08.280+0000 [http-nio-127.0.0.1-8280-exec-2] DEBUG {} [MSPluginFactory] Find item. TypeName: OrganizationalUnit 2017-05-10 10:23:08.287+0000 [http-nio-127.0.0.1-8280-exec-4] DEBUG {} [MSPluginFactory] Find item. TypeName: OrganizationalUnit 2017-05-10 10:23:08.810+0000 [http-nio-127.0.0.1-8280-exec-3] WARN {} [Container] Error while adding Item to a list in the container Staging-DirSync-CRM 2017-05-10 10:23:09.363+0000 [http-nio-127.0.0.1-8280-exec-3] WARN {} [Container] Error while adding Item to a list in the container Staging-DirSync-CRM 2017-05-10 10:23:09.896+0000 [http-nio-127.0.0.1-8280-exec-3] WARN {} [Container] Error while adding Item to a list in the container Staging-DirSync-CRM 2017-05-10 10:23:10.459+0000 [http-nio-127.0.0.1-8280-exec-3] WARN {} [Container] Error while adding Item to a list in the container Staging-DirSync-CRM 2017-05-10 10:23:11.011+0000 [http-nio-127.0.0.1-8280-exec-3] WARN {} [Container] Error while adding Item to a list in the container Staging-DirSync-CRM 2017-05-10 10:26:54.929+0000 [tokenLifetimeMonitorScheduler-1] INFO {} [OAuthTokenLifetimeService] Renewing of security tokens activated for 0 tokens expiring between 2017-05-10 10:25:54.928 and 2017-05-10 10:42:54.928. 2017-05-10 10:29:19.165+0000 [http-nio-127.0.0.1-8280-exec-3] DEBUG {} [MSPluginFactory] Find item. TypeName: OrganizationalUnit 2017-05-10 10:29:19.170+0000 [http-nio-127.0.0.1-8280-exec-8] DEBUG {} [MSPluginFactory] Find item. TypeName: OrganizationalUnit 2017-05-10 10:29:19.688+0000 [http-nio-127.0.0.1-8280-exec-9] WARN {} [Container] Error while adding Item to a list in the container Staging-DirSync-CRM 2017-05-10 10:29:20.199+0000 [http-nio-127.0.0.1-8280-exec-9] WARN {} [Container] Error while adding Item to a list in the container Staging-DirSync-CRM 2017-05-10 10:29:20.716+0000 [http-nio-127.0.0.1-8280-exec-9] WARN {} [Container] Error while adding Item to a list in the container Staging-DirSync-CRM 2017-05-10 10:29:21.282+0000 [http-nio-127.0.0.1-8280-exec-9] WARN {} [Container] Error while adding Item to a list in the container Staging-DirSync-CRM 2017-05-10 10:29:21.799+0000 [http-nio-127.0.0.1-8280-exec-9] WARN {} [Container] Error while adding Item to a list in the container Staging-DirSync-CRM |
Thanks very much for the information, I'll work on it in my lab, but will change the form to use a search field for the time being,
Thanks for the information.
Using search list instead is viable option in case you don't need to work with the whole list .
To enable additional logging that will reveal the reason for WARN message you can edint
/etc/vco/app-server/log4j.xml file and replace
<!-- Default root category -->
with
<category additivity="true" name="ch.dunes.ad.object">
<priority value="DEBUG"/>
<appender-ref ref="FILE"/>
</category>
<!-- Default root category -->
I ran into this issue trying to return AD Computers using the Plugin. Using the code you linked to above, here is how I fixed it:
/* getAdComputer.js
This script retrieves a list of AD Computers. It differs from getComputerAd() in that it does not have the 1000 record limit.
REFERENCES
Action - getAdComputer
INPUTS
name (string) - Specifies a regex filter. Only computers with names matching the pattern will be returned. This is required. AD only returns 1000 records by default.
distinguishedName (string) - Specifies a regex filter. Only computers with DNs matching the pattern will be returned.
OUTPUTS
ComputerAD[]
*/
// Get the default AD Host configured in VRO:
var adServer = AD_HostManager.findAllHosts()[0];
var rootBaseDn = adServer.hostConfiguration.ldapBase;
var ldapClient = adServer.getLdapClient();
// Return only name and DN from the LDAP lookup (so we can filter on them):
var propertyList = [
"name",
"distinguishedName"
];
var searchRequest = LdapSearchRequest.createRequest(
rootBaseDn,
"(&(objectCategory=computer)(objectClass=computer))",
LdapSearchScope.SUB,
propertyList,
LdapDereferencePolicy.ALWAYS);
var resumeCookie = null;
var allResults = [];
var iterations = 0;
while ( true && iterations < 1000 )
{
// Don't let the loop go over 1000 iterations to avoid infinite loop.
iterations ++;
pagedSearchControl = new LdapSimplePagedResultsControl(999999, resumeCookie, true);
// Make sure there is no another control registered with same OID
searchRequest.removeControlByOid(pagedSearchControl.getOID());
// add SimplePagedResultsControl to current search control
searchRequest.addControl(pagedSearchControl);
var searchResult = ldapClient.searchBySearchRequest(searchRequest);
var ldapEntries = searchResult.getSearchEntries();
if ( ldapEntries )
{
ldapEntries.map(function (elem) { allResults.push(elem); });
}
var responseControl = LdapSimplePagedResultsControl.get(searchResult);
if ( responseControl.moreResultsToReturn() )
{
resumeCookie = responseControl.getCookieBytes();
}
else
{
break;
}
}
if ( allResults )
{
// Filter by name, if provided:
if ( name )
{
var regx = new RegExp(name,"gi");
allResults = allResults.filter(function (elem) { return elem.getAttributeValue("name").match(regx) } );
}
// Filter by distinguishedName, if provided:
if ( distinguishedName )
{
var regx = new RegExp(distinguishedName,"gi");
allResults = allResults.filter(function (elem) { return elem.getAttributeValue("distinguishedName").match(regx) } );
}
// Convert the LdapEntry results to ComputerAD objects:
var adComputers = allResults.map(function (elem) { return ActiveDirectory.getComputerAD(elem.getAttributeValue("name", null)); });
// Sort the collection by name:
adComputers = adComputers.sort(function (itm1,itm2) { if ( itm1.name.toLowerCase() >= itm2.name.toLowerCase() ) { return 1; } else { return -1; } });
System.debug("Computer objects found: " + adComputers.length);
return adComputers;
}
else
{
throw "No LDAP entries found for AD Computers in " + rootBaseDn + "!";
}
I implemented this as an action that returns Computer objects, but you can modify the code to return other objects as you like.
I realise it's been a while since the original post, but I have hit the same issue recently so did a write up about it... In short, it's an AD limitation, although you can change it using ntdsutil...
https://www.6cd.co.uk/2019/01/issues-with-vro-and-active-directory.html