8 Replies Latest reply on Jan 25, 2019 10:41 AM by samwalker1380

    vRO ActiveDirectory Plugin does not return all objects in an orgnizationalUnit

    marsherian Novice

      Hello all,

       

      I am on-site working with a customer that has approximately 35,000 objects in a single OU. Unfortunately, I need to present a selection/search criteria to the users to select security groups to add an object (AD:ComputerAD). However, in the AD Browse-Tree there are only 898 elements returned from this OU.

      I have invoked the "Configure Active Directory Plug-in Options" workflow, for all configured AD End-Points with the following:

       

      searchSizeLimitnumber50000.0
      searchSizeLImitPerSerernumber100000.0

       

      I have updated to the latest AD-Plugin, details below:

       

       

      vRO:

      Version: 7.2.0

      Build number: 4629841

      Build date: Nov 10, 2016

      Database version: 1.77

       

      Plugins:

      AD 3.0.6.5491166

      AMQP 1.0.4.4512213

      Azure 1.0.0.4635872

      Cli 2.0.0.3036461

      Configurator 7.2.0.4629841

      DataManagement 1.0.0.2504108

      DynamicTypes 1.2.0.4542195

      Enums 7.0.1.4542193

      GEF 2.0.0.4278511

      Library 7.0.1.4542193

      Mail 7.0.1.4542193

      NSX 1.0.4.4233593

      Net 7.0.1.4542193

      PowerShell 1.0.10.4512207

      REST 2.2.1.4512209

      RPProvisionVCAC 7.2.0-SNAPSHOT.4644246

      RPTestJenkins 7.2.0-SNAPSHOT.4644246

      SNMP 1.0.3.4538184

      SOAP 2.0.0.4552786

      SQL 1.1.4.4009493

      SSH 7.0.1.4445953

      TeamFoundationServer 7.2.0-SNAPSHOT.4644246

      VAPI 7.2.0.4542196

      VC 6.5.0.4512205

      VCACDevopsRPEngine 7.2.0-SNAPSHOT.4644246

      VCO 7.2.0.4629841

      VRAServer 7.2.0-SNAPSHOT.4644246

      Workflow documentation 7.0.1.4542193

      XML 7.0.1.4542193

      vCAC 7.2.0.4601797

      vCACCAFE 7.2.0.4602407

       

      I've set the AD Plugin to debug mode and am seeing the following log entries:

      ==> /storage/log/vmware/vco/app-server/integration-server.log <==

      2017-05-10 10:23:08.280+0000 vco: [component="MSPluginFactory" priority="DEBUG" thread="http-nio-127.0.0.1-8280-exec-2" user="" context="" token="" wfid="" wfname="" anctoken="" wfstack="" instanceid="a7609700-d9ff-4e97-a1d7-c8d27bfb65f0"] Find item. TypeName: OrganizationalUnit

      2017-05-10 10:23:08.287+0000 vco: [component="MSPluginFactory" priority="DEBUG" thread="http-nio-127.0.0.1-8280-exec-4" user="" context="" token="" wfid="" wfname="" anctoken="" wfstack="" instanceid="a7609700-d9ff-4e97-a1d7-c8d27bfb65f0"] Find item. TypeName: OrganizationalUnit

      2017-05-10 10:23:08.810+0000 vco: [component="Container" priority="WARN" thread="http-nio-127.0.0.1-8280-exec-3" user="" context="" token="" wfid="" wfname="" anctoken="" wfstack="" instanceid="a7609700-d9ff-4e97-a1d7-c8d27bfb65f0"] Error while adding Item to a list in the container [REDACTED_OU_NAME]

      2017-05-10 10:23:09.363+0000 vco: [component="Container" priority="WARN" thread="http-nio-127.0.0.1-8280-exec-3" user="" context="" token="" wfid="" wfname="" anctoken="" wfstack="" instanceid="a7609700-d9ff-4e97-a1d7-c8d27bfb65f0"] Error while adding Item to a list in the container [REDACTED_OU_NAME]

      2017-05-10 10:23:09.896+0000 vco: [component="Container" priority="WARN" thread="http-nio-127.0.0.1-8280-exec-3" user="" context="" token="" wfid="" wfname="" anctoken="" wfstack="" instanceid="a7609700-d9ff-4e97-a1d7-c8d27bfb65f0"] Error while adding Item to a list in the container [REDACTED_OU_NAME]

      2017-05-10 10:23:10.459+0000 vco: [component="Container" priority="WARN" thread="http-nio-127.0.0.1-8280-exec-3" user="" context="" token="" wfid="" wfname="" anctoken="" wfstack="" instanceid="a7609700-d9ff-4e97-a1d7-c8d27bfb65f0"] Error while adding Item to a list in the container [REDACTED_OU_NAME]

      2017-05-10 10:23:11.011+0000 vco: [component="Container" priority="WARN" thread="http-nio-127.0.0.1-8280-exec-3" user="" context="" token="" wfid="" wfname="" anctoken="" wfstack="" instanceid="a7609700-d9ff-4e97-a1d7-c8d27bfb65f0"] Error while adding Item to a list in the container [REDACTED_OU_NAME]

       

      If anyone has a work-around or a potential fix for this it would be really welcomed.

        • 1. Re: vRO ActiveDirectory Plugin does not return all objects in an orgnizationalUnit
          iiliev Champion
          Community WarriorsVMware Employees

          Hi,

           

          How did you set AD plug-in in debug mode? Could you also check the content of the file server.log (in the same folder as integration-server.log)?

           

          For each such error "Error while adding Item to a list in the container" in debug log mode there should be two log messages - one short on WARN level (as shown in your log excerpt) and one longer on DEBUG level, which should also log the whole Java exception stack trace. This stack trace should provide additional details on what exactly is the underlying LDAP exception.

          • 2. Re: vRO ActiveDirectory Plugin does not return all objects in an orgnizationalUnit
            igaydajiev Expert
            VMware Employees

            There is additional limitation implied directly from Microsoft Active Directory Server.

            Which limits results set to 1000 entities per page. Which means that you will not be able to list more than 1000 elements in vRO inventory tree (without modifying Active Directory server settings which is not recommended ).

            Since vRO inventory does not apply any filtering when you try to expand the OU containing it will request all child elements for corresponding OU from AD server. But AD server will send only first 1000.

             

            If you really need to work with large result sets (bigger than 1000 elements ) than only option is to use new feature added in (AD plug-in version 3.x) allowing for performing paged queries. You can check "Added support for SimplePagedResultsControl." example here  Active Directory plugin 3.0.0 - New and Noteworthy .

            AD plugin 3.x adds support for generic LDAP client that can be used for executing arbitrary LDAP queries and recieve paged results.

             

            In this case you can create a custom vRO action accepting required filtering parameters and perform an LDAP query based on those parameters. Result's  from this action can be presented to the user for selecting required entity.

            • 3. Re: vRO ActiveDirectory Plugin does not return all objects in an orgnizationalUnit
              igaydajiev Expert
              VMware Employees

              2017-05-10 10:23:09.363+0000 vco: [component="Container" priority="WARN" thread="http-nio-127.0.0.1-8280-exec-3" user="" context="" token="" wfid="" wfname="" anctoken="" wfstack="" instanceid="a7609700-d9ff-4e97-a1d7-c8d27bfb65f0"] Error while adding Item to a list in the container [REDACTED_OU_NAME]

               

              Above messages are also interesting. It looks like some of the OU child elements are not present in the inventory since we are not able to create corresponding vRO objects fro them.

              Providing the complete server.log file will help to understand the root cause for this. But have in mind that even then you will be able to see maximum of 1000 elements,

              • 4. Re: vRO ActiveDirectory Plugin does not return all objects in an orgnizationalUnit
                marsherian Novice

                I set the plug-in to debug mode via the configuration UI drop down.

                 

                The logs entries from the server.log for the same time frame:

                server.log

                2017-05-10 10:20:35.398+0000 [http-nio-127.0.0.1-8280-exec-5] DEBUG {} [MSPluginFactory] Create MSPluginFactory for username [REDACTED], sessionId [ee0775c4-bad1-4ddc-810c-f4de1264a63b]

                2017-05-10 10:20:35.400+0000 [http-nio-127.0.0.1-8280-exec-5] DEBUG {} [MSPluginFactory] Find item. TypeName: AdHost

                2017-05-10 10:20:35.484+0000 [http-nio-127.0.0.1-8280-exec-1] DEBUG {} [MSPluginFactory] Find item. TypeName: AdHost

                2017-05-10 10:21:42.419+0000 [http-nio-127.0.0.1-8280-exec-1] WARN  {} [ADObjectFactory] Unsupported entry type! Found ldap entry CN=NTDS Quotas,DC=bsci,DC=bossci,DC=com object classes null

                2017-05-10 10:21:53.925+0000 [http-nio-127.0.0.1-8280-exec-4] WARN  {} [Container] Error while adding Item to a list in the container Staging-DirSync-CRM

                2017-05-10 10:21:54.441+0000 [http-nio-127.0.0.1-8280-exec-4] WARN  {} [Container] Error while adding Item to a list in the container Staging-DirSync-CRM

                2017-05-10 10:21:54.959+0000 [http-nio-127.0.0.1-8280-exec-4] WARN  {} [Container] Error while adding Item to a list in the container Staging-DirSync-CRM

                2017-05-10 10:21:55.470+0000 [http-nio-127.0.0.1-8280-exec-4] WARN  {} [Container] Error while adding Item to a list in the container Staging-DirSync-CRM

                2017-05-10 10:21:56.017+0000 [http-nio-127.0.0.1-8280-exec-4] WARN  {} [Container] Error while adding Item to a list in the container Staging-DirSync-CRM

                2017-05-10 10:21:57.556+0000 [http-nio-127.0.0.1-8280-exec-1] INFO  {} [Service_linux] Starting: [/usr/bin/sudo, /etc/init.d/vco-server, status]

                2017-05-10 10:21:57.626+0000 [http-nio-127.0.0.1-8280-exec-6] INFO  {} [Service_linux] Starting: [/usr/bin/sudo, /etc/init.d/vco-server, status]

                2017-05-10 10:22:01.553+0000 [http-nio-127.0.0.1-8280-exec-4] WARN  {} [Container] Error while adding Item to a list in the container Staging-DirSync-CRM

                2017-05-10 10:23:08.280+0000 [http-nio-127.0.0.1-8280-exec-2] DEBUG {} [MSPluginFactory] Find item. TypeName: OrganizationalUnit

                2017-05-10 10:23:08.287+0000 [http-nio-127.0.0.1-8280-exec-4] DEBUG {} [MSPluginFactory] Find item. TypeName: OrganizationalUnit

                2017-05-10 10:23:08.810+0000 [http-nio-127.0.0.1-8280-exec-3] WARN  {} [Container] Error while adding Item to a list in the container Staging-DirSync-CRM

                2017-05-10 10:23:09.363+0000 [http-nio-127.0.0.1-8280-exec-3] WARN  {} [Container] Error while adding Item to a list in the container Staging-DirSync-CRM

                2017-05-10 10:23:09.896+0000 [http-nio-127.0.0.1-8280-exec-3] WARN  {} [Container] Error while adding Item to a list in the container Staging-DirSync-CRM

                2017-05-10 10:23:10.459+0000 [http-nio-127.0.0.1-8280-exec-3] WARN  {} [Container] Error while adding Item to a list in the container Staging-DirSync-CRM

                2017-05-10 10:23:11.011+0000 [http-nio-127.0.0.1-8280-exec-3] WARN  {} [Container] Error while adding Item to a list in the container Staging-DirSync-CRM

                2017-05-10 10:26:54.929+0000 [tokenLifetimeMonitorScheduler-1] INFO  {} [OAuthTokenLifetimeService] Renewing of security tokens activated for 0 tokens expiring between 2017-05-10 10:25:54.928 and 2017-05-10 10:42:54.928.

                2017-05-10 10:29:19.165+0000 [http-nio-127.0.0.1-8280-exec-3] DEBUG {} [MSPluginFactory] Find item. TypeName: OrganizationalUnit

                2017-05-10 10:29:19.170+0000 [http-nio-127.0.0.1-8280-exec-8] DEBUG {} [MSPluginFactory] Find item. TypeName: OrganizationalUnit

                2017-05-10 10:29:19.688+0000 [http-nio-127.0.0.1-8280-exec-9] WARN  {} [Container] Error while adding Item to a list in the container Staging-DirSync-CRM

                2017-05-10 10:29:20.199+0000 [http-nio-127.0.0.1-8280-exec-9] WARN  {} [Container] Error while adding Item to a list in the container Staging-DirSync-CRM

                2017-05-10 10:29:20.716+0000 [http-nio-127.0.0.1-8280-exec-9] WARN  {} [Container] Error while adding Item to a list in the container Staging-DirSync-CRM

                2017-05-10 10:29:21.282+0000 [http-nio-127.0.0.1-8280-exec-9] WARN  {} [Container] Error while adding Item to a list in the container Staging-DirSync-CRM

                2017-05-10 10:29:21.799+0000 [http-nio-127.0.0.1-8280-exec-9] WARN  {} [Container] Error while adding Item to a list in the container Staging-DirSync-CRM

                • 5. Re: vRO ActiveDirectory Plugin does not return all objects in an orgnizationalUnit
                  marsherian Novice

                  Thanks very much for the information, I'll work on it in my lab, but will change the form to use a search field for the time being,

                  • 6. Re: vRO ActiveDirectory Plugin does not return all objects in an orgnizationalUnit
                    igaydajiev Expert
                    VMware Employees

                    Thanks for the information.

                     

                    Using search list instead is viable option in case you don't need to work with the whole list .

                     

                    To enable additional logging that will reveal the reason for WARN message you can edint

                     

                    /etc/vco/app-server/log4j.xml file and replace

                     

                        <!-- Default root category -->
                    
                    

                    with

                    
                    
                      <category additivity="true" name="ch.dunes.ad.object">
                            <priority value="DEBUG"/>
                            <appender-ref ref="FILE"/>
                        </category>
                        <!-- Default root category -->
                    
                    • 7. Re: vRO ActiveDirectory Plugin does not return all objects in an orgnizationalUnit
                      bsti Enthusiast

                      I ran into this issue trying to return AD Computers using the Plugin.  Using the code you linked to above, here is how I fixed it:

                       

                      /*  getAdComputer.js
                      
                      
                      This script retrieves a list of AD Computers.  It differs from getComputerAd() in that it does not have the 1000 record limit.
                      
                      
                      REFERENCES
                        Action - getAdComputer
                      
                      
                      INPUTS
                        name              (string)   - Specifies a regex filter.  Only computers with names matching the pattern will be returned. This is required.  AD only returns 1000 records by default.
                        distinguishedName (string)   - Specifies a regex filter.  Only computers with DNs matching the pattern will be returned.
                      
                      
                      OUTPUTS
                        ComputerAD[]
                        
                      */
                      
                      
                      //  Get the default AD Host configured in VRO:
                      var adServer = AD_HostManager.findAllHosts()[0];
                      var rootBaseDn = adServer.hostConfiguration.ldapBase;
                      var ldapClient = adServer.getLdapClient();
                      
                      
                      //  Return only name and DN from the LDAP lookup (so we can filter on them):
                      var propertyList = [
                        "name", 
                        "distinguishedName"
                      ];
                      
                      
                      var searchRequest = LdapSearchRequest.createRequest(  
                        rootBaseDn,
                        "(&(objectCategory=computer)(objectClass=computer))",  
                        LdapSearchScope.SUB,  
                        propertyList,
                        LdapDereferencePolicy.ALWAYS);  
                        
                      var resumeCookie = null;
                      var allResults = [];
                      var iterations = 0;
                      while ( true && iterations < 1000 )
                      {
                        //  Don't let the loop go over 1000 iterations to avoid infinite loop.
                        iterations ++;
                      
                      
                        pagedSearchControl = new LdapSimplePagedResultsControl(999999, resumeCookie, true);
                        // Make sure there is no another control registered with same OID  
                        searchRequest.removeControlByOid(pagedSearchControl.getOID());  
                        // add SimplePagedResultsControl to current search control  
                        searchRequest.addControl(pagedSearchControl); 
                      
                      
                        var searchResult = ldapClient.searchBySearchRequest(searchRequest);  
                        var ldapEntries = searchResult.getSearchEntries();
                      
                      
                        if ( ldapEntries )
                        {
                          ldapEntries.map(function (elem) { allResults.push(elem); });
                        }
                      
                      
                        var responseControl = LdapSimplePagedResultsControl.get(searchResult);
                      
                      
                        if ( responseControl.moreResultsToReturn() )
                        {
                          resumeCookie = responseControl.getCookieBytes();
                        }
                        else
                        {
                          break;
                        }
                      } 
                      
                      
                      if ( allResults )
                      {
                        //  Filter by name, if provided:
                        if ( name )
                        {
                          var regx = new RegExp(name,"gi");
                          allResults = allResults.filter(function (elem) { return elem.getAttributeValue("name").match(regx) } );
                        }
                      
                      
                        //  Filter by distinguishedName, if provided:
                        if ( distinguishedName )
                        {
                          var regx = new RegExp(distinguishedName,"gi");
                          allResults = allResults.filter(function (elem) { return elem.getAttributeValue("distinguishedName").match(regx) } );
                        }
                      
                      
                        //  Convert the LdapEntry results to ComputerAD objects:
                        var adComputers = allResults.map(function (elem) { return ActiveDirectory.getComputerAD(elem.getAttributeValue("name", null)); });
                      
                      
                        //  Sort the collection by name:
                        adComputers = adComputers.sort(function (itm1,itm2) { if ( itm1.name.toLowerCase() >= itm2.name.toLowerCase() ) { return 1; } else { return -1; } });
                          
                        System.debug("Computer objects found: " + adComputers.length);
                      
                      
                        return adComputers;
                      }
                      else
                      {
                        throw "No LDAP entries found for AD Computers in " + rootBaseDn + "!";
                      }
                      

                       

                      I implemented this as an action that returns Computer objects, but you can modify the code to return other objects as you like.

                      • 8. Re: vRO ActiveDirectory Plugin does not return all objects in an orgnizationalUnit
                        samwalker1380 Novice
                        vExpertVMware Employees

                        I realise it's been a while since the original post, but I have hit the same issue recently so did a write up about it... In short, it's an AD limitation, although you can change it using ntdsutil...

                         

                        https://www.6cd.co.uk/2019/01/issues-with-vro-and-active-directory.html

                        1 person found this helpful