4 Replies Latest reply on Jun 7, 2017 1:27 AM by Czernobog

    vRO 7.2 - AD query fails randomly

    Czernobog Hot Shot

      Hello,

      I'm using the vRA embedded vRO 7.2 Instance. It's running with the AD Plugin ver. 3.0.3

      I've added 2 different domains using the plugin, using the domain URL as the AD Host URL (not single domain controllers): mycorp.domain.local

      Configuration options:

      port 3269 (LDAPS), Use SSL & Do not ask when importing cert = yes, shared session = yes

      When querying for AD Users using following script:

      var Users = ActiveDirectory.searchExactMatch("User" , UPN , 2);
      

      I often get an error:

      item: "MyWorkflow/item17", state: "failed", business state: "null", exception: "java.lang.NullPointerException
      

       

      The full error stack is as follows:

       

      2017-06-02 08:11:47.926+0000 [WorkflowExecutorPool-Thread-163] ERROR {my.user@mycorp.domain.local:MyWorflow:e6537d59-64fa-484c-8168-86d37b199217:token=ff8080815c62278e015c67dc44fc05c1} [AdHost] Unable to create initial LDAP Context LDAPException(resultCode=81 (server down), errorMessage="An error occurred while attempting to send the LDAP message to server mycorp.domain.local:3269: javax.net.ssl.SSLHandshakeException: server certificate change is restricted during renegotiation") at com.unboundid.ldap.sdk.LDAPConnectionInternals.sendMessage(LDAPConnectionInternals.java:574) at com.unboundid.ldap.sdk.LDAPConnection.sendMessage(LDAPConnection.java:4247) at com.unboundid.ldap.sdk.SimpleBindRequest.process(SimpleBindRequest.java:551) at com.unboundid.ldap.sdk.LDAPConnection.bind(LDAPConnection.java:2141) at com.unboundid.ldap.sdk.LDAPConnection.bind(LDAPConnection.java:2086) at com.vmware.o11n.plugin.ad.connection.ConnectionManager.getConnection(ConnectionManager.java:50) at com.vmware.o11n.plugin.ad.model.AdHost.getLdapConnection(AdHost.java:164) at com.vmware.o11n.plugin.ad.model.AdHost.initConnection(AdHost.java:169) at com.vmware.o11n.plugin.ad.model.AdHost.assureInitialized(AdHost.java:158) at com.vmware.o11n.plugin.ad.model.AdHost.getActiveDirectory(AdHost.java:52) at ch.dunes.ad.object.ADSingleton.searchExactMatch(ADSingleton.java:37) at sun.reflect.GeneratedMethodAccessor2003.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at ch.dunes.vso.sdk.WrappedJavaMethod.invoke(WrappedJavaMethod.java:217) at ch.dunes.vso.sdk.WrappedJavaMethod.call(WrappedJavaMethod.java:165) at org.mozilla.javascript.Interpreter.interpretLoop(Interpreter.java:1473) at org.mozilla.javascript.Interpreter.interpret(Interpreter.java:815) at org.mozilla.javascript.InterpretedFunction.call(InterpretedFunction.java:109) at ch.dunes.scripting.server.script.DynamicFunction.call(DynamicFunction.java:179) at org.mozilla.javascript.Interpreter.interpretLoop(Interpreter.java:1473) at org.mozilla.javascript.Interpreter.interpret(Interpreter.java:815) at org.mozilla.javascript.InterpretedFunction.call(InterpretedFunction.java:109) at ch.dunes.scripting.server.script.DynamicFunction.call(DynamicFunction.java:179) at org.mozilla.javascript.Interpreter.interpretLoop(Interpreter.java:1473) at org.mozilla.javascript.Interpreter.interpret(Interpreter.java:815) at org.mozilla.javascript.InterpretedFunction.call(InterpretedFunction.java:109) at ch.dunes.scripting.server.script.DynamicFunction.call(DynamicFunction.java:179) at org.mozilla.javascript.Interpreter.interpretLoop(Interpreter.java:1473) at org.mozilla.javascript.Interpreter.interpret(Interpreter.java:815) at org.mozilla.javascript.InterpretedFunction.call(InterpretedFunction.java:109) at org.mozilla.javascript.ContextFactory.doTopCall(ContextFactory.java:394) at org.mozilla.javascript.ScriptRuntime.doTopCall(ScriptRuntime.java:3091) at org.mozilla.javascript.InterpretedFunction.exec(InterpretedFunction.java:120) at ch.dunes.scripting.server.script.MainScriptingObject.executeScript(MainScriptingObject.java:257) at ch.dunes.scripting.server.script.MainScriptingObject.executeScript(MainScriptingObject.java:243) at ch.dunes.workflow.engine.mbean.WorkflowScriptRunner.execute(WorkflowScriptRunner.java:185) at ch.dunes.workflow.engine.mbean.runner.WorkflowItemTaskRunner.execute(WorkflowItemTaskRunner.java:44) at ch.dunes.workflow.engine.mbean.runner.WorkflowItemTaskRunner.execute(WorkflowItemTaskRunner.java:25) at ch.dunes.workflow.engine.mbean.helper.WorkflowHandler.executeItem(WorkflowHandler.java:1176) at ch.dunes.workflow.engine.mbean.helper.WorkflowHandler.requestElementExecution(WorkflowHandler.java:1136) at ch.dunes.workflow.engine.mbean.helper.WorkflowHandler.handleWorkflowTokenNextStep(WorkflowHandler.java:799) at ch.dunes.workflow.engine.mbean.helper.WorkflowHandler.executeToken(WorkflowHandler.java:684) at ch.dunes.workflow.engine.mbean.helper.WorkflowHandler.handleTokenExecution(WorkflowHandler.java:610) at ch.dunes.workflow.engine.mbean.helper.WorkflowHandler.access$100(WorkflowHandler.java:114) at ch.dunes.workflow.engine.mbean.helper.WorkflowHandler$1.execute(WorkflowHandler.java:398) at ch.dunes.model.ar.AccessRightsTemplate.executeWithAccessRights(AccessRightsTemplate.java:16) at ch.dunes.workflow.engine.mbean.helper.WorkflowHandler.doExecute(WorkflowHandler.java:394) at ch.dunes.workflow.engine.mbean.helper.WorkflowHandler.run(WorkflowHandler.java:265) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: javax.net.ssl.SSLHandshakeException: server certificate change is restricted during renegotiation at sun.security.ssl.Alerts.getSSLException(Alerts.java:192) at sun.security.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1949) at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:302) at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:292) at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1471) at sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:216) at sun.security.ssl.Handshaker.processLoop(Handshaker.java:979) at sun.security.ssl.Handshaker.process_record(Handshaker.java:914) at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1062) at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1375) at sun.security.ssl.SSLSocketImpl.writeRecord(SSLSocketImpl.java:747) at sun.security.ssl.AppOutputStream.write(AppOutputStream.java:123) at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82) at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140) at com.unboundid.ldap.sdk.LDAPConnectionInternals.sendMessage(LDAPConnectionInternals.java:543) ... 53 more

       

      I've already imported the root certificate of the domain. Do I have to import the ssl certificates of all DC's that are members of the domain too? Or does the error lie elsewhere?

        • 1. Re: vRO 7.2 - AD query fails randomly
          igaydajiev Expert
          VMware Employees

          This error message in a consequence of code hardening in Java. (SSL V3.0 Poodle Vulnerability - CVE-2014-3566" )

          You can take a look at following link for a bit more info and few suggested workarounds.

          tomcat - What means "javax.net.ssl.SSLHandshakeException: server certificate change is restrictedduring renegotiation" a… 

           

          If possible I would first try option

          A third option is to "improve" your server certificates to include all IP addresses of your cluster members as Subject Alternative Names according to this post in Burp forum

           

          If third option is not possible then I would try

          A second option is to relax client check to still allow renegotiation with the following properties:

          -Djdk.tls.allowUnsafeServerCertChange=true -Dsun.security.ssl.allowUnsafeRenegotiation=true

          In context of vRO this settings can be applied in /var/lib/vco-appserver/bin/setenv.sh file

          #Allow renegotioation

          JVM_OPTS="$JVM_OPTS -Djdk.tls.allowUnsafeServerCertChange=true -Dsun.security.ssl.allowUnsafeRenegotiation=true"

          1 person found this helpful
          • 2. Re: vRO 7.2 - AD query fails randomly
            Czernobog Hot Shot

            I've edited the setenv.sh file, to include the part:

            #Allow renegotioation
            JVM_OPTS="$JVM_OPTS -Djdk.tls.allowUnsafeServerCertChange=true -Dsun.security.ssl.allowUnsafeRenegotiation=true"
            

            After restarting the vco-server service, I still get the same error. Do I have to restart the vRA appliance?

             

            Also, in the last few days I was able to mitigate the issue by adding single domain controllers to the AD Plugin, setting one of those as default, and running my workflows with that small pool of AD controllers.

            • 3. Re: vRO 7.2 - AD query fails randomly
              igaydajiev Expert
              VMware Employees

              >After restarting the vco-server service, I still get the same error. Do I have to restart the vRA appliance?

              Restarting vco-service is enough.

              Just make sure that

              >JVM_OPTS="$JVM_OPTS -Djdk.tls.allowUnsafeServerCertChange=true -Dsun.security.ssl.allowUnsafeRenegotiation=true"

              is added before

              >CATALINA_OPTS="$REMOTE_DEBUG $JVM_OPTS $AGENT_PATHS $JAVA_AGENTS $JAVA_LIBRARY_PATH"

               

               

              >Also, in the last few days I was able to mitigate the issue by adding single domain controllers to the AD Plugin, setting one of those as default, and running my workflows with that small pool of AD controllers.

              Actually recommendation is to add DC directly (not domain). Note that latest tech preview version (Ad 3.0.6) adds support for client side HA/Load-balancing. In AD 3.0.6 you can configure  alternative host list. This list is used be used either for HA or load balancing.

              if HA mode is selected in case of failure of request toward main DC configured same request is retried on other DC

              If Load-balancing mode is selected then requests will be load balanced between all configured hosts.

              • 4. Re: vRO 7.2 - AD query fails randomly
                Czernobog Hot Shot

                I can live with adding single DC's for now, since my workflow does the "load balancing" and will wait for the official release of the new AD Plugin.