1 2 Previous Next 17 Replies Latest reply on Nov 28, 2016 8:12 AM by mobcdi

    VDP 6.1.2 appliance can't connect to vCenter after initial configuration

    jkhardy95 Novice

      I've deployed VDP 6.1.2 and the configuration went smoothly (test connection to vCenter with proper permissions worked successfully), but after the the appliance rebooted it hung at 70% complete for the VDP:Configure appliance task.  In the /usr/local/avamar/var/vdr/server_logs/vdr-server.log the VDP appliance can't connect to the vCenter server.  Any idea as to how to fix this?  Is there a config file I need to update?

       

      2016-08-24 12:30:32,895 INFO  [Thread-7]-vi.ViJavaServiceInstanceProviderImpl: vcenter-ignore-cert ? true

      2016-08-24 12:30:33,007 WARN  [Thread-7]-vi.VCenterServiceImpl: No VCenter found in MC root domain

      2016-08-24 12:30:33,008 INFO  [Thread-7]-vi.ViJavaServiceInstanceProviderImpl: visdkUrl = https:/sdk

      2016-08-24 12:30:33,008 ERROR [Thread-7]-vi.ViJavaServiceInstanceProviderImpl: Failed To Create ViJava ServiceInstance owing to Remote VCenter connection error

      java.rmi.RemoteException: VI SDK invoke exception:java.lang.IllegalArgumentException: protocol = https host = null; nested exception is:

              java.lang.IllegalArgumentException: protocol = https host = null

              at com.vmware.vim25.ws.WSClient.invoke(WSClient.java:139)

              at com.vmware.vim25.ws.VimStub.retrieveServiceContent(VimStub.java:2114)

              at com.vmware.vim25.mo.ServiceInstance.<init>(ServiceInstance.java:117)

              at com.vmware.vim25.mo.ServiceInstance.<init>(ServiceInstance.java:95)

              at com.emc.vdp2.common.vi.ViJavaServiceInstanceProviderImpl.createViJavaServiceInstance(ViJavaServiceInstanceProviderImpl.java:297)

              at com.emc.vdp2.common.vi.ViJavaServiceInstanceProviderImpl.createViJavaServiceInstance(ViJavaServiceInstanceProviderImpl.java:159)

              at com.emc.vdp2.common.vi.ViJavaServiceInstanceProviderImpl.createViJavaServiceInstance(ViJavaServiceInstanceProviderImpl.java:104)

              at com.emc.vdp2.common.vi.ViJavaServiceInstanceProviderImpl.createViJavaServiceInstance(ViJavaServiceInstanceProviderImpl.java:96)

              at com.emc.vdp2.common.vi.ViJavaServiceInstanceProviderImpl.getViJavaServiceInstance(ViJavaServiceInstanceProviderImpl.java:74)

              at com.emc.vdp2.common.vi.ViJavaServiceInstanceProviderImpl.waitForViJavaServiceInstance(ViJavaServiceInstanceProviderImpl.java:212)

              at com.emc.vdp2.server.VDRServletLifeCycleListener$1.run(VDRServletLifeCycleListener.java:71)

              at java.lang.Thread.run(Unknown Source)

      Caused by: java.lang.IllegalArgumentException: protocol = https host = null

              at sun.net.spi.DefaultProxySelector.select(Unknown Source)

              at sun.net.www.protocol.http.HttpURLConnection.plainConnect0(Unknown Source)

              at sun.net.www.protocol.http.HttpURLConnection.plainConnect(Unknown Source)

              at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(Unknown Source)

              at sun.net.www.protocol.http.HttpURLConnection.getOutputStream0(Unknown Source)

              at sun.net.www.protocol.http.HttpURLConnection.getOutputStream(Unknown Source)

              at sun.net.www.protocol.https.HttpsURLConnectionImpl.getOutputStream(Unknown Source)

              at com.vmware.vim25.ws.WSClient.post(WSClient.java:216)

              at com.vmware.vim25.ws.WSClient.invoke(WSClient.java:133)

              ... 11 more

      2016-08-24 12:30:33,009 INFO  [Thread-7]-vi.ViJavaServiceInstanceProviderImpl: Retry ViJava ServiceInstance Acquisition In 5 Seconds...

        • 1. Re: VDP 6.1.2 appliance can't connect to vCenter after initial configuration
          virtualg_uk Expert
          vExpertUser Moderators

          Could you get it to boot up fully? Then login to the VDP appliance via https://VDP_IP/vdp-configure

          and check all configuration, specifically DNS etc and report back

          • 2. Re: VDP 6.1.2 appliance can't connect to vCenter after initial configuration
            jkhardy95 Novice

            Yes, it boots up fully and I can log back into the vdp-configure webpage, but it's not in maintenance mode, it's still in configuration mode.  Connection tests are successful.

             

            Capture.JPG

            • 3. Re: VDP 6.1.2 appliance can't connect to vCenter after initial configuration
              virtualg_uk Expert
              vExpertUser Moderators

              Let's also verify the vCenter Certificate and move through the configuration.

              • 4. Re: VDP 6.1.2 appliance can't connect to vCenter after initial configuration
                jkhardy95 Novice

                I opened up a ticket with VMWare and got the following response.  I verified this is indeed the error in the mcserver.out log file.  I updated the mc perl module and then ran /usr/local/avamarclient/etc/registervcenter.sh to register the VDP appliance with vCenter and that was successful and I can log into the VDP appliance via the vSphere Web Client.  I can create new backup jobs but they fail...still working with VMWare on it.

                 

                VMWARE did a piss-poor job of regression testing vCenter 5.5 U3e with old VDP appliances.  If they would've they would've seen this problem!!!

                 

                Knowledge Base Article: 000487555

                MCS cannot connect to vcenter due to Unsupported curve: 1.2.840.10045.3.1.7 (000487555)

                Version: 4 Article Type: Break Fix Audience: Level 30 = Customers Last Published: Tue Aug 16 21:34:49 GMT 2016 Summary:

                Issue:

                Mcs cannot add or reconnect to vcenter due to Jsafe 5.0/Java8/Vcenter certain certificate/server compatibility

                mcserver.out shows the following java stack trace

                Started
                Caught Exception : Exception : org.apache.axis.AxisFault Message : ; nested exception is:

                javax.net.ssl.SSLHandshakeException: Unsupported curve: 1.2.840.10045.3.1.7 StackTrace : AxisFault

                faultCode: {http://schemas.xmlsoap.org/soap/envelope/}Server.userException faultSubcode:
                faultString: javax.net.ssl.SSLHandshakeException: Unsupported curve: 1.2.840.10045.3.1.7 faultActor:
                faultNode:
                faultDetail:

                {http://xml.apache.org/axis/}stackTrace:javax.net.ssl.SSLHandshakeException: Unsupported curve: 1.2.840.10045.3.1.7

                Cause: Interoperability between some between Java 8 ( used in 7.2.1+ avamar ) , EMC RSA java library called JSAFE/Bsafe 5.0 and certain vcenter server SSL ciphers settings in vcenter 5.5 U3e and Vcenter 6.5 beta.


                Change: Either of the following causes can trigger this issue.
                1. Upgraded vcenter to 5.5U3e/ vcenter 6.5 beta
                2. Upgrade to avamar 7.2 SP1/7.3 , or MCS hotfix 250666 for 7.1 SP2, mcs hotfix 2555641 for 7.2.0 3. Adding a new vcenter running 5.5U3e/ 6.5 beta or an existing avamar enviroment


                Resolution: Short term solution.

                Removed RSA BSAFE java library and use the Java JCE provider library instead, this should only be considered a short term workaround as emc preferred library is RSA.

                1. edit the file /usr/local/avamar/lib/mcsutils.pm
                2. Add the line to the $prefs variable in the mc perl module (note the leading . and blank space before the ending quote): . "-Dsecurity.provider.rsa.JsafeJCE.position=last "

                before:
                my $prefs = "-Djava.util.logging.config.file=$mcsvar::lib_dir/mcserver_logging.properties "

                . "-Djava.security.egd=file:/dev/./urandom "

                . "-Djava.io.tmpdir=$mcsvar::tmp_dir "
                . "-Djava.util.prefs.PreferencesFactory=com.avamar.mc.util.MCServerPreferencesFactory "
                . "-Djavax.xml.parsers.DocumentBuilderFactory=org.apache.xerces.jaxp.DocumentBuilderFactoryImpl " . "-Djavax.net.ssl.keyStore=" . MCServer::get( "rmi_ssl_keystore" ) ." "
                . "-Djavax.net.ssl.trustStore=" . MCServer::get( "rmi_ssl_keystore" ) ." "
                . "-Dfile.encoding=UTF-8 "
                . "-Dlog4j.configuration=file://$mcsvar::lib_dir/log4j.properties "; # vmware/axis

                after:
                my $prefs = "-Djava.util.logging.config.file=$mcsvar::lib_dir/mcserver_logging.properties "

                . "-Djava.security.egd=file:/dev/./urandom "
                . "-Djava.io.tmpdir=$mcsvar::tmp_dir "
                . "-Djava.util.prefs.PreferencesFactory=com.avamar.mc.util.MCServerPreferencesFactory "
                . "-Djavax.xml.parsers.DocumentBuilderFactory=org.apache.xerces.jaxp.DocumentBuilderFactoryImpl " . "-Djavax.net.ssl.keyStore=" . MCServer::get( "rmi_ssl_keystore" ) ." "
                . "-Djavax.net.ssl.trustStore=" . MCServer::get( "rmi_ssl_keystore" ) ." "
                . "-Dfile.encoding=UTF-8 "
                . "-Dsecurity.provider.rsa.JsafeJCE.position=last "
                . "-Dlog4j.configuration=file://$mcsvar::lib_dir/log4j.properties "; # vmware/axis


                3. Restart mcs service: mcserver.sh --restart command.

                Long term solution and prefered solution:  A long term fix for other avamar releases is being discussed with engineering

                • 5. Re: VDP 6.1.2 appliance can't connect to vCenter after initial configuration
                  didi2008 Lurker

                  I tried this, but the error is still there. The mcserver.sh --restart didn't work (after su - admin, too).

                  When should I make the changes? After the first start or when it hangs at 70%?

                  I tried to make the change after the first start of the VDP. Then the configuration hangs at 35%.

                  • 6. Re: VDP 6.1.2 appliance can't connect to vCenter after initial configuration
                    lhromadka Lurker

                    I have the same problem with VDP 5.8.4 and vCenter 5.5.0 Update 3e.


                    Initial VDP configuration hangs on 70%.


                    /usr/local/avamar/var/vdr/server_logs/vdr-server.log:


                    2016-09-03 08:21:07,192 WARN  [Thread-12]-vi.VCenterServiceImpl: No VCenter found in MC root domain

                    2016-09-03 08:21:07,217 INFO  [Thread-12]-vi.ViJavaServiceInstanceProviderImpl: vcenter-ignore-cert ? true

                    2016-09-03 08:21:07,272 ERROR [Thread-12]-vi.ViJavaServiceInstanceProviderImpl: Failed To Create ViJava ServiceInstance

                    java.rmi.RemoteException: VI SDK invoke exception:java.lang.IllegalArgumentException: protocol = https host = null

                      at com.vmware.vim25.ws.WSClient.invoke(WSClient.java:219)

                      at com.vmware.vim25.ws.WSClient.invoke(WSClient.java:144)

                      at com.vmware.vim25.ws.VimStub.retrieveServiceContent(VimStub.java:1480)

                      at com.vmware.vim25.mo.ServiceInstance.<init>(ServiceInstance.java:111)

                      at com.vmware.vim25.mo.ServiceInstance.<init>(ServiceInstance.java:95)

                      at com.emc.vdp2.common.vi.ViJavaServiceInstanceProviderImpl.createViJavaServiceInstance(ViJavaServiceInstanceProviderImpl.java:252)

                      at com.emc.vdp2.common.vi.ViJavaServiceInstanceProviderImpl.createViJavaServiceInstance(ViJavaServiceInstanceProviderImpl.java:150)

                      at com.emc.vdp2.common.vi.ViJavaServiceInstanceProviderImpl.createViJavaServiceInstance(ViJavaServiceInstanceProviderImpl.java:92)

                      at com.emc.vdp2.common.vi.ViJavaServiceInstanceProviderImpl.getViJavaServiceInstance(ViJavaServiceInstanceProviderImpl.java:70)

                      at com.emc.vdp2.common.vi.ViJavaServiceInstanceProviderImpl.waitForViJavaServiceInstance(ViJavaServiceInstanceProviderImpl.java:166)

                      at com.emc.vdp2.server.VDRServletLifeCycleListener$1.run(VDRServletLifeCycleListener.java:73)

                      at java.lang.Thread.run(Unknown Source)

                     

                    /space/avamar/var/mc/server_log/mcserver.out:

                     

                    === BEGIN === check.mcs (prestart)

                    check.mcs                        passed

                    === PASS === check.mcs PASSED OVERALL (prestart)

                    Starting Administrator Server at: Fri Sep  2 23:17:05 PDT 2016

                    Starting Administrator Server...

                    Warning:  org.apache.xerces.jaxp.SAXParserImpl$JAXPSAXParser: Property 'http://www.oracle.com/xml/jaxp/properties/entityExpansionLimit' is not recognized.

                    Compiler warnings:

                      WARNING:  'org.apache.xerces.jaxp.SAXParserImpl: Property 'http://javax.xml.XMLConstants/property/accessExternalDTD' is not recognized.'

                    2016-09-02 23:17:37.583:INFO::Logging to STDERR via org.mortbay.log.StdErrLog

                    2016-09-02 23:17:37.702:INFO::jetty-6.1.23

                    2016-09-02 23:17:37.760:INFO::Extract lib/mcsdk-axis2.war to /usr/local/avamar/var/mc/server_tmp/Jetty_0_0_0_0_9443_mcsdk.axis2.war____.xz4n3v/webapp

                    2016-09-02 23:17:44.819:INFO::Started SslSocketConnector@0.0.0.0:9443

                    Administrator Server started.

                    Started

                    Caught Exception :  Exception : org.apache.axis.AxisFault Message : ; nested exception is:

                      javax.net.ssl.SSLHandshakeException: Unsupported curve: 1.2.840.10045.3.1.7 StackTrace :

                    AxisFault

                    faultCode: {http://schemas.xmlsoap.org/soap/envelope/}Server.userException

                    faultSubcode:

                    faultString: javax.net.ssl.SSLHandshakeException: Unsupported curve: 1.2.840.10045.3.1.7

                    faultActor:

                    faultNode:

                    faultDetail:

                      {http://xml.apache.org/axis/}stackTrace:javax.net.ssl.SSLHandshakeException: Unsupported curve: 1.2.840.10045.3.1.7

                      at sun.security.ssl.HandshakeMessage$ECDH_ServerKeyExchange.&lt;init&gt;(Unknown Source)

                      at sun.security.ssl.ClientHandshaker.processMessage(Unknown Source)



                    I updated the /usr/local/avamar/lib/mcsutils.pm. I add then line with JsafeJCE.position=last.

                     

                    admin@vmdp1:~/>: mcserver.sh --restart:

                     

                    === BEGIN === check.mcs (poststart)

                    check.mcs                        passed

                    === PASS === check.mcs PASSED OVERALL (poststart)

                    --restart will restart the Administrator Server.

                    Do you want to proceed with the restart Y/N? [Y]: y

                     

                     

                    Administrator Server shutdown initiated.

                    Stopping Administrator Server...

                    Administrator Server stopped.

                    Database server is running...

                    INFO: Starting messaging service.

                    INFO: Started messaging service.

                    === BEGIN === check.mcs (prestart)

                    check.mcs                        passed

                    === PASS === check.mcs PASSED OVERALL (prestart)

                    Starting Administrator Server at: Sat Sep  3 10:53:58 CEST 2016

                    Starting Administrator Server...

                    Warning:  org.apache.xerces.jaxp.SAXParserImpl$JAXPSAXParser: Property 'http://www.oracle.com/xml/jaxp/properties/entityExpansionLimit' is not recognized.

                    Compiler warnings:

                      WARNING:  'org.apache.xerces.jaxp.SAXParserImpl: Property 'http://javax.xml.XMLConstants/property/accessExternalDTD' is not recognized.'

                    2016-09-03 10:54:30.301:INFO::Logging to STDERR via org.mortbay.log.StdErrLog

                    2016-09-03 10:54:30.384:INFO::jetty-6.1.23

                    2016-09-03 10:54:30.425:INFO::Extract lib/mcsdk-axis2.war to /usr/local/avamar/var/mc/server_tmp/Jetty_0_0_0_0_9443_mcsdk.axis2.war____.xz4n3v/webapp

                    2016-09-03 10:54:36.621:INFO::Started SslSocketConnector@0.0.0.0:9443

                    Administrator Server started.

                    INFO: Starting Data Domain SNMP Manager....

                    INFO: Connecting to MCS Server: vmdp1.gordic.cz at port: 7778...

                    INFO: Successfully connected to MCS Server: vmdp1.gordic.cz at port: 7778.

                    INFO: No trap listeners were started, Data Domain SNMP Manager didn't start.

                     

                    Page https://....vdp....:8543/vdp-configure/ is still in configuration mode.


                    Page https://....vcenter..../mob/?moid=ExtensionManager show only com.vmware.vdp2.config extension. The com.vmware.vdp2 extension missing.


                    The VDP VM are still displaing task "com.vmware.vdp2.config.initialconfig".


                    What to do next? Restart VDP VM with uncomplete configuration?


                    • 7. Re: VDP 6.1.2 appliance can't connect to vCenter after initial configuration
                      jarik_boom Lurker

                      I also have the same problem with VDP 5.8.4 and vCenter 5.5.0 Update 3e (5.5.0.30500 Build 4180648). VDP configuration stuck at 70%.

                      The problem was solved by adding a line in /usr/local/avamar/lib/mcsutils.pm

                      . "-Dsecurity.provider.rsa.JsafeJCE.position=last "

                      before the first start of VDP configuration wizard (immediately after deploy OVF template)


                      Thank you, jkhardy95.

                      • 8. Re: VDP 6.1.2 appliance can't connect to vCenter after initial configuration
                        komanek Enthusiast

                        Hello,

                         

                        thank you very much for this thread, it helped me a lot. After vCenter upgrade to 5.5u3e, my VDP 6.0.3 appliance was unusable. As far as there is nothing regarding this problem in vCenter or VDP release notes and VDP 6.0.x is officially supported with vCenter 5.5u3, I tried many obscure things, but with no luck. I can also confirm that the same issue appears for VDP 5.8.x after vCenter upgrade.

                         

                        So it was great to finally find the solution here. It helped me to reconnect VDP 6.0.3 to the upgrade vCenter at one site (I edited the library file and restarted service as subbested) and also to deply a new instance of VDP 6.0.4 at my second site (I edited the library file after OVA deployment, before I entered the configuration wizard).

                         

                        Still, I have one question regarding in-place upgrade of VDP 6.0.3 to 6.0.4. Is there a working scenario in this situation or shloudl I wait for 6.0.5 with official patch (if it ever will be released) ? It still could be tricky because of kernel hotfix, which needs to be installed before the upgrade to 6.0.4, but which is not part of the upgrade ISO :-/

                         

                        Thanks again and best regards,

                         

                        David

                        • 9. Re: VDP 6.1.2 appliance can't connect to vCenter after initial configuration
                          VictorWangBeginner Lurker

                          Hi, Jarik and other kind helpers,

                           

                          Can somebody explain in details how to update the file?

                           

                          /usr/local/avamar/lib/mcsutils.pm

                           

                          I tried to install VPD 6.1.1 and VDP6.1.2 on ESXi 5.5 (latest version/patch), both got stucked at 70% after initial setup. I am using vCenter version:

                           

                          VMware vCenter Server Appliance
                          Update to version 5.5.0 Update 3e

                          5.5.0.30500 Build 4180648

                           

                          Is this mcsutils.pm file located on VDP Appliance? or on vCenter or ESXi host? It will be too much for me to add a line to a file on VDP Appliance's disk, without powering it on.

                           

                          The reason why I want to install a new VDP is that my VDP6.1.1 suddenly stopped working, with "Log file blank" error on all backup jobs. I suspect that this is related with my recent update of vCenter 5.5.

                           

                          Thank you very much.

                           

                          Victor Wang

                          • 10. Re: VDP 6.1.2 appliance can't connect to vCenter after initial configuration
                            VictorWangBeginner Lurker

                            I believe that I figured it out:

                             

                            1. Deploy OVF, I am using 6.1.2.

                            2. Power it ON, but do not run VDP config.

                            3. Logon to console with user "root" password "changeme".

                            4. Go to /usr/local/avamar/lib/ ,use VIM to edit mcsutil.pm, save it. You need VIM command ":w!" to save a read only file.

                            5. Restart service "mcserver.sh --restart". In my case I got an error "ERROR: flag needs two dashes: "-restart', stopped at mcflags.pm line 380. As a precaution, I reboot this VM from vCenter's page.

                            6. After reboot, wait for 5 minutes, go to VDP config page: https://YOURVDP IP ADDRESS:8543/vdp-configure

                            7. Start config it normally.

                             

                            Hope this will help you. Thanks.

                            • 11. Re: VDP 6.1.2 appliance can't connect to vCenter after initial configuration
                              VictorWangBeginner Lurker

                              72 Hours after fresh installation, the vm send me an email:

                               

                              Start the Backup Scheduler by using the vSphere Data Protection Configuration utility.

                               

                              Tried to connect to VDP-Config but stucked with a grey bar with one block on left (inside bar). Tried to access it from vCenter, it seems still functioning. I rebooted this VM, hopfully this is just random problem.

                               

                              The VDP did backed up several VMs, I am trying to restore one VM now. Will keep posting.  After several hours: The restore of my test VM was success. I will keep an eye on that backup scheduler error.

                              • 12. Re: VDP 6.1.2 appliance can't connect to vCenter after initial configuration
                                ACTScott Lurker

                                This solution worked perfectly for me as well but I too am getting the warnings regarding scheduler service even though the backups are working as scheduled. Could be a false alarm. I have VMware calling me back today and I will post back anything I hear back from them.

                                • 13. Re: VDP 6.1.2 appliance can't connect to vCenter after initial configuration
                                  GMZSE Novice

                                  The extra line of code made my appliance configure to 100%, however I can't connect to any VDP appliances from the Web Client. After clicking 'Connect', it just hangs and nothing happens.

                                   

                                  • vCenter: 5.5 U3e (Build 4180647)
                                  • Two older appliances, version 5.8.1.7. These worked after upgrading vCenter.
                                  • One newer appliance, version 5.8.4.6 . This replaced a previous appliance (version 5.8.3.13) that stopped working after the vCenter upgrade.


                                  Has anyone run into this issue yet? What was the resolution?

                                   

                                  • 14. Re: VDP 6.1.2 appliance can't connect to vCenter after initial configuration
                                    itttam Lurker

                                    . "-Dsecurity.provider.rsa.JsafeJCE.position=last "

                                    Where should this line go in the mcsutils.pm?

                                     

                                    thanks

                                     

                                    NM, found where it goes after rereading the thread. Giving it another go. Hopefully no longer stuck at 70%

                                     

                                     

                                    vCSA version 5.5.0 Update 3e

                                    5.5.0.30500 Build 4180648


                                    VDP 6.1.2.19

                                    1 2 Previous Next