VMware Cloud Community
jkhardy95
Contributor
Contributor

VDP 6.1.2 appliance can't connect to vCenter after initial configuration

I've deployed VDP 6.1.2 and the configuration went smoothly (test connection to vCenter with proper permissions worked successfully), but after the the appliance rebooted it hung at 70% complete for the VDP:Configure appliance task.  In the /usr/local/avamar/var/vdr/server_logs/vdr-server.log the VDP appliance can't connect to the vCenter server.  Any idea as to how to fix this?  Is there a config file I need to update?

2016-08-24 12:30:32,895 INFO  [Thread-7]-vi.ViJavaServiceInstanceProviderImpl: vcenter-ignore-cert ? true

2016-08-24 12:30:33,007 WARN  [Thread-7]-vi.VCenterServiceImpl: No VCenter found in MC root domain

2016-08-24 12:30:33,008 INFO  [Thread-7]-vi.ViJavaServiceInstanceProviderImpl: visdkUrl = https:/sdk

2016-08-24 12:30:33,008 ERROR [Thread-7]-vi.ViJavaServiceInstanceProviderImpl: Failed To Create ViJava ServiceInstance owing to Remote VCenter connection error

java.rmi.RemoteException: VI SDK invoke exception:java.lang.IllegalArgumentException: protocol = https host = null; nested exception is:

        java.lang.IllegalArgumentException: protocol = https host = null

        at com.vmware.vim25.ws.WSClient.invoke(WSClient.java:139)

        at com.vmware.vim25.ws.VimStub.retrieveServiceContent(VimStub.java:2114)

        at com.vmware.vim25.mo.ServiceInstance.<init>(ServiceInstance.java:117)

        at com.vmware.vim25.mo.ServiceInstance.<init>(ServiceInstance.java:95)

        at com.emc.vdp2.common.vi.ViJavaServiceInstanceProviderImpl.createViJavaServiceInstance(ViJavaServiceInstanceProviderImpl.java:297)

        at com.emc.vdp2.common.vi.ViJavaServiceInstanceProviderImpl.createViJavaServiceInstance(ViJavaServiceInstanceProviderImpl.java:159)

        at com.emc.vdp2.common.vi.ViJavaServiceInstanceProviderImpl.createViJavaServiceInstance(ViJavaServiceInstanceProviderImpl.java:104)

        at com.emc.vdp2.common.vi.ViJavaServiceInstanceProviderImpl.createViJavaServiceInstance(ViJavaServiceInstanceProviderImpl.java:96)

        at com.emc.vdp2.common.vi.ViJavaServiceInstanceProviderImpl.getViJavaServiceInstance(ViJavaServiceInstanceProviderImpl.java:74)

        at com.emc.vdp2.common.vi.ViJavaServiceInstanceProviderImpl.waitForViJavaServiceInstance(ViJavaServiceInstanceProviderImpl.java:212)

        at com.emc.vdp2.server.VDRServletLifeCycleListener$1.run(VDRServletLifeCycleListener.java:71)

        at java.lang.Thread.run(Unknown Source)

Caused by: java.lang.IllegalArgumentException: protocol = https host = null

        at sun.net.spi.DefaultProxySelector.select(Unknown Source)

        at sun.net.www.protocol.http.HttpURLConnection.plainConnect0(Unknown Source)

        at sun.net.www.protocol.http.HttpURLConnection.plainConnect(Unknown Source)

        at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(Unknown Source)

        at sun.net.www.protocol.http.HttpURLConnection.getOutputStream0(Unknown Source)

        at sun.net.www.protocol.http.HttpURLConnection.getOutputStream(Unknown Source)

        at sun.net.www.protocol.https.HttpsURLConnectionImpl.getOutputStream(Unknown Source)

        at com.vmware.vim25.ws.WSClient.post(WSClient.java:216)

        at com.vmware.vim25.ws.WSClient.invoke(WSClient.java:133)

        ... 11 more

2016-08-24 12:30:33,009 INFO  [Thread-7]-vi.ViJavaServiceInstanceProviderImpl: Retry ViJava ServiceInstance Acquisition In 5 Seconds...

17 Replies
virtualg_uk
Leadership
Leadership

Could you get it to boot up fully? Then login to the VDP appliance via https://VDP_IP/vdp-configure

and check all configuration, specifically DNS etc and report back


Graham | User Moderator | https://virtualg.uk
Reply
0 Kudos
jkhardy95
Contributor
Contributor

Yes, it boots up fully and I can log back into the vdp-configure webpage, but it's not in maintenance mode, it's still in configuration mode.  Connection tests are successful.

Capture.JPG

Reply
0 Kudos
virtualg_uk
Leadership
Leadership

Let's also verify the vCenter Certificate and move through the configuration.


Graham | User Moderator | https://virtualg.uk
Reply
0 Kudos
jkhardy95
Contributor
Contributor

I opened up a ticket with VMWare and got the following response.  I verified this is indeed the error in the mcserver.out log file.  I updated the mc perl module and then ran /usr/local/avamarclient/etc/registervcenter.sh to register the VDP appliance with vCenter and that was successful and I can log into the VDP appliance via the vSphere Web Client.  I can create new backup jobs but they fail...still working with VMWare on it.

VMWARE did a piss-poor job of regression testing vCenter 5.5 U3e with old VDP appliances.  If they would've they would've seen this problem!!!

Knowledge Base Article: 000487555

MCS cannot connect to vcenter due to Unsupported curve: 1.2.840.10045.3.1.7 (000487555)

Version: 4 Article Type: Break Fix Audience: Level 30 = Customers Last Published: Tue Aug 16 21:34:49 GMT 2016 Summary:

Issue:

Mcs cannot add or reconnect to vcenter due to Jsafe 5.0/Java8/Vcenter certain certificate/server compatibility

mcserver.out shows the following java stack trace

Started
Caught Exception : Exception : org.apache.axis.AxisFault Message : ; nested exception is:

javax.net.ssl.SSLHandshakeException: Unsupported curve: 1.2.840.10045.3.1.7 StackTrace : AxisFault

faultCode: {http://schemas.xmlsoap.org/soap/envelope/}Server.userException faultSubcode:
faultString: javax.net.ssl.SSLHandshakeException: Unsupported curve: 1.2.840.10045.3.1.7 faultActor:
faultNode:
faultDetail:

{http://xml.apache.org/axis/}stackTrace:javax.net.ssl.SSLHandshakeException: Unsupported curve: 1.2.840.10045.3.1.7

Cause: Interoperability between some between Java 8 ( used in 7.2.1+ avamar ) , EMC RSA java library called JSAFE/Bsafe 5.0 and certain vcenter server SSL ciphers settings in vcenter 5.5 U3e and Vcenter 6.5 beta.


Change: Either of the following causes can trigger this issue.
1. Upgraded vcenter to 5.5U3e/ vcenter 6.5 beta
2. Upgrade to avamar 7.2 SP1/7.3 , or MCS hotfix 250666 for 7.1 SP2, mcs hotfix 2555641 for 7.2.0 3. Adding a new vcenter running 5.5U3e/ 6.5 beta or an existing avamar enviroment


Resolution: Short term solution.

Removed RSA BSAFE java library and use the Java JCE provider library instead, this should only be considered a short term workaround as emc preferred library is RSA.

1. edit the file /usr/local/avamar/lib/mcsutils.pm
2. Add the line to the $prefs variable in the mc perl module (note the leading . and blank space before the ending quote): . "-Dsecurity.provider.rsa.JsafeJCE.position=last "

before:
my $prefs = "-Djava.util.logging.config.file=$mcsvar::lib_dir/mcserver_logging.properties "

. "-Djava.security.egd=file:/dev/./urandom "

. "-Djava.io.tmpdir=$mcsvar::tmp_dir "
. "-Djava.util.prefs.PreferencesFactory=com.avamar.mc.util.MCServerPreferencesFactory "
. "-Djavax.xml.parsers.DocumentBuilderFactory=org.apache.xerces.jaxp.DocumentBuilderFactoryImpl " . "-Djavax.net.ssl.keyStore=" . MCServer::get( "rmi_ssl_keystore" ) ." "
. "-Djavax.net.ssl.trustStore=" . MCServer::get( "rmi_ssl_keystore" ) ." "
. "-Dfile.encoding=UTF-8 "
. "-Dlog4j.configuration=file://$mcsvar::lib_dir/log4j.properties "; # vmware/axis

after:
my $prefs = "-Djava.util.logging.config.file=$mcsvar::lib_dir/mcserver_logging.properties "

. "-Djava.security.egd=file:/dev/./urandom "
. "-Djava.io.tmpdir=$mcsvar::tmp_dir "
. "-Djava.util.prefs.PreferencesFactory=com.avamar.mc.util.MCServerPreferencesFactory "
. "-Djavax.xml.parsers.DocumentBuilderFactory=org.apache.xerces.jaxp.DocumentBuilderFactoryImpl " . "-Djavax.net.ssl.keyStore=" . MCServer::get( "rmi_ssl_keystore" ) ." "
. "-Djavax.net.ssl.trustStore=" . MCServer::get( "rmi_ssl_keystore" ) ." "
. "-Dfile.encoding=UTF-8 "
. "-Dsecurity.provider.rsa.JsafeJCE.position=last "
. "-Dlog4j.configuration=file://$mcsvar::lib_dir/log4j.properties "; # vmware/axis


3. Restart mcs service: mcserver.sh --restart command.

Long term solution and prefered solution:  A long term fix for other avamar releases is being discussed with engineering

didi2008
Contributor
Contributor

I tried this, but the error is still there. The mcserver.sh --restart didn't work (after su - admin, too).

When should I make the changes? After the first start or when it hangs at 70%?

I tried to make the change after the first start of the VDP. Then the configuration hangs at 35%.

Reply
0 Kudos
lhromadka
Contributor
Contributor

I have the same problem with VDP 5.8.4 and vCenter 5.5.0 Update 3e.


Initial VDP configuration hangs on 70%.


/usr/local/avamar/var/vdr/server_logs/vdr-server.log:


2016-09-03 08:21:07,192 WARN  [Thread-12]-vi.VCenterServiceImpl: No VCenter found in MC root domain

2016-09-03 08:21:07,217 INFO  [Thread-12]-vi.ViJavaServiceInstanceProviderImpl: vcenter-ignore-cert ? true

2016-09-03 08:21:07,272 ERROR [Thread-12]-vi.ViJavaServiceInstanceProviderImpl: Failed To Create ViJava ServiceInstance

java.rmi.RemoteException: VI SDK invoke exception:java.lang.IllegalArgumentException: protocol = https host = null

  at com.vmware.vim25.ws.WSClient.invoke(WSClient.java:219)

  at com.vmware.vim25.ws.WSClient.invoke(WSClient.java:144)

  at com.vmware.vim25.ws.VimStub.retrieveServiceContent(VimStub.java:1480)

  at com.vmware.vim25.mo.ServiceInstance.<init>(ServiceInstance.java:111)

  at com.vmware.vim25.mo.ServiceInstance.<init>(ServiceInstance.java:95)

  at com.emc.vdp2.common.vi.ViJavaServiceInstanceProviderImpl.createViJavaServiceInstance(ViJavaServiceInstanceProviderImpl.java:252)

  at com.emc.vdp2.common.vi.ViJavaServiceInstanceProviderImpl.createViJavaServiceInstance(ViJavaServiceInstanceProviderImpl.java:150)

  at com.emc.vdp2.common.vi.ViJavaServiceInstanceProviderImpl.createViJavaServiceInstance(ViJavaServiceInstanceProviderImpl.java:92)

  at com.emc.vdp2.common.vi.ViJavaServiceInstanceProviderImpl.getViJavaServiceInstance(ViJavaServiceInstanceProviderImpl.java:70)

  at com.emc.vdp2.common.vi.ViJavaServiceInstanceProviderImpl.waitForViJavaServiceInstance(ViJavaServiceInstanceProviderImpl.java:166)

  at com.emc.vdp2.server.VDRServletLifeCycleListener$1.run(VDRServletLifeCycleListener.java:73)

  at java.lang.Thread.run(Unknown Source)

/space/avamar/var/mc/server_log/mcserver.out:

=== BEGIN === check.mcs (prestart)

check.mcs                        passed

=== PASS === check.mcs PASSED OVERALL (prestart)

Starting Administrator Server at: Fri Sep  2 23:17:05 PDT 2016

Starting Administrator Server...

Warning:  org.apache.xerces.jaxp.SAXParserImpl$JAXPSAXParser: Property 'http://www.oracle.com/xml/jaxp/properties/entityExpansionLimit' is not recognized.

Compiler warnings:

  WARNING:  'org.apache.xerces.jaxp.SAXParserImpl: Property 'http://javax.xml.XMLConstants/property/accessExternalDTD' is not recognized.'

2016-09-02 23:17:37.583:INFO::Logging to STDERR via org.mortbay.log.StdErrLog

2016-09-02 23:17:37.702:INFO::jetty-6.1.23

2016-09-02 23:17:37.760:INFO::Extract lib/mcsdk-axis2.war to /usr/local/avamar/var/mc/server_tmp/Jetty_0_0_0_0_9443_mcsdk.axis2.war____.xz4n3v/webapp

2016-09-02 23:17:44.819:INFO::Started SslSocketConnector@0.0.0.0:9443

Administrator Server started.

Started

Caught Exception :  Exception : org.apache.axis.AxisFault Message : ; nested exception is:

  javax.net.ssl.SSLHandshakeException: Unsupported curve: 1.2.840.10045.3.1.7 StackTrace :

AxisFault

faultCode: {http://schemas.xmlsoap.org/soap/envelope/}Server.userException

faultSubcode:

faultString: javax.net.ssl.SSLHandshakeException: Unsupported curve: 1.2.840.10045.3.1.7

faultActor:

faultNode:

faultDetail:

  {http://xml.apache.org/axis/}stackTrace:javax.net.ssl.SSLHandshakeException: Unsupported curve: 1.2.840.10045.3.1.7

  at sun.security.ssl.HandshakeMessage$ECDH_ServerKeyExchange.&lt;init&gt;(Unknown Source)

  at sun.security.ssl.ClientHandshaker.processMessage(Unknown Source)



I updated the /usr/local/avamar/lib/mcsutils.pm. I add then line with JsafeJCE.position=last.

admin@vmdp1:~/>: mcserver.sh --restart:

=== BEGIN === check.mcs (poststart)

check.mcs                        passed

=== PASS === check.mcs PASSED OVERALL (poststart)

--restart will restart the Administrator Server.

Do you want to proceed with the restart Y/N? [Y]: y

Administrator Server shutdown initiated.

Stopping Administrator Server...

Administrator Server stopped.

Database server is running...

INFO: Starting messaging service.

INFO: Started messaging service.

=== BEGIN === check.mcs (prestart)

check.mcs                        passed

=== PASS === check.mcs PASSED OVERALL (prestart)

Starting Administrator Server at: Sat Sep  3 10:53:58 CEST 2016

Starting Administrator Server...

Warning:  org.apache.xerces.jaxp.SAXParserImpl$JAXPSAXParser: Property 'http://www.oracle.com/xml/jaxp/properties/entityExpansionLimit' is not recognized.

Compiler warnings:

  WARNING:  'org.apache.xerces.jaxp.SAXParserImpl: Property 'http://javax.xml.XMLConstants/property/accessExternalDTD' is not recognized.'

2016-09-03 10:54:30.301:INFO::Logging to STDERR via org.mortbay.log.StdErrLog

2016-09-03 10:54:30.384:INFO::jetty-6.1.23

2016-09-03 10:54:30.425:INFO::Extract lib/mcsdk-axis2.war to /usr/local/avamar/var/mc/server_tmp/Jetty_0_0_0_0_9443_mcsdk.axis2.war____.xz4n3v/webapp

2016-09-03 10:54:36.621:INFO::Started SslSocketConnector@0.0.0.0:9443

Administrator Server started.

INFO: Starting Data Domain SNMP Manager....

INFO: Connecting to MCS Server: vmdp1.gordic.cz at port: 7778...

INFO: Successfully connected to MCS Server: vmdp1.gordic.cz at port: 7778.

INFO: No trap listeners were started, Data Domain SNMP Manager didn't start.

Page https://....vdp....:8543/vdp-configure/ is still in configuration mode.


Page https://....vcenter..../mob/?moid=ExtensionManager show only com.vmware.vdp2.config extension. The com.vmware.vdp2 extension missing.


The VDP VM are still displaing task "com.vmware.vdp2.config.initialconfig".


What to do next? Restart VDP VM with uncomplete configuration?


Reply
0 Kudos
jarik_boom
Contributor
Contributor

I also have the same problem with VDP 5.8.4 and vCenter 5.5.0 Update 3e (5.5.0.30500 Build 4180648). VDP configuration stuck at 70%.

The problem was solved by adding a line in /usr/local/avamar/lib/mcsutils.pm

. "-Dsecurity.provider.rsa.JsafeJCE.position=last "

before the first start of VDP configuration wizard (immediately after deploy OVF template)


Thank you, jkhardy95.

komanek
Enthusiast
Enthusiast

Hello,

thank you very much for this thread, it helped me a lot. After vCenter upgrade to 5.5u3e, my VDP 6.0.3 appliance was unusable. As far as there is nothing regarding this problem in vCenter or VDP release notes and VDP 6.0.x is officially supported with vCenter 5.5u3, I tried many obscure things, but with no luck. I can also confirm that the same issue appears for VDP 5.8.x after vCenter upgrade.

So it was great to finally find the solution here. It helped me to reconnect VDP 6.0.3 to the upgrade vCenter at one site (I edited the library file and restarted service as subbested) and also to deply a new instance of VDP 6.0.4 at my second site (I edited the library file after OVA deployment, before I entered the configuration wizard).

Still, I have one question regarding in-place upgrade of VDP 6.0.3 to 6.0.4. Is there a working scenario in this situation or shloudl I wait for 6.0.5 with official patch (if it ever will be released) ? It still could be tricky because of kernel hotfix, which needs to be installed before the upgrade to 6.0.4, but which is not part of the upgrade ISO 😕

Thanks again and best regards,

David

Reply
0 Kudos
VictorWangBegin
Contributor
Contributor

Hi, Jarik and other kind helpers,

Can somebody explain in details how to update the file?

/usr/local/avamar/lib/mcsutils.pm

I tried to install VPD 6.1.1 and VDP6.1.2 on ESXi 5.5 (latest version/patch), both got stucked at 70% after initial setup. I am using vCenter version:

VMware vCenter Server Appliance
Update to version 5.5.0 Update 3e

5.5.0.30500 Build 4180648

Is this mcsutils.pm file located on VDP Appliance? or on vCenter or ESXi host? It will be too much for me to add a line to a file on VDP Appliance's disk, without powering it on. Smiley Happy

The reason why I want to install a new VDP is that my VDP6.1.1 suddenly stopped working, with "Log file blank" error on all backup jobs. I suspect that this is related with my recent update of vCenter 5.5.

Thank you very much.

Victor Wang

Reply
0 Kudos
VictorWangBegin
Contributor
Contributor

I believe that I figured it out:

1. Deploy OVF, I am using 6.1.2.

2. Power it ON, but do not run VDP config.

3. Logon to console with user "root" password "changeme".

4. Go to /usr/local/avamar/lib/ ,use VIM to edit mcsutil.pm, save it. You need VIM command ":w!" to save a read only file.

5. Restart service "mcserver.sh --restart". In my case I got an error "ERROR: flag needs two dashes: "-restart', stopped at mcflags.pm line 380. As a precaution, I reboot this VM from vCenter's page.

6. After reboot, wait for 5 minutes, go to VDP config page: https://YOURVDP IP ADDRESS:8543/vdp-configure

7. Start config it normally.

Hope this will help you. Thanks.

VictorWangBegin
Contributor
Contributor

72 Hours after fresh installation, the vm send me an email:

Start the Backup Scheduler by using the vSphere Data Protection Configuration utility.

Tried to connect to VDP-Config but stucked with a grey bar with one block on left (inside bar). Tried to access it from vCenter, it seems still functioning. I rebooted this VM, hopfully this is just random problem.

The VDP did backed up several VMs, I am trying to restore one VM now. Will keep posting.  After several hours: The restore of my test VM was success. I will keep an eye on that backup scheduler error.

Reply
0 Kudos
ACTScott
Contributor
Contributor

This solution worked perfectly for me as well but I too am getting the warnings regarding scheduler service even though the backups are working as scheduled. Could be a false alarm. I have VMware calling me back today and I will post back anything I hear back from them.

Reply
0 Kudos
GMZSE
Contributor
Contributor

The extra line of code made my appliance configure to 100%, however I can't connect to any VDP appliances from the Web Client. After clicking 'Connect', it just hangs and nothing happens.

  • vCenter: 5.5 U3e (Build 4180647)
  • Two older appliances, version 5.8.1.7. These worked after upgrading vCenter.
  • One newer appliance, version 5.8.4.6 . This replaced a previous appliance (version 5.8.3.13) that stopped working after the vCenter upgrade.


Has anyone run into this issue yet? What was the resolution?

Reply
0 Kudos
itttam
Contributor
Contributor

. "-Dsecurity.provider.rsa.JsafeJCE.position=last "

Where should this line go in the mcsutils.pm?

thanks

NM, found where it goes after rereading the thread. Giving it another go. Hopefully no longer stuck at 70%

vCSA version 5.5.0 Update 3e

5.5.0.30500 Build 4180648


VDP 6.1.2.19

Reply
0 Kudos
user3158
Contributor
Contributor

This is exactly same problem I'm facing.

1. Deploy OVF and perform regular setup - Configuring hangs at 70%

2. Delete the VDP VM and start deployment again, then add the line of code mentioned above and configure as normal. This makes it to COnfigure successfullly.

3. Login to vSphere web client, new menu item appears - vSphere Data Protection 5.8. When I click it, there is a dropbox with only 1 option (the VDP appliance) and Connect button. Pressing Connect hangs the vSphere client.

vCenter is 5.5 U3e, VDP is 5.8.4

Any help appreciated.

update: I have deployed a second VDP appliance v.6.1.2, nothing changed. A fix is still required that adds line of code to mcsutils.pm and then vSphere client hangs on Connect button. It seems that VDP is completely not operational. Did someone manage to make it work? Any version ever?

Reply
0 Kudos
littlefat
Contributor
Contributor

This issue occurs because the vSphere Data Protection MCS subsystem cannot communicate with the vCenter Server due to JsafeJCE 5.0 / Java 8 and vCenter Server certificate compatibility.


https://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=21468...

Little@FAT
Reply
0 Kudos
mobcdi
Enthusiast
Enthusiast

The kb doesn't mention VDP 5.8.4 is it possible to apply the hotfix mentioned to a 5.8.4 vdp appliance or do I need to manually evaluate the hotfix and attempt to retro-fit it ?

Reply
0 Kudos