VDP stop working after domain administrator password change.

[robertomignucci]

We recently changed domain administrator password and stop the domain administrator permission to the vCentre. Looks like now the VDP machine stop working.

The symptoms are: cannot access VDP via web, only ssh.

I looked trough the vdr-server.log and there are a lot of these errors:

2015-04-15 14:30:05,470 INFO  [com.emc.vdr.server.VDRServer$1]-server.ConnectionService: Trying to establish connection with vCenter.

2015-04-15 14:30:05,471 INFO  [com.emc.vdr.server.VDRServer$1]-server.VDRExternalServerAccessConfig: loading default values forcefully: true

2015-04-15 14:30:05,471 INFO  [com.emc.vdr.server.VDRServer$1]-server.VDRExternalServerAccessConfig: Get mc access...

2015-04-15 14:30:05,480 INFO  [com.emc.vdr.server.VDRServer$1]-service.AdapterUtils: MCS Web Services URL: https://hydra.ecom90.co.uk:9443/services/mcService  MCUserId="MCUser"  MCUserPswd="*****************************"

2015-04-15 14:30:05,492 INFO  [com.emc.vdr.server.VDRServer$1]-httpclient.HttpMethodDirector: I/O exception (java.net.ConnectException) caught when processing request: Connection refused

2015-04-15 14:30:05,492 INFO  [com.emc.vdr.server.VDRServer$1]-httpclient.HttpMethodDirector: Retrying request

2015-04-15 14:30:05,513 INFO  [com.emc.vdr.server.VDRServer$1]-httpclient.HttpMethodDirector: I/O exception (java.net.ConnectException) caught when processing request: Connection refused

2015-04-15 14:30:05,513 INFO  [com.emc.vdr.server.VDRServer$1]-httpclient.HttpMethodDirector: Retrying request

2015-04-15 14:30:05,514 INFO  [com.emc.vdr.server.VDRServer$1]-httpclient.HttpMethodDirector: I/O exception (java.net.ConnectException) caught when processing request: Connection refused

2015-04-15 14:30:05,514 INFO  [com.emc.vdr.server.VDRServer$1]-httpclient.HttpMethodDirector: Retrying request

2015-04-15 14:30:05,514 INFO  [com.emc.vdr.server.VDRServer$1]-http.HTTPSender: Unable to sendViaPost to url[https://hydra.ecom90.co.uk:9443/services/mcService?wsdl]

java.net.ConnectException: Connection refused

  at java.net.PlainSocketImpl.socketConnect(Native Method)

  at java.net.PlainSocketImpl.doConnect(Unknown Source)

  at java.net.PlainSocketImpl.connectToAddress(Unknown Source)

  at java.net.PlainSocketImpl.connect(Unknown Source)

  at java.net.SocksSocketImpl.connect(Unknown Source)

  at java.net.Socket.connect(Unknown Source)

  at com.sun.net.ssl.internal.ssl.SSLSocketImpl.connect(Unknown Source)

  at com.emc.avamar.mcsdk.adapter.service.TrustAllSSLProtocolSocketFactory.createSocket(TrustAllSSLProtocolSocketFactory.java:128)

  at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707)

  at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361)

  at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387)

  at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171)

  at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397)

  at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:346)

  at org.apache.axis2.transport.http.AbstractHTTPSender.executeMethod(AbstractHTTPSender.java:542)

  at org.apache.axis2.transport.http.HTTPSender.sendViaPost(HTTPSender.java:199)

  at org.apache.axis2.transport.http.HTTPSender.send(HTTPSender.java:76)

  at org.apache.axis2.transport.http.CommonsHTTPTransportSender.writeMessageWithCommons(CommonsHTTPTransportSender.java:400)

  at org.apache.axis2.transport.http.CommonsHTTPTransportSender.invoke(CommonsHTTPTransportSender.java:225)

  at org.apache.axis2.engine.AxisEngine.send(AxisEngine.java:435)

  at org.apache.axis2.description.OutInAxisOperationClient.send(OutInAxisOperation.java:402)

  at org.apache.axis2.description.OutInAxisOperationClient.executeImpl(OutInAxisOperation.java:229)

  at org.apache.axis2.client.OperationClient.execute(OperationClient.java:165)

  at com.avamar.mc.sdk.client.McServiceStub.login(McServiceStub.java:5084)

  at com.avamar.mc.sdk.client.McServiceClient.login(McServiceClient.java:119)

  at com.avamar.mc.sdk.client.McServiceClient.connect(McServiceClient.java:105)

  at com.emc.avamar.mcsdk.adapter.service.ServiceInstance.connect(ServiceInstance.java:313)

  at com.emc.avamar.mcsdk.adapter.service.ServiceInstance.<init>(ServiceInstance.java:225)

  at com.emc.avamar.mcsdk.adapter.service.ServiceInstance.<init>(ServiceInstance.java:187)

  at com.emc.vdr.server.VDRExternalServerAccessConfig.getFromAdapterDefault(VDRExternalServerAccessConfig.java:67)

  at com.emc.vdr.server.VDRExternalServerAccessConfig.getFromPropsDefault(VDRExternalServerAccessConfig.java:126)

  at com.emc.vdr.server.ConnectionService.run(ConnectionService.java:40)

  at java.lang.Thread.run(Unknown Source)

2015-04-15 14:30:05,514 ERROR [com.emc.vdr.server.VDRServer$1]-service.AdapterUtils: ServiceInstance failed to establish connection

org.apache.axis2.AxisFault: Connection refused

  at org.apache.axis2.AxisFault.makeFault(AxisFault.java:430)

  at org.apache.axis2.transport.http.HTTPSender.sendViaPost(HTTPSender.java:203)

  at org.apache.axis2.transport.http.HTTPSender.send(HTTPSender.java:76)

  at org.apache.axis2.transport.http.CommonsHTTPTransportSender.writeMessageWithCommons(CommonsHTTPTransportSender.java:400)

  at org.apache.axis2.transport.http.CommonsHTTPTransportSender.invoke(CommonsHTTPTransportSender.java:225)

  at org.apache.axis2.engine.AxisEngine.send(AxisEngine.java:435)

  at org.apache.axis2.description.OutInAxisOperationClient.send(OutInAxisOperation.java:402)

  at org.apache.axis2.description.OutInAxisOperationClient.executeImpl(OutInAxisOperation.java:229)

  at org.apache.axis2.client.OperationClient.execute(OperationClient.java:165)

  at com.avamar.mc.sdk.client.McServiceStub.login(McServiceStub.java:5084)

  at com.avamar.mc.sdk.client.McServiceClient.login(McServiceClient.java:119)

  at com.avamar.mc.sdk.client.McServiceClient.connect(McServiceClient.java:105)

  at com.emc.avamar.mcsdk.adapter.service.ServiceInstance.connect(ServiceInstance.java:313)

  at com.emc.avamar.mcsdk.adapter.service.ServiceInstance.<init>(ServiceInstance.java:225)

  at com.emc.avamar.mcsdk.adapter.service.ServiceInstance.<init>(ServiceInstance.java:187)

  at com.emc.vdr.server.VDRExternalServerAccessConfig.getFromAdapterDefault(VDRExternalServerAccessConfig.java:67)

  at com.emc.vdr.server.VDRExternalServerAccessConfig.getFromPropsDefault(VDRExternalServerAccessConfig.java:126)

  at com.emc.vdr.server.ConnectionService.run(ConnectionService.java:40)

  at java.lang.Thread.run(Unknown Source)

Caused by: java.net.ConnectException: Connection refused

  at java.net.PlainSocketImpl.socketConnect(Native Method)

  at java.net.PlainSocketImpl.doConnect(Unknown Source)

  at java.net.PlainSocketImpl.connectToAddress(Unknown Source)

  at java.net.PlainSocketImpl.connect(Unknown Source)

  at java.net.SocksSocketImpl.connect(Unknown Source)

  at java.net.Socket.connect(Unknown Source)

  at com.sun.net.ssl.internal.ssl.SSLSocketImpl.connect(Unknown Source)

  at com.emc.avamar.mcsdk.adapter.service.TrustAllSSLProtocolSocketFactory.createSocket(TrustAllSSLProtocolSocketFactory.java:128)

  at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707)

  at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361)

  at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387)

  at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171)

  at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397)

  at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:346)

  at org.apache.axis2.transport.http.AbstractHTTPSender.executeMethod(AbstractHTTPSender.java:542)

  at org.apache.axis2.transport.http.HTTPSender.sendViaPost(HTTPSender.java:199)

  ... 17 more

2015-04-15 14:30:05,515 ERROR [com.emc.vdr.server.VDRServer$1]-service.AdapterUtils: ServiceInstance failed to obtain domain "/"

java.lang.NullPointerException

  at com.emc.avamar.mcsdk.adapter.service.ServiceInstance.getDomain(ServiceInstance.java:496)

  at com.emc.avamar.mcsdk.adapter.service.ServiceInstance.getRootDomain(ServiceInstance.java:462)

  at com.emc.vdr.server.VDRServer.getVCenterClient(VDRServer.java:402)

  at com.emc.vdr.server.VDRExternalServerAccessConfig.getFromAdapterDefault(VDRExternalServerAccessConfig.java:86)

  at com.emc.vdr.server.VDRExternalServerAccessConfig.getFromPropsDefault(VDRExternalServerAccessConfig.java:126)

  at com.emc.vdr.server.ConnectionService.run(ConnectionService.java:40)

  at java.lang.Thread.run(Unknown Source)

2015-04-15 14:30:05,515 ERROR [com.emc.vdr.server.VDRServer$1]-server.VDRServer: VDRServer.getVCenterClient cannot obtain VCenterClient

java.lang.NullPointerException

  at com.emc.vdr.server.VDRServer.getVCenterClient(VDRServer.java:403)

  at com.emc.vdr.server.VDRExternalServerAccessConfig.getFromAdapterDefault(VDRExternalServerAccessConfig.java:86)

  at com.emc.vdr.server.VDRExternalServerAccessConfig.getFromPropsDefault(VDRExternalServerAccessConfig.java:126)

  at com.emc.vdr.server.ConnectionService.run(ConnectionService.java:40)

  at java.lang.Thread.run(Unknown Source)

2015-04-15 14:30:05,515 ERROR [com.emc.vdr.server.VDRServer$1]-server.VDRExternalServerAccessConfig: Unable to fetch the vCenter credential from web service: null

2015-04-15 14:30:05,515 ERROR [com.emc.vdr.server.VDRServer$1]-server.ConnectionService: Unable to get the vi access: no protocol:

2015-04-15 14:30:05,515 INFO  [com.emc.vdr.server.VDRServer$1]-server.ConnectionService: Retry after 5 seconds ...

In a desperate attempt I created a new VDP machine and got the same symptoms

I tried the dpnctl start all but it errors (see attached file).

Those machine worked until few days ago and now they gone!

Does anyone know how to resolve these issue?

Thanks in advance.

Rob

P.S.

I am using vCenter 5.1 VDP 5.1

[LeslieBNS9]

What version of VDP are you running? Did you update the vCenter registration info on VDP after changing the pass? If not heres how...

Go to your VDP configuration page..

https://VDPIP:8543/vdp-configure/

On VDP 5.8 and 6.0 you click the sprocket and select vCenter Registration. On the next screen you can update your vcenter user/pass.

[robertomignucci]

Thanks Leslie,

I wish I could do that, the tomcat server is not running or is not working I tried to restart it with no joy (emwebapp.sh --restart).

[LeslieBNS9]

What version of VDP is it?

[robertomignucci]

VDP 5.1

[LeslieBNS9]

Try running this script on the VDP appliance.... /usr/local/avamarclient/etc/registervcenter.sh

[robertomignucci]

Hi Leslie,

Thank you for your help I manage to resuscitate one of the VDP and access it via the vCenter (but I cannot still access them via the web admin: 8543/vdp-configure/).

On the machine that still not working when I run the /usr/local/avamarclient/etc/registervcenter.sh I get allot of:

Identity added: /home/dpn/.ssh/dpnid (/home/dpn/.ssh/dpnid)

dpnctl: INFO: MCS status: down.

dpnctl: INFO: [see log file "/usr/local/avamar/var/log/dpnctl.log"]

.... Waiting for MCS to start - Fri Apr 17 08:41:56 BST 2015

Identity added: /home/dpn/.ssh/dpnid (/home/dpn/.ssh/dpnid)

dpnctl: INFO: MCS status: down.

dpnctl: INFO: [see log file "/usr/local/avamar/var/log/dpnctl.log"]

.... Waiting for MCS to start - Fri Apr 17 08:42:04 BST 2015

Identity added: /home/dpn/.ssh/dpnid (/home/dpn/.ssh/dpnid)

dpnctl: INFO: MCS status: down.

But it does not eventually ask me to enter the credential to connect to the vCenter like the other one did?

[LeslieBNS9]

Upload the /usr/local/avamar/var/log/dpnctl.log again. I'd like to see what errors it's throwing.

[robertomignucci]

Hi Leslie,

Here is the log (see attached).

I start the registervcentre.sh first and got the log soon after.

[LeslieBNS9]

Hmm.....

You can try running this command.

mccli client add --type=vcenter --virtual-center-username=$VCENTER_USER --virtual-center-password=$VCENTER_PASSWORD --name=$VCENTER_ADDRESS

Then restart all the services...

su - admin

mcserver.sh --restart

exit

emwebapp.sh --restart

If that doesn't work you might be better off deploying a new VDP appliance. Last time I had a problem like this I was able to get VDP back up and running, but it took a lot of effort and still acted funny. I ended up rolling it's checkpoint back and that resolved my issues completely.

[robertomignucci]

Hi Leslie,

Thank you so much, for all you did so far.

I run the command you sent:

root@hydra:~/#: mccli client add --type=vcenter --virtual-center-username=$VCENTER_USER --virtual-center-password=$VCENTER_PASSWORD --name=$VCENTER_ADDRESS

1,23996,CLI failed to connect to MCS.

I change the $ values with the actual credentials and ipaddress!

then I run:

admin@hydra:~/>: mcserver.sh --restart

The Administrator Server is not running.

=== BEGIN === check.mcs (prestart)

ERROR: check.mcs: [hydra] Executing: /usr/local/avamar/bin/avmaint nodelist CHECK FAILED: ERROR: avmaint: nodelist: cannot connect to server hydra at 192.168.212.115:27000

ERROR: check.mcs: [hydra] Executing: /usr/local/avamar/bin/avmaint nodelist CHECK FAILED: ERROR: avmaint: nodelist: cannot connect to server hydra at 192.168.212.115:27000

+++ WARN +++ check.mcs: [hydra] checking that the dispatcher is not suspended CHECK FAILED: ('' matching /false/)

ERROR: check.mcs: [hydra] Checking tar command CHECK FAILED: tar: /usr/local/avamar/var/mc/server_data/prefs/mcserver.xml: Cannot utime: Operation not permitted

check.mcs                        failed

*** FAIL *** check.mcs FAILED OVERALL (prestart)

--restart will restart the Administrator Server.

Do you want to proceed with the restart Y/N? [Y]:

Database server is running...

=== BEGIN === check.mcs (prestart)

ERROR: check.mcs: [hydra] Executing: /usr/local/avamar/bin/avmaint nodelist CHECK FAILED: ERROR: avmaint: nodelist: cannot connect to server hydra at 192.168.212.115:27000

And for good mesure:

root@hydra:~/#: dpnctl start all

Identity added: /home/dpn/.ssh/dpnid (/home/dpn/.ssh/dpnid)

dpnctl: INFO: Checking that gsan was shut down cleanly...

  -  -  -  -  -  -  -  -  -  -  -  -  -  -  -  -  -  -  -  -

Here are the most recent validated and non-validated checkpoints:

  Tue Mar 31 09:13:26 2015 UTC Validated(type=rolling)

  Wed Apr  8 09:34:00 2015 UTC Not Validated

A rollback is recommended: the gsan was not shut down cleanly.

The choices are as follows:

  1   roll back to the most recent checkpoint, whether or not validated

  2   roll back to the most recent validated checkpoint

  3   select a specific checkpoint to which to roll back

  4   restart, but do not roll back (NOT RECOMMENDED:

                                     gsan reported unclean shutdown)

  5   do not restart

  q   quit/exit

(Entering an empty (blank) line twice quits/exits.)

> 2

  -  -  -  -  -  -  -  -  -  -  -  -  -  -  -  -  -  -  -  -

You have selected this checkpoint:

  name:       cp.20150331091326

  date:       Tue Mar 31 09:13:26 2015 UTC

  validated:  yes

  age:        20 days, 5 hours

Roll back to this checkpoint?

Answering y(es)  accepts this checkpoint and initiates rollback

          n(o)   rejects this checkpoint and returns to the main menu

          q(uit) exits

y(es), n(o), q(uit/exit): y

dpnctl: INFO: Initiating rollback to "cp.20150331091326" with gsan restart (this may take some time)...

dpnctl: INFO: To monitor progress, run in another window: tail -f /tmp/dpnctl-gsan-restart-output-26759

dpnctl: ERROR: error return from "/usr/bin/yes no | /usr/local/avamar/bin/rollback.dpn --cptag="cp.20150331091326"" - exit status 25

dpnctl: ERROR: 2 errors seen in output of "/usr/bin/yes no | /usr/local/avamar/bin/rollback.dpn --cptag="cp.20150331091326""

dpnctl: ERROR: error return from "/usr/bin/yes no | /usr/local/avamar/bin/rollback.dpn --cptag="cp.20150331091326"" - exit status 25

dpnctl: ERROR: 2 errors seen in gsan error logs:

- - - - - - - - - - - - - - - - - - - - BEGIN

(0.0) 2015/04/20-14:51:58.38620 {0.0} [licensevalidator:114]  ERROR: <0948> license grace period expired 521 days ago (Fri Nov 15 10:35:49 2013)

(0.0) 2015/04/20-14:51:58.38631 {0.0} [licensevalidator:114]  FATAL ERROR: <0001> licensevalidator::body no valid license found

- - - - - - - - - - - - - - - - - - - - END

dpnctl: WARNING: 3 warnings seen in gsan error logs:

- - - - - - - - - - - - - - - - - - - - BEGIN

(0.0) 2015/04/20-14:51:34.24672 {0.0} [log-messages:109]  WARN: <0002> kernel warning: Apr  7 15:59:26 hydra sshd[18133]: error: PAM: Authentication failure for root from 192.168.1.6

(0.0) 2015/04/20-14:51:34.66543 {0.0} [log-messages:109]  WARN: <0002> kernel warning: Apr 15 12:52:58 hydra sshd[22328]: error: PAM: Authentication failure for root from 192.168.212.19

(0.0) 2015/04/20-14:51:34.84061 {0.0} [log-messages:109]  WARN: <0002> kernel warning: Apr 17 08:39:15 hydra sshd[17894]: error: PAM: Authentication failure for root from 192.168.212.19

- - - - - - - - - - - - - - - - - - - - END

dpnctl: ERROR: gsan is not up or is exhibiting errors - not performing MCS restore

dpnctl: ERROR: MCS restore did not succeed, so not restarting MCS

dpnctl: ERROR: cannot resume backup scheduler: gsan is down and MCS is down

dpnctl: ERROR: gsan is not up, so not starting axionfs

dpnctl: INFO: [see log file "/usr/local/avamar/var/log/dpnctl.log"]

root@hydra:~/#:

I am beginning to think to give up on this machine!

Could I ask if you know why I cannot get the web ui working on the VDP machines? I have one of the machine working again and I can see it and admin it in the vcenter but still cannot see the web ui(vdp-configure)?  

Thanks again

Roberto

[LeslieBNS9]

Yea I would recommend a rollback or rebuild. Any backups that have happened since the point you rollback to will be lost.

I honestly am not sure why the web configure area wouldn't be working for you.

I wish I could be of more assistance, but you have me pretty stumped.

[soportetp]

Do you know if this task affect virtual machines state?:

- roll back to the most recent validated checkpoint