VDP appliance is not responding issue

[PeterBlatherwic]

Hi,

We are seeing troubles accessing VDR from the Web Client.  When go to the vSphere Data Protection plugin tab, and click [Connect] for our VDP instance, getting the following error:

     ERROR:

     The VDP appliance is not responding. Please try your request again.

     Would you like to be directed to the VDP configuration screen to troubleshoot the issue?

Reconfiguring VDP trough vdp-configure appears to connect properly to the vCenter -- [Test Connection] returns success. Does not fix the issue however.

Setup:

- Sphere 5.1.0

- VDP 5.1.1 0.5 TB appliance

Don't know if it is related, but in vdr-server.log we are seeing the below pattern repeating over and over. 

2013-08-28 13:54:50,476 INFO  [com.emc.vdr.server.VDRServer$1]-server.ConnectionService: Retry after 5 seconds ...

2013-08-28 13:54:55,477 INFO  [com.emc.vdr.server.VDRServer$1]-server.ConnectionService: Trying to establish connection with vCenter.

2013-08-28 13:54:55,477 INFO  [com.emc.vdr.server.VDRServer$1]-server.VDRExternalServerAccessConfig: loading default values forcefully: true

2013-08-28 13:54:55,477 INFO  [com.emc.vdr.server.VDRServer$1]-server.VDRExternalServerAccessConfig: Get mc access...

2013-08-28 13:54:55,487 INFO  [com.emc.vdr.server.VDRServer$1]-service.AdapterUtils: MCS Web Services URL: https://vdp-02.vlab.local:9443/services/mcService  MCUserId="MCUser"  MCUserPswd="*****************************"

2013-08-28 13:54:56,655 INFO  [com.emc.vdr.server.VDRServer$1]-service.ServiceInstance: ServiceInstanceMoref desc=Service Id: urn:uuid:123EEFAE88AFBCA74D1377712498579 name=urn:uuid:123EEFAE88AFBCA74D1377712498579 value=SERVICE

2013-08-28 13:54:56,816 WARN  [com.emc.vdr.server.VDRServer$1]-server.VDRServer: VDRServer.Unable to find the vCenterClient in 10.45.99.12 domain which has VirtualMachines as sub domain

2013-08-28 13:54:56,816 ERROR [com.emc.vdr.server.VDRServer$1]-server.VDRServer: VDRServer.getVCenterClient cannot obtain VCenterClient

java.lang.NullPointerException: VCenterClient list is empty

    at com.emc.vdr.server.VDRServer.getVCenterClient(VDRServer.java:441)

    at com.emc.vdr.server.VDRExternalServerAccessConfig.getFromAdapterDefault(VDRExternalServerAccessConfig.java:86)

    at com.emc.vdr.server.VDRExternalServerAccessConfig.getFromPropsDefault(VDRExternalServerAccessConfig.java:126)

    at com.emc.vdr.server.ConnectionService.run(ConnectionService.java:40)

    at java.lang.Thread.run(Unknown Source)

2013-08-28 13:54:56,816 ERROR [com.emc.vdr.server.VDRServer$1]-server.VDRExternalServerAccessConfig: Unable to fetch the vCenter credential from web service: null

2013-08-28 13:54:56,822 ERROR [com.emc.vdr.server.VDRServer$1]-server.ConnectionService: Unable to get the vi access: no protocol:

2013-08-28 13:54:56,822 INFO  [com.emc.vdr.server.VDRServer$1]-server.ConnectionService: Retry after 5 seconds ...

BTW, yes, did see the other discussion :    But this appears to be a different issue -- we did not attempt to change any port numbers or anything like that. 

Anyone else seeing this? 

Any thoughts on what might be wrong??

Cheers,

PeterB

[GSparks]

Peter -

Try this and see if it helps:

http://kb.vmware.com/selfservice/microsites/search.do?cmd=displayKC&docType=kc&externalId=2047852&sl...

If that does not work, the next step to take is open up a putty / ssh session to the VDP.

Run command 'tail -f /usr/local/avamar/var/vdr/server_logs/vdr-server.log'

Try to connect to your VDP appliance.

In this particular log, the entries that show from connection attempts all have a thread id that starts with "[http-8543-exec-#]"

If you see an entry with that thread id, and an error, post it here.  If you see NO entry with that thread id, then it is an apache issue (KB fixes most commonly occurring issue). 

Let me know what you find.

Greg

[PeterBlatherwic]

Thanks Greg,

First trick (modifying Tomcat server.xml file) did not do the trick.

I have attached vdr-server.log file here, taken immediately follow a connect attempt from Web Client.  (I snipped top part of it out just to somewhat reduce size.) 

Yes, do see lots of [http-8543-exec lines.  Most are ERRORs with things like "cannot obtain VCenterClient : java.lang.NullPointerException: VCenterClient list is empty", null pointer exceptions and so forth.

I'm going to go ahead and guess these are not good signs of health ;^)

Any ideas??

-- PeterB

[PeterBlatherwic]

Hi all,

Just letting the Community know results here, in case others are seeing something similar.

After LOADS of off-line poking around with Greg, the issue seems to be resolved now.  Connecting to VDP happily, backups happy.

Here's the short story:

- Poking about various logs, appeared something was not working right with SSO on the vCenter.  Inclusive at this point.

- Another possibility was Tomcat issue, regarding size of authentication token.  Fix for that had no effect.

- Licensing issue eliminated; that part is fine.

- Errors appear to be coming from call to vCenter Inventory service, to ask whether it has the VDP appliance within its inventory. 

- Fixing the connect issue involved reverting vCenter to a snapshot before we installed VSA Manager.  Multiple issues had started to occur in our setup around then.  Partial success -- VDP started connecting happily, backups working on most clusters in our setup. See next.

- BUT there were several other issues related to moving the vCenter back in time more than a week (lots of orphaned VMs, and unable to perform many operation on some of our hosts).  Including, VDP backups failed for our management cluster, where vCenter, vShield etc live.  Not good. OK, ugly. 

- Undoing the revert, ie going back to before above step, VDP connects worked!  No idea why at this point.  Able to set backup plans, etc.

- However, actual backups did not get queued (not coming up in Tasks list).  That was fixed by restarting the Backup Scheduler service in VDP, and also re-synching storage.  Both issues that are apparently fixed in latest VDP. 

In the end, working now.  But root cause(s) are unclear.  VSA Manager is a likely suspect for some sort of interaction.

THANKS GREG.  Please add anything you think may help others in future.  Esp on latest release and best practices. 

Hope this helps someone some day.

PeterB