mknunez
Contributor
Contributor

BDE vSphere plugin error: Can not add network. Internal error: REST API transport layer error.

This error happening when trying to add Datastore:

build: VMware-BigDataExtensions-1.0.0.332-1315424_OVF10.ova

error log:

/opt/serengeti/logs/serengeti.log

2013 Dec 03 18:49:36,857+0000 ERROR http-8443-2| com.vmware.bdd.rest.RestResource: rest call error

com.vmware.bdd.exception.BddException: Internal error: REST API transport layer error.

        at com.vmware.bdd.exception.BddException.INTERNAL(BddException.java:77)

        at com.vmware.bdd.exception.BddException.wrapIfNeeded(BddException.java:72)

        at com.vmware.bdd.rest.RestResource.handleException(RestResource.java:603)

        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)

        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

        at java.lang.reflect.Method.invoke(Method.java:616)

...

        at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:489)

        at java.lang.Thread.run(Thread.java:679)

Caused by: com.vmware.bdd.utils.AuAssert: FAILURE

        at com.vmware.bdd.utils.AuAssert.FAILURE(AuAssert.java:28)

        at com.vmware.bdd.utils.AuAssert.check(AuAssert.java:43)

        at com.vmware.bdd.utils.AuAssert.check(AuAssert.java:50)

        at com.vmware.bdd.service.resmgmt.impl.ResourceService.refreshDatastore(ResourceService.java:196)

        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)

        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

        at java.lang.reflect.Method.invoke(Method.java:616)

...

        at sun.proxy.$Proxy40.addDatastores(Unknown Source)

        at com.vmware.bdd.rest.RestResource.addDatastore(RestResource.java:441)

        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)

        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

        at java.lang.reflect.Method.invoke(Method.java:616)

        at org.springframework.web.bind.annotation.support.HandlerMethodInvoker.invokeHandlerMethod(HandlerMethodInvoker.java:176)

        at org.springframework.web.servlet.mvc.annotation.AnnotationMethodHandlerAdapter.invokeHandlerMethod(AnnotationMethodHandlerAdapter.java:440)

        at org.springframework.web.servlet.mvc.annotation.AnnotationMethodHandlerAdapter.handle(AnnotationMethodHandlerAdapter.java:428)

        at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherSer

0 Kudos
30 Replies
mknunez
Contributor
Contributor

Good find. I fixed the issue of name resolution, uniformly for the host, vcenter, serengeti. Still didn't work. So I reinstalled serengeti because the name resolution didn't fix the next REST problem related to:

1) 2013 Dec 05 21:27:10,076+0000 ERROR main| com.vmware.bdd.web.ResourceInitializer: Resource initialization failed.

com.vmware.aurora.exception.VcException: Failed to execute vCenter Server command: sun.proxy.$Proxy101 cannot be cast to com.vmware.vim.binding.vim.ClusterComputeResource.

2) tomcat failed to generate uuid in serengeti.properties file in 5 minutes. It's probably a vc connection issue, please check /opt/serengeti/logs/serengeti.log

I've attached the new logs based on reinstall and name resolution corrections.

Thanks,

M

0 Kudos
mknunez
Contributor
Contributor

I need this fixed to do configuration, I fail again when creating a new Hadoop Cluster:

2013 Dec 05 22:39:50,272+0000 INFO http-8443-4| com.vmware.bdd.manager.ClusterManager: added automation resource pools:

2013 Dec 05 22:39:50,273+0000 INFO http-8443-4| com.vmware.bdd.aop.logging.ExceptionHandlerAspect: Aspect for exception handling

2013 Dec 05 22:39:50,273+0000 ERROR http-8443-4| com.vmware.bdd.aop.logging.ExceptionHandlerAspect: Service error

com.vmware.bdd.exception.BddException: Internal error: Service AOP.

at com.vmware.bdd.exception.BddException.INTERNAL(BddException.java:77)

at com.vmware.bdd.exception.BddException.wrapIfNeeded(BddException.java:72)

Is there a .jar(s) I can replace, or a new .ova for the fix?

M

0 Kudos
mknunez
Contributor
Contributor

I found this reference to my current problem:

https://issuetracker.springsource.com/browse/SERENGETI-1042?attachmentSortBy=dateTime

And a comment:

Hong Cheng (c)<https://issuetracker.springsource.com/secure/ViewProfile.jspa?name=hongcheng> added a comment - 30/Jan/13 11:03 PM

It is fixed in the latest build.

How can I get the latest build?

Thanks

MN

0 Kudos
admin
Immortal
Immortal

Did you check the FQDN of VCVA ?   Plz make sure it is not null and correct.

0 Kudos
admin
Immortal
Immortal

That reference you found differ with current problem. And that fixing has been put into the BDE 1.0.  You said you had fixed the issue of name resolution, do you ping nunezm-vcenter.mater.us successfully now ?

0 Kudos
mknunez
Contributor
Contributor

Both forward and reverse lookups are working with fully qualified name. All components pings with FQDN correctly to my knowledge including ESX, vCenter, and the Serengeti Management-Server

Does it bother you at all that there is absolutely no useful information given to the user either in the UI of the CLI about where problems are occurring. This is basic lack of fit and finish. I shouldn't have to scrape your log files to understand where root problems occur.

I'm about to bail on this and go to a useful implementation. I'l just build up my own VM's as a Hadoop cluster and not waste my time here.

I do appreciate your help, but you shouldn't have to be doing this.

MN

0 Kudos
jessehuvmw
Enthusiast
Enthusiast

Hi @mknunez,

I'm a developer of BDE. We are glad you are trying BDE. Sorry for the REST API error you suffered.  We do want to help you to use BDE smoothly.

Would you mind deploying a new BDE 1.0 vApp again (the same build you used) after the following items are done ?

1) add all ESX hosts into a vcenter cluster

2) make sure DNS/IP forward and reverse resolution works

3) BDE vApp must deployed in a resourcepool

4) any other requirements mentioned in BDE Admin Guide for deploying BDE vApp

If you still see the REST API error or any other error, please send me the files in /opt/serengeti/logs/ . I believe we can get you through.

Thanks

Jesse Hu

Cheers, Jesse Hu
0 Kudos
gguanglu
VMware Employee
VMware Employee

There is a known issue on VC 5.5, please check whether that's applicable to your environment.

vCenter Virtual Appliance sets a wrong hostname after deployment

If the vCenter Server Appliance is deployed with OVF environment that has static IP configuration and the hostname is blank, reverse lookup from the IP is not performed properly.

To resolve this issue, set the desired vCenter Server Appliance hostname in the OVF properties during deployment.

0 Kudos
gguanglu
VMware Employee
VMware Employee

We want to help you out the current issue.

could you elaborate out how you fixed the host name issue on vcva? Because it's used widely  in many configuration within the vm, so simply  manually changing host name or dns resolving  file to make it pingable doesn't remove all issues underneath.

THe recommended way to resolve this vcva host name from vc engineering team, is to fresh reinstall vc and make sure host name is input as expected. Please take a look at the notes I sent out before.

0 Kudos
Yonghua
Contributor
Contributor

I'm sorry for what you met during your experience to use BDE. I know that at this time, you probably just want to manual do everything related to Hadoop cluster.

But I still want to explain a little bit in case some time later you might still want to try BDE in your environment.

Basically for why this issue happen, is that VC 5.5 is not installed correctly, but you might not found problem in some other workload. All the solutions in this thread already explained how to fix env DNS issue, I think you've tried and fixed that issue. But after that you still need to reinstall VC 5.5 to completely remove this problem.

But if you don't reinstall VC 5.5, there is still one workaround in BDE. That is to change vc.properties at /opt/serengeti/conf directly. Replacethe wrong VC FQDN or IP in that file with correct ip, so that Serengeti will get the correct VC's URL to go ahead operations afterwards. This is the root cause for what you saw in the log.

And then, you need to remove /opt/serengeti/etc/serengeti-firstboot if it exists, and execute /opt/serengeti/sbin/serengeti-firstboot.rb

It's a little bit tricky so we didn't describe this workaround too much, but anyway, it's solution to continue your work in this status.

In addition, as you said, the error message in CLI/UI does not say anything useful for this error, that's annoying. We'll improve it in next release.

Thanks for giving us your feedback, and looking forward your next chance to try our product.

BR

Emma

0 Kudos
JunW
Enthusiast
Enthusiast

We have added a discussion topic about this issue at Solution to REST API transport layer error during operations after deploying Big Data Extensions where you may find the cause related to your environments.

Regards,

Jun

0 Kudos