mknunez
Contributor
Contributor

BDE vSphere plugin error: Can not add network. Internal error: REST API transport layer error.

This error happening when trying to add Datastore:

build: VMware-BigDataExtensions-1.0.0.332-1315424_OVF10.ova

error log:

/opt/serengeti/logs/serengeti.log

2013 Dec 03 18:49:36,857+0000 ERROR http-8443-2| com.vmware.bdd.rest.RestResource: rest call error

com.vmware.bdd.exception.BddException: Internal error: REST API transport layer error.

        at com.vmware.bdd.exception.BddException.INTERNAL(BddException.java:77)

        at com.vmware.bdd.exception.BddException.wrapIfNeeded(BddException.java:72)

        at com.vmware.bdd.rest.RestResource.handleException(RestResource.java:603)

        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)

        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

        at java.lang.reflect.Method.invoke(Method.java:616)

...

        at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:489)

        at java.lang.Thread.run(Thread.java:679)

Caused by: com.vmware.bdd.utils.AuAssert: FAILURE

        at com.vmware.bdd.utils.AuAssert.FAILURE(AuAssert.java:28)

        at com.vmware.bdd.utils.AuAssert.check(AuAssert.java:43)

        at com.vmware.bdd.utils.AuAssert.check(AuAssert.java:50)

        at com.vmware.bdd.service.resmgmt.impl.ResourceService.refreshDatastore(ResourceService.java:196)

        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)

        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

        at java.lang.reflect.Method.invoke(Method.java:616)

...

        at sun.proxy.$Proxy40.addDatastores(Unknown Source)

        at com.vmware.bdd.rest.RestResource.addDatastore(RestResource.java:441)

        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)

        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

        at java.lang.reflect.Method.invoke(Method.java:616)

        at org.springframework.web.bind.annotation.support.HandlerMethodInvoker.invokeHandlerMethod(HandlerMethodInvoker.java:176)

        at org.springframework.web.servlet.mvc.annotation.AnnotationMethodHandlerAdapter.invokeHandlerMethod(AnnotationMethodHandlerAdapter.java:440)

        at org.springframework.web.servlet.mvc.annotation.AnnotationMethodHandlerAdapter.handle(AnnotationMethodHandlerAdapter.java:428)

        at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherSer

0 Kudos
30 Replies
MichaelWest
VMware Employee
VMware Employee


You did not post the exact command that you executed, so I can't verify that syntax.  Either way, we should have captured any syntax error through the CLI.  I can't tell you exactly why you are getting this error, but I have been able to fix it by restarting all of the serengeti services.  login to the management console and execute /opt/serengeti/sbin/serengeti-stop-services.sh and then /opt/serengeti/sbin/serengeti-start-services.sh.

0 Kudos
mknunez
Contributor
Contributor

No help. After issuing these two commands (stop/start-services.sh) I still get the same error.

It doesn't matter what I type in, I just filled in the two fields in the dialog with a valid datastore name.

This is *not* through the CLI. Maybe I need to switch over to the CLI if the REST layer/vSphere plugin is busted.

0 Kudos
MichaelWest
VMware Employee
VMware Employee

Has the vcenter plugin worked previously and now is not working?  Or is this a new install?

Try the CLI and see if that works

Some things to think about on the Plugin:  the plugin must match the version of the appliance.  So if you upgraded the BDE appliance, you must install the plugin from that appliance.  If you have multiple appliances deployed, make sure you are connected to the right one.  You might also try disconnecting from the server and reconnecting.   Also ensure that you are pointing to the correct SSOLookup url.  I'm not sure either of those is the problem, but worth trying once you isolate this to the plugin.

Also, is this a datastore or a network you cannot add?  You mention both.

0 Kudos
mknunez
Contributor
Contributor

this is a new install with the 1.0 distro (see original note above about the Build ID) today, so this has never worked. This is a first install of the plugin with vSphere 5.5 GA.

What do you mean by "the plugin must match the version of the appliance."? Should I go to the /MOB and verify this?

I only have one version of the BDE deployed, I am connected to that one. Connect worked fine. I filled in the correct SSO URL, otherwise Connect would fail.

Will try the CLI next.

0 Kudos
MichaelWest
VMware Employee
VMware Employee

after deploying the vapp, you connect to the management server:8443/register-plugin.  This installs the plugin from that server.  If you were to upgrade later on to a new version of the management server, you would need to reinstall the plugin from that server.

On vapp deployment server is a check box that also you to use the datastore and network into which you deployed the vapp to deploy your hadoop cluster.  You should click on resources and then datastores and networks tabs in the UI to see if there are any values there.  If there are default values, then you have already added the network/datastore associated with the vapp deployment.  Adding again would generate an error.

0 Kudos
mknunez
Contributor
Contributor

The vApp in the vSphere > Hosts and Clusters view is showing the proper Management Server > Related Objects > Network and Datastore values. Just the BDE plugin is not working properly. No values are displayed in either of the views in the BDE plugin Resources area.

0 Kudos
MichaelWest
VMware Employee
VMware Employee

what happened when you tried the CLI to add a datastore?

Just to be clear on the format:  the NAME is an identifier that you will use when referring to that set of datastores or networks within BDE.  the DATASTORE or PORTGROUP part of the command (Depending on whether you are adding a network or datastore) is the vcenter name for it:  Example:

if I have datastores called  ds-1, ds-2 and ds-3 in vcenter, to add all of them into BDE I would specify a NAME (like BDE-DSs) and a datastore value of ds-*.  That would associate ds-1, ds-2, ds-3 with the name BDE-DSs.  So when you create a cluster later on, you can specify the particular datastore(s) to use by referring to BDE-DSs.  I write this because I've had some issues when using a NAME that is the same as a vcenter datastore name.

Also, are you using datastores or datastore clusters.  i don't think we have added support for datastore clusters yet.

0 Kudos
mknunez
Contributor
Contributor

will read the CLI guide and do that next.

0 Kudos
MichaelWest
VMware Employee
VMware Employee


let me help:

from the management server

type serengeti

connect --host localhost:8443

username:  your sso username@domain

password:

datastore add --name NAME --spec "yourwildcard string" --type LOCAL

See my example above where NAME was  BDE-DSs and yourwildcard string was ds-*

type can be LOCAL or SHARED though we don't actually check it.

0 Kudos
mknunez
Contributor
Contributor

hum...I get the same error at the CLI

$ serengeti

=================================================

*  _____                                 _   _  *

* / ____|  ___ _ __ ___ _ __   __ _  ___| |_(_) *

* \____ \ / _ \ '__/ _ \ '_ \ / _` |/ _ \ __| | *

*  ____) |  __/ | |  __/ | | | (_| |  __/ |_| | *

* |_____/ \___|_|  \___|_| |_|\__, |\___|\__|_| *

*                             |___/             *

*                                               *

=================================================

Version: 1.0.0

Welcome to Serengeti CLI

serengeti>connect --host localhost:8443

Enter the username: ******

Enter the password: ******

Connected

serengeti>datastore add --name BDE-Datastores --spec datastore1 --type LOCAL

datastore BDE-Datastores add failed: Internal error: REST API transport layer error.

(where datastore1 is a valid Datastore in vSphere)

0 Kudos
MichaelWest
VMware Employee
VMware Employee

and datastore1 is a datastore, not a datastore cluster, right?

0 Kudos
mknunez
Contributor
Contributor

correct.

0 Kudos
yufeim
Contributor
Contributor

It's not easy to find the problem by these information, would you please supply your whole serengeti logfile?

0 Kudos
mknunez
Contributor
Contributor

Log file attached. Thanks.

MN

0 Kudos
yufeim
Contributor
Contributor

It seems that a deploy problem. How did you deploy the BDE? Would you provide the vc.properties, serengeti-firstboot.log and serengeti-firstboot.err?

0 Kudos
mknunez
Contributor
Contributor

Attached. Thanks for your help.

MN

0 Kudos
mknunez
Contributor
Contributor

I deployed the BDE as a .ova in vSphere, just as instructed in the admin guide. This was indicated in my original problem description.

MN

0 Kudos
yufeim
Contributor
Contributor

Have you pinged the FQDN, nunezm-vcenter.mater.us? If you fail pinging it, you maybe choosed DHCP type when you deployed BDE ova.
I'll give you 2 solutions to fix this.
Method 1, you need configure a DNS address in the management-server.
Method 2, you need login the vCenter server, then click Administration > Advanced Settings > change the incorrect FQDN to the correct IP address.
Meanwhile, change the "https://nunezm-vcenter.mater.us/vsm/extensionService" of the vc.properties of the management-server as the correct IP as well.
Otherwise, you can ping it successfully. Maybe you forget configure the FQDN in the VCVA. If so, you need reinstall VCVA.

0 Kudos
mknunez
Contributor
Contributor

The workaround for now is to create a cluster by hand and move the host into the cluster, then the REST layer starts working.

M

0 Kudos