VMware Cloud Community
Kedar_D
Enthusiast
Enthusiast

Hadoop as a service with vRealize Automation

Hello All,

I am looking for Serengeti documentation which can help me deploy different distributions of Hadoop using vRealize orchestrator. Can some one point me to the documentation / article / blog which can help me achieve this?

Scenario i am working on is  described below:

1. I have vRealize Automation configured in test environment.

2. BDE installed and configured with vCenter and vRealize Orchestrator.

3. Hadoop as a service configured in vRealize Automation with default workflows provided with BDE plugin.

4. When i try to deploy Hadoop cluster using vRealize Automation, there is no option to select another distribution.

Do i need another workflow to deploy different distribution or i can tweek some parameters in existing workflow to enable me to deploy different distributions? Has any one tried this?

------------------------------------------------------------ If you find this or any other answer useful please mark the answer as correct or helpful. Kedar http://virtualtechknow.wordpress.com/ ------------------------------------------------------------
Reply
0 Kudos
9 Replies
jessehuvmw
Enthusiast
Enthusiast

You need to tweek your workflow to retrieve all the available distributions on the BDE Server by calling BDE REST API  https://<bde_server_ip>:8443/serengeti/api/distros .  Then in the workflow of creating a cluster, the json body should be like this :

{"name": "cluster_name", "distro": "the_distro_name", "type": "HDFS_MAPRED", "networkConfig": { "MGT_NETWORK": ["defaultNetwork"]} }



You can get the BDE REST API doc through the following steps.

     ssh to your Serengeti Management server in BDE vAPP.
     run  " /opt/serengeti/sbin/cfgrestschema on "
     then you can see the API doc in "http://<Serengeti Management Server IP>:8080/restschema/service?APPLICATION=BigDataExtensions"

Big Data Extensions uses Spring Security In-Memory Authentication for authentication and user management.

The Spring Security provides a default authentication URL that monitored by the authentication filter. To be authenticated, you must send a HTTP POST request to this URL with correct vc user name and password in the request body.

The authentication URL is:
https://hostname:8443/serengeti/j_spring_security_check

The HTTP request body is:
j_username=replace-with-your-vc-username&j_password=replace-with-your-vc-password

The Content-Type header in the HTTP request should be:
'application/x-www-form-urlencoded'

On unsuccessful authentication, the HTTP response code is 401 (unauthorized); On successful authentication, the HTTP response code is 200 (OK) and Spring Security returns a cookie in the HTTP response header (using Set-Cookie). The session id must be put into the request header in order to access the REST APIs and logout. By default, the session timeout is 30 minutes.

You may try sending an authentication request by using LINUX command 'curl':
curl -i -k -d 'j_username=replace-with-your-vc-username&j_password=replace-with-your-vc-password' -X POST https://hostname:8443/serengeti/j_spring_security_check

On successful authentication, you will receive the HTTP response like:
HTTP/1.1 200 OK
Server: Apache-Coyote/1.1
Set-Cookie: JSESSIONID=session-id-value; Path=/serengeti; Secure
Content-Length: 0
Date: Wed, 27 Aug 2014 11:58:17 GMT
Connection: keep-alive
Length: 0

Big Data Extensions uses an HTTPS server with a self-signed certificate. For this reason, the client may need to implement a certificate trust mechanism or just ignore the certificate validation.

Cheers, Jesse Hu
Reply
0 Kudos
Kedar_D
Enthusiast
Enthusiast

Hello Jesse,

Thanks for prompt reply.

When i run the command to generate the documentation i get the following error.

[root@172 ~]# /opt/serengeti/sbin/cfgrestschema on

cp: cannot create regular file `/opt/serengeti/tomcat6/webapps/restschema.war': Read-only file system

Do you know any workaround this issue?

------------------------------------------------------------ If you find this or any other answer useful please mark the answer as correct or helpful. Kedar http://virtualtechknow.wordpress.com/ ------------------------------------------------------------
Reply
0 Kudos
fuxiaoting0822
VMware Employee
VMware Employee

The cfgrestschema script is very simple. It just execute "cp /opt/serengeti/restschema/restschema.war /opt/serengeti/tomcat/webapps/"

I checked folder webapps is owned by serengeti user, so could you try to use serengeti user to execute again?

Amazing that I can use root user to execute this command successfully.

Reply
0 Kudos
Kedar_D
Enthusiast
Enthusiast

Thanks for your reply,

When i run checked the permissions it shows me below output.

[root@172 webapps]# pwd

/opt/serengeti/tomcat6/webapps

[root@172 webapps]# ls -la

total 20

drwx------ 5 serengeti serengeti 4096 Sep 30  2014 .

drwx------ 9 serengeti serengeti 4096 Mar  3 15:08 ..

drwx------ 7 serengeti serengeti 4096 Sep 30  2014 register-plugin

drwx------ 3 serengeti serengeti 4096 Sep 30  2014 ROOT

drwx------ 4 serengeti serengeti 4096 Sep 30  2014 serengeti

Not sure if i can change permissions on these directories.  I tried with Serengeti and root users. Not able to copy file.

The version of Serengeti server is 2.1.0

------------------------------------------------------------ If you find this or any other answer useful please mark the answer as correct or helpful. Kedar http://virtualtechknow.wordpress.com/ ------------------------------------------------------------
Reply
0 Kudos
fuxiaoting0822
VMware Employee
VMware Employee

I can also run successfully on BDE 2.1.0.

And has the same permission as you.

[root@sin2-pekaurora-bdcqevlan89-66 ~]# ls -al /opt/serengeti/tomcat6/webapps/

total 20

drwx------ 5 serengeti serengeti 4096 Apr 15 07:14 .

drwx------ 9 serengeti serengeti 4096 Apr 15 06:56 ..

drwx------ 7 serengeti serengeti 4096 Sep 30  2014 register-plugin

drwx------ 3 serengeti serengeti 4096 Sep 30  2014 ROOT

drwx------ 4 serengeti serengeti 4096 Sep 30  2014 serengeti

Could you try to restart tomcat ?

#service tomcat restart

Reply
0 Kudos
Kedar_D
Enthusiast
Enthusiast

Tried to restart tomcat service. It failed!

[root@172 cli]# service tomcat restart

Wed Apr 15 10:52:45 UTC 2015 tomcat: Stopping Tomcat

Using CATALINA_BASE:   /opt/serengeti/tomcat6

Using CATALINA_HOME:   /opt/serengeti/tomcat6

Using CATALINA_TMPDIR: /opt/serengeti/tomcat6/temp

Using JRE_HOME:        /usr

Using CLASSPATH:       /opt/serengeti/tomcat6/bin/bootstrap.jar

Using CATALINA_PID:    /var/tmp/catalina.pid

Tomcat did not stop in time. PID file was not removed.

Wed Apr 15 10:52:50 UTC 2015 tomcat: Waiting for Tomcat process (15385) to shutdown completely. 9 seconds left.

Wed Apr 15 10:52:54 UTC 2015 tomcat: Waiting for Tomcat process (15385) to shutdown completely. 6 seconds left.

Wed Apr 15 10:52:57 UTC 2015 tomcat: Waiting for Tomcat process (15385) to shutdown completely. 3 seconds left.

Wed Apr 15 10:53:00 UTC 2015 tomcat: Forecly kill Tomcat process 15385

rm: cannot remove `/var/tmp/catalina.pid': Read-only file system

Wed Apr 15 10:53:00 UTC 2015 tomcat: could not remove stale /var/tmp/catalina.pid

Wed Apr 15 10:53:00 UTC 2015 tomcat: Stop failed

Wed Apr 15 10:53:00 UTC 2015 tomcat: Starting Tomcat

rm: cannot remove `/var/tmp/catalina.pid': Read-only file system

Wed Apr 15 10:53:00 UTC 2015 tomcat: could not remove stale /var/tmp/catalina.pid

Wed Apr 15 10:53:00 UTC 2015 tomcat: Start failed

------------------------------------------------------------ If you find this or any other answer useful please mark the answer as correct or helpful. Kedar http://virtualtechknow.wordpress.com/ ------------------------------------------------------------
Reply
0 Kudos
Kedar_D
Enthusiast
Enthusiast

I Restarted the Serengeti appliance and i was able to copy the documentation to specified locaiton.

------------------------------------------------------------ If you find this or any other answer useful please mark the answer as correct or helpful. Kedar http://virtualtechknow.wordpress.com/ ------------------------------------------------------------
Reply
0 Kudos
Kedar_D
Enthusiast
Enthusiast

Does any one have custom workflows created in orchestrator to deploy different distributions?

------------------------------------------------------------ If you find this or any other answer useful please mark the answer as correct or helpful. Kedar http://virtualtechknow.wordpress.com/ ------------------------------------------------------------
Reply
0 Kudos
fuxiaoting0822
VMware Employee
VMware Employee

Modify the cluster spec (java script)in vCO workflow: Hadoop As A Service->Configuration->Execute Operations->Execute Create Cluster Operation.

Add one parameter  "distro":"<distroName>" into the java script. Change roles accordingly.

Thus you can create a cluster using different distros.

Reply
0 Kudos