Hello All,
I am looking for Serengeti documentation which can help me deploy different distributions of Hadoop using vRealize orchestrator. Can some one point me to the documentation / article / blog which can help me achieve this?
Scenario i am working on is described below:
1. I have vRealize Automation configured in test environment.
2. BDE installed and configured with vCenter and vRealize Orchestrator.
3. Hadoop as a service configured in vRealize Automation with default workflows provided with BDE plugin.
4. When i try to deploy Hadoop cluster using vRealize Automation, there is no option to select another distribution.
Do i need another workflow to deploy different distribution or i can tweek some parameters in existing workflow to enable me to deploy different distributions? Has any one tried this?
You need to tweek your workflow to retrieve all the available distributions on the BDE Server by calling BDE REST API https://<bde_server_ip>:8443/serengeti/api/distros . Then in the workflow of creating a cluster, the json body should be like this :
{"name": "cluster_name", "distro": "the_distro_name", "type": "HDFS_MAPRED", "networkConfig": { "MGT_NETWORK": ["defaultNetwork"]} }
You can get the BDE REST API doc through the following steps.
ssh to your Serengeti Management server in BDE vAPP.
run " /opt/serengeti/sbin/cfgrestschema on "
then you can see the API doc in "http://<Serengeti Management Server IP>:8080/restschema/service?APPLICATION=BigDataExtensions"
Big Data Extensions uses Spring Security In-Memory Authentication for authentication and user management.
The Spring Security provides a default authentication URL that monitored by the authentication filter. To be authenticated, you must send a HTTP POST request to this URL with correct vc user name and password in the request body.
The authentication URL is:
https://hostname:8443/serengeti/j_spring_security_check
The HTTP request body is:
j_username=replace-with-your-vc-username&j_password=replace-with-your-vc-password
The Content-Type header in the HTTP request should be:
'application/x-www-form-urlencoded'
On unsuccessful authentication, the HTTP response code is 401 (unauthorized); On successful authentication, the HTTP response code is 200 (OK) and Spring Security returns a cookie in the HTTP response header (using Set-Cookie). The session id must be put into the request header in order to access the REST APIs and logout. By default, the session timeout is 30 minutes.
You may try sending an authentication request by using LINUX command 'curl':
curl -i -k -d 'j_username=replace-with-your-vc-username&j_password=replace-with-your-vc-password' -X POST https://hostname:8443/serengeti/j_spring_security_check
On successful authentication, you will receive the HTTP response like:
HTTP/1.1 200 OK
Server: Apache-Coyote/1.1
Set-Cookie: JSESSIONID=session-id-value; Path=/serengeti; Secure
Content-Length: 0
Date: Wed, 27 Aug 2014 11:58:17 GMT
Connection: keep-alive
Length: 0
Big Data Extensions uses an HTTPS server with a self-signed certificate. For this reason, the client may need to implement a certificate trust mechanism or just ignore the certificate validation.
Hello Jesse,
Thanks for prompt reply.
When i run the command to generate the documentation i get the following error.
[root@172 ~]# /opt/serengeti/sbin/cfgrestschema on
cp: cannot create regular file `/opt/serengeti/tomcat6/webapps/restschema.war': Read-only file system
Do you know any workaround this issue?
The cfgrestschema script is very simple. It just execute "cp /opt/serengeti/restschema/restschema.war /opt/serengeti/tomcat/webapps/"
I checked folder webapps is owned by serengeti user, so could you try to use serengeti user to execute again?
Amazing that I can use root user to execute this command successfully.
Thanks for your reply,
When i run checked the permissions it shows me below output.
[root@172 webapps]# pwd
/opt/serengeti/tomcat6/webapps
[root@172 webapps]# ls -la
total 20
drwx------ 5 serengeti serengeti 4096 Sep 30 2014 .
drwx------ 9 serengeti serengeti 4096 Mar 3 15:08 ..
drwx------ 7 serengeti serengeti 4096 Sep 30 2014 register-plugin
drwx------ 3 serengeti serengeti 4096 Sep 30 2014 ROOT
drwx------ 4 serengeti serengeti 4096 Sep 30 2014 serengeti
Not sure if i can change permissions on these directories. I tried with Serengeti and root users. Not able to copy file.
The version of Serengeti server is 2.1.0
I can also run successfully on BDE 2.1.0.
And has the same permission as you.
[root@sin2-pekaurora-bdcqevlan89-66 ~]# ls -al /opt/serengeti/tomcat6/webapps/
total 20
drwx------ 5 serengeti serengeti 4096 Apr 15 07:14 .
drwx------ 9 serengeti serengeti 4096 Apr 15 06:56 ..
drwx------ 7 serengeti serengeti 4096 Sep 30 2014 register-plugin
drwx------ 3 serengeti serengeti 4096 Sep 30 2014 ROOT
drwx------ 4 serengeti serengeti 4096 Sep 30 2014 serengeti
Could you try to restart tomcat ?
#service tomcat restart
Tried to restart tomcat service. It failed!
[root@172 cli]# service tomcat restart
Wed Apr 15 10:52:45 UTC 2015 tomcat: Stopping Tomcat
Using CATALINA_BASE: /opt/serengeti/tomcat6
Using CATALINA_HOME: /opt/serengeti/tomcat6
Using CATALINA_TMPDIR: /opt/serengeti/tomcat6/temp
Using JRE_HOME: /usr
Using CLASSPATH: /opt/serengeti/tomcat6/bin/bootstrap.jar
Using CATALINA_PID: /var/tmp/catalina.pid
Tomcat did not stop in time. PID file was not removed.
Wed Apr 15 10:52:50 UTC 2015 tomcat: Waiting for Tomcat process (15385) to shutdown completely. 9 seconds left.
Wed Apr 15 10:52:54 UTC 2015 tomcat: Waiting for Tomcat process (15385) to shutdown completely. 6 seconds left.
Wed Apr 15 10:52:57 UTC 2015 tomcat: Waiting for Tomcat process (15385) to shutdown completely. 3 seconds left.
Wed Apr 15 10:53:00 UTC 2015 tomcat: Forecly kill Tomcat process 15385
rm: cannot remove `/var/tmp/catalina.pid': Read-only file system
Wed Apr 15 10:53:00 UTC 2015 tomcat: could not remove stale /var/tmp/catalina.pid
Wed Apr 15 10:53:00 UTC 2015 tomcat: Stop failed
Wed Apr 15 10:53:00 UTC 2015 tomcat: Starting Tomcat
rm: cannot remove `/var/tmp/catalina.pid': Read-only file system
Wed Apr 15 10:53:00 UTC 2015 tomcat: could not remove stale /var/tmp/catalina.pid
Wed Apr 15 10:53:00 UTC 2015 tomcat: Start failed
I Restarted the Serengeti appliance and i was able to copy the documentation to specified locaiton.
Does any one have custom workflows created in orchestrator to deploy different distributions?
Modify the cluster spec (java script)in vCO workflow: Hadoop As A Service->Configuration->Execute Operations->Execute Create Cluster Operation.
Add one parameter "distro":"<distroName>" into the java script. Change roles accordingly.
Thus you can create a cluster using different distros.