VMware Cloud Community
carlb
Contributor
Contributor
Jump to solution

Create a cluster via serengeti without an Hadoop distribution installed?

Hi,

Is it possible to have Serengeti create a cluster without Hadoop? I know this sounds like a silly question but I'm wanting to know whether I can use Serengeti to basically create a set of VMs and assign IPs with defined storage and RAM. I'll then want to install my own packages after, not Hadoop but if Serengeti can create the initial VMs this would be good.

Thanks in advance,

Carl.

Reply
0 Kudos
1 Solution

Accepted Solutions
carlb
Contributor
Contributor
Jump to solution

This worked, thanks. So I guess it was sudo sed -i '/"hadoop_namenode",/a "basic",' /opt/serengeti/www/distros/manifest

COMPLETED 100%

node group: worker,  instance number: 1

roles:[basic]

  NAME            IP             STATUS         TASK

  --------------------------------------------------

  basic-worker-0  10.100.12.128  Service Ready

node group: master,  instance number: 1

roles:[basic]

  NAME            IP             STATUS         TASK

  --------------------------------------------------

  basic-master-0  10.100.12.129  Service Ready

cluster basic created

serengeti>

So thanks. It's placed the two VMs in different resource pools. Which config file defines this as I don't see it referenced in the .json templates in /opt/serengeti/conf.

View solution in original post

Reply
0 Kudos
15 Replies
MichaelWest
VMware Employee
VMware Employee
Jump to solution

Carl,

Not a silly question.  This functionality will be available in BDE 2.0, which is scheduled to GA in June.  Some of the use cases for this are to deploy/resize cluster nodes and then install various big data applications with the app specific management tool.  You may also think about using a frontend portal (like vCAC) to make REST calls to BDE and then the other management tools to automate provisioning of any kind of cluster.

Reply
0 Kudos
jessehuvmw
Enthusiast
Enthusiast
Jump to solution

Hi Carl,

If you are using BDE 1.1 , you can follow these steps to create the empty cluster you want :

1) apply this patch VMware KB:    HDP 2.0 patch installation for Big Data Extensions 1.1 on BDE 1.1 Server

2) login BDE server as user serengeti, and create a new file /opt/serengeti/cookbooks/roles/basic.rb with the content below :

name    'basic'

description 'Basic role for any software deployment. It includes seting up IP/FQDN, mounting and formatting local disks.'

run_list *%w[

  hadoop_common::pre_run

  hadoop_common::mount_disks

]

Then upload this new role:   knife role from file /opt/serengeti/cookbooks/roles/basic.rb

3) create a cluster spec file basic_cluster.json with the content below :

// This is a sample cluster spec for creating a cluster without installing any hadoop stuff.

// You can create a set of emtpy VMs (with OS installed and data disks mounted) using this spec or you can add your customized roles into this spec to deploy your application.

{

  "nodeGroups":[

    {

      "name": "master",

      "roles": [

        "basic"

      ],

      "instanceNum": 1,

      "cpuNum": 2,

      "memCapacityMB": 3768,

      "storage": {

        "type": "SHARED",

        "sizeGB": 10

      },

      "haFlag": "on"

    },

    {

      "name": "worker",

      "roles": [

        "basic"

      ],

      "instanceNum": 1,

      "cpuNum": 2,

      "memCapacityMB": 3768,

      "storage": {

        "type": "LOCAL",

        "sizeGB": 10

      },

      "haFlag": "off"

    }

  ]

}

4) create the empty cluster :  cluster create --name basic --specFile basic_cluster.json

Thanks

Jesse

Cheers, Jesse Hu
Reply
0 Kudos
carlb
Contributor
Contributor
Jump to solution

Thanks alot Jesse and Michael.

I worked through your suggestion Jesse and get the following error:

serengeti>cluster create  --name basic --specFile basic_cluster.json --networkName defaultNetwork

cluster basic create failed: Invalid values master.roles=basic. worker.roles=basic.

The networkName needs adding as it appears to be mandatory.

So I guess the problem was with adding the basic role? But the below implies it's in.

[serengeti@10 ~]$ knife role show basic

chef_type:           role

default_attributes:

description:         Basic role for software deployment. It includes setting up IP/FQDN, mounting and formatting local disks.

env_run_lists:

json_class:          Chef::Role

name:                basic

override_attributes:

run_list:

  recipe[hadoop_common::pre_run]

  recipe[hadoop_common::mount_disks]

[serengeti@10 ~]$

Any suggestions.

Thanks,

Carl.

Reply
0 Kudos
jessehuvmw
Enthusiast
Enthusiast
Jump to solution

Did you see any error when applying the patch by following these steps ? 

1. Login to the Serengeti server as user serengeti.

2. Download patch-hdp2-BDE-1.1.0.tar to your /tmp directory. The patch is attached to this article.

3. Run the following commands to install the patch.

cd /tmp

tar -xvf patch-hdp2-BDE-1.1.0.tar

cd patch-hdp2-BDE-1.1.0

./install.sh

sudo /sbin/service tomcat restart


And could you paste the output of "ll /opt/serengeti/cli/lib/commons-serengeti-1.1.0.jar "

Cheers, Jesse Hu
Reply
0 Kudos
jessehuvmw
Enthusiast
Enthusiast
Jump to solution

If the patch is applied successfully and the tomcat service is restarted, you also need to logout the Serengeti CLI and login again to create the basic cluster.

Cheers, Jesse Hu
Reply
0 Kudos
carlb
Contributor
Contributor
Jump to solution

I'd already applied the patch a few weeks ago and have created a HDP 2.0.11 cluster without problems.

The output is:

[serengeti@10 ~]$ ll /opt/serengeti/cli/lib/commons-serengeti-1.1.0.jar

-rw-r--r-- 1 serengeti serengeti 144323 Apr 22 11:55 /opt/serengeti/cli/lib/commons-serengeti-1.1.0.jar

For good measure i restarted tomcat with sudo service tomcat restart and exited and login back into the Serengenti CLI. Still the same error.

Version: 1.1.0

Welcome to Serengeti CLI

serengeti>connect --host 10.100.12.81:8443

Enter the username: administrator@vsphere.local

Enter the password: ********

Connected

serengeti>cluster create  --name basic --specFile basic_cluster.json --networkName defaultNetwork

cluster basic create failed: Invalid values master.roles=basic. worker.roles=basic.

serengeti>

So thanks for the suggestion but no luck on that.

Regards,

Carl.

Reply
0 Kudos
jessehuvmw
Enthusiast
Enthusiast
Jump to solution

That's strange. I can create the basic cluster with this patch and have the same file /opt/serengeti/cli/lib/commons-serengeti-1.1.0.jar as yours.

Let's try this appoach:

[serengeti@bde-server ~]$ sudo sed -i '/"hadoop_namenode",/a "basic",' /opt/serengeti/www/distros/manifest

[serengeti@bde-server ~]$ sudo service tomcat restart

[serengeti@bde-server ~]$ serengeti

[serengeti@bde-server ~]$ serengeti

serengeti>cluster create  --name basic --specFile basic_cluster.json --networkName defaultNetwork

And please send me the /opt/serengeti/logs/serengeti.log if it still doesn't work.

Cheers, Jesse Hu
Reply
0 Kudos
carlb
Contributor
Contributor
Jump to solution

This worked, thanks. So I guess it was sudo sed -i '/"hadoop_namenode",/a "basic",' /opt/serengeti/www/distros/manifest

COMPLETED 100%

node group: worker,  instance number: 1

roles:[basic]

  NAME            IP             STATUS         TASK

  --------------------------------------------------

  basic-worker-0  10.100.12.128  Service Ready

node group: master,  instance number: 1

roles:[basic]

  NAME            IP             STATUS         TASK

  --------------------------------------------------

  basic-master-0  10.100.12.129  Service Ready

cluster basic created

serengeti>

So thanks. It's placed the two VMs in different resource pools. Which config file defines this as I don't see it referenced in the .json templates in /opt/serengeti/conf.

Reply
0 Kudos
jessehuvmw
Enthusiast
Enthusiast
Jump to solution

glad to see it works.  Actually after I deployed the BDE 1.1 and applied the patch VMware KB:    HDP 2.0 patch installation for Big Data Extensions 1.1 , I'm able to create the basic cluster. Not sure why it doesn't work in your env.

If you want to install other software in the VMs, you can write your own chef cookbooks and roles to install it, upload them into BDE server, then create a cluster spec with your chef roles (instead of the basic role).

If you need to get all IPs of the VMs in the cluster, you can run 'serengeti-ssh.sh basic' to display the IPs of cluster basic line by line.

Thanks.

Cheers, Jesse Hu
Reply
0 Kudos
carlb
Contributor
Contributor
Jump to solution

Jesse,

Thanks for the help and advice. Can you point me to any good online resources regarding creating custom cookbooks. I look around to see if anyone has used BDE to deploy apps other than Hadoop and didn't see anything so i assume no one is really using it for anything other than Hadoop.

Carl.

Reply
0 Kudos
jessehuvmw
Enthusiast
Enthusiast
Jump to solution

Hi Carl,

The best place to learn Chef and write cookbook is https://learnchef.opscode.com/

There are existing open sourced Chef Cookbooks can be used to add into BDE server.

Hadoop: https://github.com/infochimps-labs/ironfan-pantry/tree/master/cookbooks/

Non-Hadoop: https://github.com/opscode-cookbooks

The BDE built in cookbooks are in /opt/serengeti/cookbooks/cookbooks , you can also take them as examples.

You can put your cookbooks in /opt/serengeti/cookbooks/cookbooks and your roles in /opt/serengeti/cookbooks/roles , then upload them using these commands:

knife cookbook upload --all

knife role from file /opt/serengeti/cookbooks/roles/*.rb

Cheers, Jesse Hu
Reply
0 Kudos
carlb
Contributor
Contributor
Jump to solution

You're a star. I'll look through these. thanks.

Reply
0 Kudos
jessehuvmw
Enthusiast
Enthusiast
Jump to solution

Thanks for using BDE. Could I know which software are you going to deploy using BDE ? Is it hadoop ecosystem software or non-hadoop ?

Cheers, Jesse Hu
Reply
0 Kudos
carlb
Contributor
Contributor
Jump to solution

It will be a mixture of applications which are distributed. At the moment we are evaluating a few platforms which could be used but the potential for BDE for us is compelling.

Thanks for your help. I'll let you know if we go this route and have success.

Regards,

Carl.

Reply
0 Kudos
jessehuvmw
Enthusiast
Enthusiast
Jump to solution

Thanks for the info. Wish you good luck with BDE. The BDE team is willing to offer help.

As @ mentioned, in next BDE release, the customer is able to easily deploy any distributed or non-distributed software application (hadoop or non-hadoop) by extending BDE with your own chef cookbooks and roles like this:

1) Write your own chef cookbooks and roles, upload them into BDE Server

2) Add a new distro with vendor GENERIC :

config-distro.rb --name mydistro --vendor GENERIC --version 1.0 --repos http://url_to_your_yum_server1.repo,http://url_to_your_yum_server2.repo --roles role1,role2

3) Create your cluster with your chef roles :

cluster create --name mycluster --distro mydistro --specFile mycluster.json

You can put any rpm packages into your yum servers. The Node Template can be CentOS 6.x, RHEL 6.x and Fedora.

Cheers, Jesse Hu
Reply
0 Kudos