trailx
Enthusiast
Enthusiast

Adding HDP 2.1.1 to BDE 2.1 with default application manager ?

Jump to solution

Hi,

I followed KB http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=209105... in order to add HDP 2.1.1

I had an error at this step (deploy basic cluster ) :

installHDP.png

Logs:

Chef::Exceptions::Package: package[hadoop] (hadoop_cluster::default line 363) had an error: Chef::Exceptions::Package: No version specified, and no candidate version available for hadoop

My repo seems ok (hadoop repository):

repo_hadoop.png

Any help would be very appreciate Smiley Wink

--- If you found this information useful, please consider awarding points for "Correct" or "Helpful". Thanks!
0 Kudos
1 Solution

Accepted Solutions
jessehuvmw
Enthusiast
Enthusiast

Could you attach the full ironfan.log and /opt/serengeti/www/distros/manifest on the bde server ?

The issue you met is not related to 2.1.1. But I strongly suggest use the latest HDP 2.1.10. The only difference between 2.1.1 and 2.1.10 is the step 4 in the KB VMware KB: Adding the Hortonworks Data Platform (HDP) Distribution for Hadoop for use with VMware vS...,

4. Enter these information in the new repo file and save the file:

[hortonworks-hdp2]

name=Hortonworks Data Platform Version - HDP-2.1.10.0

baseurl=http://public-repo-1.hortonworks.com/HDP/centos6/2.x/updates/2.1.10.0

gpgcheck=1

gpgkey=http://public-repo-1.hortonworks.com/HDP/centos6/2.x/updates/2.1.10.0/RPM-GPG-KEY/RPM-GPG-KEY-Jenkin...

enabled=1

priority=1


Cheers, Jesse Hu

View solution in original post

0 Kudos
5 Replies
jessehuvmw
Enthusiast
Enthusiast

‌did you run this command when creating the yum repo? You can send the full /opt/Serengeti/logs/ironfan.log for us to debug.

  1. Create and configure the repository by running this command:

    createrepo /var/www/html/hdp/2

BTW,  hdp 2.1.1 has a bug and cannot work with bde. You have to use 2.1.2+. Hdp 2.2 is not supported by default app manager, but supported by ambari app manager in BDE 2.2 released recently.

Cheers, Jesse Hu
0 Kudos
trailx
Enthusiast
Enthusiast

EDIT: Will try with 2.1.15

Hi Jessie,

Yes I created and configured the repository.

About ironfan.log :

10.197.177.160 [2015-07-07T11:36:48+00:00] FATAL: Chef::Exceptions::ChildConvergeError: Chef run process exited unsuccessfully (exit code 1)^M

Bootstrapping node HDP-Worker-0 (10.197.177.160) completed with exit status 1

[2015-07-07T13:36:48+02:00] INFO: Monitoring bootstrap progress of cluster HDP with data: [1, <Ironfan::Common::Server {"name"=>"HDP-Worker-0", "chef_attributes"=>{}, "instances"=>1, "environment"=>:_default, "hadoop_distro"=>"BDE_HortonWorks", "run_list"=>["role[hadoop_datanode]", "role[hadoop_nodemanager]", "role[HDP-cluster]", "role[HDP-Worker-facet]"] chef: HDP-Worker-0 fog: }>]

10.197.177.161 ^M

10.197.177.161     ^[[31m* No version specified, and no candidate version available for hadoop^[[0m^[[0m^M

10.197.177.161     ================================================================================^[[0m^M

10.197.177.161     ^[[31mError executing action `install` on resource 'package[hadoop]'^[[0m^M

10.197.177.161     ================================================================================^[[0m^M

10.197.177.161     ^M

10.197.177.161 ^[[0m    Chef::Exceptions::Package^[[0m^M

10.197.177.161     -------------------------^[[0m^M

10.197.177.161     No version specified, and no candidate version available for hadoop^[[0m^M

10.197.177.161     ^M

10.197.177.161 ^[[0m    Resource Declaration:^[[0m^M

10.197.177.161     ---------------------^[[0m^M

10.197.177.161     # In /var/chef/cache/cookbooks/hadoop_cluster/libraries/hadoop_cluster.rb^M

10.197.177.161 ^[[0m    ^M

10.197.177.161 ^[[0m    363:       package name do^M

10.197.177.161 ^[[0m    364:         # Add retry for basic hadoop rpms which have big file size (about 20M).^M

10.197.177.161 ^[[0m    365:         # The yum install command might timeout when the yum server is overloaded.^M

10.197.177.161 ^[[0m    366:         retries 6^M

10.197.177.161 ^[[0m    367:         retry_delay 5^M

10.197.177.161 ^[[0m    368: ^M

10.197.177.161 ^[[0m    369:         if node[:hadoop][:package_version] != 'current'^M

10.197.177.161 ^[[0m    370:           version node[:hadoop][:package_version]^M

10.197.177.161 ^[[0m    371:         end^M

10.197.177.161 ^[[0m    372:       end^M

10.197.177.161 ^[[0m    ^M

10.197.177.161 ^[[0m    Compiled Resource:^[[0m^M

10.197.177.161     ------------------^[[0m^M

10.197.177.161     # Declared in /var/chef/cache/cookbooks/hadoop_cluster/libraries/hadoop_cluster.rb:363:in `block in hadoop_package'^M

10.197.177.161 ^[[0m    ^M

10.197.177.161 ^[[0m    package("hadoop") do^M

10.197.177.161 ^[[0m      action :install^M

10.197.177.161 ^[[0m      retries 0^M

10.197.177.161 ^[[0m      retry_delay 5^M

10.197.177.161 ^[[0m      guard_interpreter :default^M

10.197.177.161 ^[[0m      package_name "hadoop"^M

10.197.177.161 ^[[0m      timeout 900^M

10.197.177.161 ^[[0m      cookbook_name "hadoop_cluster"^M

10.197.177.161 ^[[0m      recipe_name "default"^M

10.197.177.161 ^[[0m    end^M

10.197.177.161 ^[[0m    ^M

10.197.177.161 ^[[0m[2015-07-07T11:36:48.917+0000] INFO: Running queued delayed notifications before re-raising exception^M

10.197.177.161 ^[[0m^M

10.197.177.161 Running handlers:^[[0m^M

10.197.177.161 [2015-07-07T11:36:48.918+0000] ERROR: Running exception handlers^M

10.197.177.161 [2015-07-07T11:36:48.918+0000] FATAL: Cannot install package hadoop on node HDP-Client-0. SSH to this node and run the command 'sudo yum install hadoop -y' to view error messages.^M

I wasn't aware of this bug between hdp 2.1.1 and BDE 2.1, do you have more information ?

Did you succeed to deploy HDP 2.1.2 with default app manager ? If yes which repository did you use ?

--- If you found this information useful, please consider awarding points for "Correct" or "Helpful". Thanks!
0 Kudos
jessehuvmw
Enthusiast
Enthusiast

Could you attach the full ironfan.log and /opt/serengeti/www/distros/manifest on the bde server ?

The issue you met is not related to 2.1.1. But I strongly suggest use the latest HDP 2.1.10. The only difference between 2.1.1 and 2.1.10 is the step 4 in the KB VMware KB: Adding the Hortonworks Data Platform (HDP) Distribution for Hadoop for use with VMware vS...,

4. Enter these information in the new repo file and save the file:

[hortonworks-hdp2]

name=Hortonworks Data Platform Version - HDP-2.1.10.0

baseurl=http://public-repo-1.hortonworks.com/HDP/centos6/2.x/updates/2.1.10.0

gpgcheck=1

gpgkey=http://public-repo-1.hortonworks.com/HDP/centos6/2.x/updates/2.1.10.0/RPM-GPG-KEY/RPM-GPG-KEY-Jenkin...

enabled=1

priority=1


Cheers, Jesse Hu
0 Kudos
trailx
Enthusiast
Enthusiast

I tried with release 2.1.15 and now it works Smiley Happy

baseurl:http://public-repo-1.hortonworks.com/HDP/centos6/2.x/updates/2.1.15.0

Maybe my repo was corrupt :smileyconfused: because my previous manifest for 2.1.1 seems ok :

"name": "BDE_HortonWorks",

    "vendor": "HDP",

    "version": "2.1.1",

    "packages": [

      {

        "package_repos": [

          "http://10.197.177.180/hdp/2/hortonworks-hdp2.repo"

        ],

        "roles": [

          "hadoop_namenode",

          "hadoop_datanode",

          "hadoop_resourcemanager",

          "hadoop_nodemanager",

          "hadoop_journalnode",

          "hadoop_client",

          "hive",

          "hive_server",

          "pig",

          "hbase_master",

          "hbase_regionserver",

          "hbase_client",

          "zookeeper"

        ]

      }

    ]

  },

--- If you found this information useful, please consider awarding points for "Correct" or "Helpful". Thanks!
0 Kudos
jessehuvmw
Enthusiast
Enthusiast

Glad to know it.  "version": "2.1.1" can be updated to 2.1.15 although it doesn't impact any function.  I will update the KB to use 2.1.15 accordingly.

Are you using BDE 2.1 or 2.2 which can also deploy HDP cluster via Ambari ? Or you just want to use the Chef based Default App Manager in BDE?

Cheers, Jesse Hu
0 Kudos