从Edge Node访问BDA上的CDH集群（非安全访问）

test1

本文将介绍如何设置从Edge Node访问BDA上CDH集群的HDFS。本文将以CDH 5版本进行测试。

安装之前：

1. 安装JDK：

从http://www.oracle.com/technetwork/java/javase/downloads/java-archive-downloads-javase7-521261.html#jdk-7u25-oth-JPR下载Oracle jdk 1.7.0_25

$rpm-ivh jdk-7u25-linux-x64.rpm

$vi /etc/profile

在最后加上：

exportJAVA_HOME=/usr/java/jdk1.7.0_25

exportPATH=$JAVA_HOME/bin:$PATH

保存后运行：

$source/etc/profile

$echo$JAVA_HOME

/usr/java/jdk1.7.0_25

2. 配置yum源

$vi /etc/yum.repos.d/cloudera-cdh5.repo

把下面这段粘贴进去然后保存:

[cloudera-cdh5]

#PackagesforCloudera’sDistributionforHadoop,Version5,onRedHatorCentOS6x86_64

name=Cloudera’sDistributionforHadoop,Version5

baseurl=http://archive.cloudera.com/cdh5/redhat/6/x86_64/cdh/5/

gpgkey=http://archive.cloudera.com/cdh5/redhat/6/x86_64/cdh/RPM-GPG-KEY-cloudera

enabled=1

gpgcheck=1

在Edge Node安装配置CDH5：

1. 添加库Key

$sudorpm importhttp://archive.cloudera.com/cdh5/redhat/6/x86_64/cdh/RPM-GPG-KEY-cloudera

2. 安装Resource Manager host

$sudoyumcleanall

$sudoyuminstallhadoop-yarn-resourcemanager-y

3. 安装 NameNode host

$sudoyumcleanall

$sudoyuminstallhadoop-hdfs-namenode-y

4. 安装 Secondary NameNode host

$sudoyumcleanall

$

sudoyuminstallhadoop-hdfs-secondarynamenode-y

5. 安装 nodemanager , datanode, mapreduce

$sudoyumcleanall

$sudoyuminstallhadoop-yarn-nodemanagerhadoop-hdfs-datanodehadoop-mapreduce-y

6. 安装 hadoop-mapreduce-historyserver hadoop-yarn-proxyserver

$sudoyumcleanall

$

sudoyuminstallhadoop-mapreduce-historyserverhadoop-yarn-proxyserver-y

7. 安装 hadoop-client

$sudoyumcleanall

$sudoyuminstallhadoop-client-y

8. 从Cloudera Manager下载YARN配置文件yarn-config.zip

9. 解压yarn-config.zip到edge node上的临时目录/tmp

$ unzip yarn-config.zip Archive: yarn-config.zip inflating: yarn-conf/hadoop-env.sh inflating: yarn-conf/hdfs-site.xml inflating: yarn-conf/core-site.xml inflating: yarn-conf/mapred-site.xml inflating: yarn-conf/log4j.properties inflating: yarn-conf/yarn-site.xml

10. 编辑 hadoop-env.sh 文件并设置环境变量：

export HADOOP_HOME=[hadoop-home-dir]/share/hadoop/mapreduce1 export HADOOP_MAPRED_HOME=$HADOOP_HOME export HADOOP_CONF_DIR=[yarn-conf-dir] export JAVA_HOME=[java-home-dir] alias hadoop=$HADOOP_HOME/bin/hadoop alias hdfs=$HADOOP_HOME/../../../bin/hdfs

11. 备份Hadoop配置文件:

# cp / full_path /yarn-conf / full_path /yarn-conf-bak

12. 覆盖Edge Node上的Hadoop配置文件

# cd /full_path/yarn-conf # cp * /etc/hadoop/conf

13. 以root用户SSH登录到 非关键节点(node 04到node 18).

14. 创建一个HDFS目录

hadoop fs -mkdir -p /test/test1

验证远程访问：

1. 访问HDFS:

以具有HDFS访问权限的用户运行如下命令：

$hadoop fs -ls /test Found 1 items drwxr-xr-x - hdfs supergroup 0 2016-12-29 21:47 /test/test

test1

从Edge Node访问BDA上的CDH集群（非安全访问）