经验首页 前端设计 程序设计 Java相关 移动开发 数据库/运维 软件/图像 大数据/云计算 其他经验
当前位置：技术经验 » 大数据/云/AI » Hadoop » 查看文章
大数据集群环境搭建之一 hadoop-ha高可用安装
来源：cnblogs　　作者：changjiangLi　　时间：2019/10/11 8:49:30　　对本文有异议

1、如果你使用root用户进行安装。 vi /etc/profile 即可 系统变量

2、如果你使用普通用户进行安装。 vi ~/.bashrc 用户变量

export HADOOP_HOME=/export/servers/hadoop-2.8.5

export PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin:

同步配置文件

[root@jiang01 servers]# vi /etc/profile

[root@jiang01 servers]#

[root@jiang01 servers]# xrsync.sh /etc/profile

=========== jiang02 : /etc/profile ===========

命令执行成功

=========== jiang03 : /etc/profile ===========

命令执行成功

[root@jiang01 servers]#

刷新配置各个机器配置：

source /etc/profile

修改下面各个配置文件：

  1. <?xml version="1.0" encoding="UTF-8"?>
  2. <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
  3. <!--
  4.   Licensed under the Apache License, Version 2.0 (the "License");
  5.   you may not use this file except in compliance with the License.
  6.   You may obtain a copy of the License at
  8.     http://www.apache.org/licenses/LICENSE-2.0
  10.   Unless required by applicable law or agreed to in writing, software
  11.   distributed under the License is distributed on an "AS IS" BASIS,
  12.   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  13.   See the License for the specific language governing permissions and
  14.   limitations under the License. See accompanying LICENSE file.
  15. -->
  16.  
  17. <!-- Put site-specific property overrides in this file. -->
  18.  
  19. <configuration>
  20.         <!-- 指定hdfs的nameservice为ns1 -->
  21.     <property>
  22.         <name>fs.defaultFS</name>
  23.         <value>hdfs://myha01/</value>
  24.     </property>
  25.         <!-- 指定hadoop临时目录 -->
  26.     <property>
  27.         <name>hadoop.tmp.dir</name>
  28.         <value>/export/servers/hadoop-2.8.5/hadoopDatas/tempDatas</value>
  29.     </property>
  30.         <!-- 指定zookeeper地址 -->
  31.     <property>
  32.         <name>ha.zookeeper.quorum</name>
  33.         <value>jiang01:2181,jiang02:2181,jiang03:2181</value>
  34.     </property>
  35. </configuration>
core-site.xml
  1. <?xml version="1.0" encoding="UTF-8"?>
  2. <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
  3. <!--
  4.   Licensed under the Apache License, Version 2.0 (the "License");
  5.   you may not use this file except in compliance with the License.
  6.   You may obtain a copy of the License at
  8.     http://www.apache.org/licenses/LICENSE-2.0
  10.   Unless required by applicable law or agreed to in writing, software
  11.   distributed under the License is distributed on an "AS IS" BASIS,
  12.   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  13.   See the License for the specific language governing permissions and
  14.   limitations under the License. See accompanying LICENSE file.
  15. -->
  16.  
  17. <!-- Put site-specific property overrides in this file. -->
  18.  
  19. <configuration>
  20.     <!--指定hdfs的nameservice为ns1，需要和core-site.xml中的保持一致 -->
  21.     <property>
  22.         <name>dfs.nameservices</name>
  23.         <value>myha01</value>
  24.     </property>
  25.     <!-- ns1下面有两个NameNode，分别是nn1，nn2 -->
  26.     <property>
  27.         <name>dfs.ha.namenodes.myha01</name>
  28.         <value>nn1,nn2</value>
  29.     </property>
  30.     <!-- nn1的RPC通信地址 -->
  31.     <property>
  32.         <name>dfs.namenode.rpc-address.myha01.nn1</name>
  33.         <value>jiang01:9000</value>
  34.     </property>
  35.     <!-- nn1的http通信地址 -->
  36.     <property>
  37.         <name>dfs.namenode.http-address.myha01.nn1</name>
  38.         <value>jiang01:50070</value>
  39.     </property>
  40.     <!-- nn2的RPC通信地址 -->
  41.     <property>
  42.         <name>dfs.namenode.rpc-address.myha01.nn2</name>
  43.         <value>jiang02:9000</value>
  44.     </property>
  45.     <!-- nn2的http通信地址 -->
  46.     <property>
  47.         <name>dfs.namenode.http-address.myha01.nn2</name>
  48.         <value>jiang02:50070</value>
  49.     </property>
  50.     <!-- 指定NameNode的元数据在JournalNode上的存放位置 -->
  51.     <property>
  52.         <name>dfs.namenode.shared.edits.dir</name>
  53.         <value>qjournal://jiang01:8485;jiang02:8485;jiang03:8485/myha01</value>
  54.     </property>
  55.     <!-- 指定JournalNode在本地磁盘存放数据的位置 -->
  56.     <property>
  57.         <name>dfs.journalnode.edits.dir</name>
  58.         <value>/opt/hadoop-2.8.5/journal</value>
  59.     </property>
  60.     <!-- 开启NameNode失败自动切换 -->
  61.     <property>
  62.         <name>dfs.ha.automatic-failover.enabled</name>
  63.         <value>true</value>
  64.     </property>
  65.     <!-- 配置失败自动切换实现方式 -->
  66.     <property>
  67.         <name>dfs.client.failover.proxy.provider.myha01</name>
  68.         <value>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider</value>
  69.     </property>
  70.     <!-- 配置隔离机制 -->
  71.     <property>
  72.         <name>dfs.ha.fencing.methods</name>
  73.         <value>sshfence</value>
  74.     </property>
  75.     <!-- 使用隔离机制时需要ssh免登陆 -->
  76.     <property>
  77.         <name>dfs.ha.fencing.ssh.private-key-files</name>
  78.         <value>/root/.ssh/id_dsa</value>
  79.     </property>
  80. </configuration>
hdfs-site.xml
  1. <?xml version="1.0"?>
  2. <!--
  3.   Licensed under the Apache License, Version 2.0 (the "License");
  4.   you may not use this file except in compliance with the License.
  5.   You may obtain a copy of the License at
  7.     http://www.apache.org/licenses/LICENSE-2.0
  9.   Unless required by applicable law or agreed to in writing, software
  10.   distributed under the License is distributed on an "AS IS" BASIS,
  11.   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  12.   See the License for the specific language governing permissions and
  13.   limitations under the License. See accompanying LICENSE file.
  14. -->
  15. <configuration>
  16.    <!-- Site specific YARN configuration properties -->
  17. <!-- 开启RM高可靠 -->
  18.         <property>
  19.             <name>yarn.resourcemanager.ha.enabled</name>
  20.             <value>true</value>
  21.         </property>
  22.         <!-- 指定RM的cluster id -->
  23.         <property>
  24.             <name>yarn.resourcemanager.cluster-id</name>
  25.             <value>yrc</value>
  26.         </property>
  27.         <!-- 指定RM的名字 -->
  28.         <property>
  29.             <name>yarn.resourcemanager.ha.rm-ids</name>
  30.             <value>rm1,rm2</value>
  31.         </property>
  32.         <!-- 分别指定RM的地址 -->
  33.         <property>
  34.             <name>yarn.resourcemanager.hostname.rm1</name>
  35.             <value>jiang02</value>
  36.         </property>
  37.         <property>
  38.             <name>yarn.resourcemanager.hostname.rm2</name>
  39.             <value>jiang03</value>
  40.         </property>
  41.         <!-- 指定zk集群地址 -->
  42.         <property>
  43.             <name>yarn.resourcemanager.zk-address</name>
  44.             <value>jiang01:2181,jiang02:2181,jiang03:2181</value>
  45.         </property>
  46.         <property>
  47.             <name>yarn.nodemanager.aux-services</name>
  48.             <value>mapreduce_shuffle</value>
  49.         </property>
  50. </configuration>
yarn-site.xml
  1. <?xml version="1.0"?>
  2. <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
  3. <!--
  4.   Licensed under the Apache License, Version 2.0 (the "License");
  5.   you may not use this file except in compliance with the License.
  6.   You may obtain a copy of the License at
  8.     http://www.apache.org/licenses/LICENSE-2.0
  10.   Unless required by applicable law or agreed to in writing, software
  11.   distributed under the License is distributed on an "AS IS" BASIS,
  12.   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  13.   See the License for the specific language governing permissions and
  14.   limitations under the License. See accompanying LICENSE file.
  15. -->
  16.  
  17. <!-- Put site-specific property overrides in this file. -->
  18.  
  19. <configuration>
  20.          <!-- 指定mr框架为yarn方式 -->
  21.     <property>
  22.         <name>mapreduce.framework.name</name>
  23.         <value>yarn</value>
  24.     </property>
  25. </configuration>
mapred-site.xml
  1. [root@jiang01 servers]#  hadoop version
  2. Hadoop 2.8.5
  3. Subversion https://git-wip-us.apache.org/repos/asf/hadoop.git -r 0b8464d75227fcee2c6e7f2410377b3d53d3d5f8
  4. Compiled by jdu on 2018-09-10T03:32Z
  5. Compiled with protoc 2.5.0
  6. From source with checksum 9942ca5c745417c14e318835f420733
  7. This command was run using /export/servers/hadoop-2.8.5/share/hadoop/common/hadoop-common-2.8.5.jar
  8. [root@jiang01 servers]#
查看hadoop版本

启动zk

  1. [root@jiang01 servers]# 
  2. [root@jiang01 servers]# xcall.sh jps -l
  3. ============= jiang01 : jps -l ============
  4. 10262 org.apache.zookeeper.server.quorum.QuorumPeerMain
  5. 10571 sun.tools.jps.Jps
  6. 命令执行成功
  7. ============= jiang02 : jps -l ============
  8. 10162 sun.tools.jps.Jps
  9. 9991 org.apache.zookeeper.server.quorum.QuorumPeerMain
  10. 命令执行成功
  11. ============= jiang03 : jps -l ============
  12. 2275 org.apache.zookeeper.server.quorum.QuorumPeerMain
  13. 2436 sun.tools.jps.Jps
  14. 命令执行成功
  15. [root@jiang01 servers]# xcall.sh zkServer.sh status
  16. ============= jiang01 : zkServer.sh status ============
  17. ZooKeeper JMX enabled by default
  18. Using config: /export/servers/zookeeper-3.4.14/bin/../conf/zoo.cfg
  19. Mode: follower
  20. 命令执行成功
  21. ============= jiang02 : zkServer.sh status ============
  22. ZooKeeper JMX enabled by default
  23. Using config: /export/servers/zookeeper-3.4.14/bin/../conf/zoo.cfg
  24. Mode: leader
  25. 命令执行成功
  26. ============= jiang03 : zkServer.sh status ============
  27. ZooKeeper JMX enabled by default
  28. Using config: /export/servers/zookeeper-3.4.14/bin/../conf/zoo.cfg
  29. Mode: follower
  30. 命令执行成功
  31. [root@jiang01 servers]#
启动zk

在你配置的各个journalnode节点启动该进程

  1. [root@jiang01 servers]# 
  2. [root@jiang01 servers]# xcall.sh hadoop-daemon.sh start journalnode
  3. ============= jiang01 : hadoop-daemon.sh start journalnode ============
  4. starting journalnode, logging to /export/servers/hadoop-2.8.5/logs/hadoop-root-journalnode-jiang01.out
  5. 命令执行成功
  6. ============= jiang02 : hadoop-daemon.sh start journalnode ============
  7. starting journalnode, logging to /export/servers/hadoop-2.8.5/logs/hadoop-root-journalnode-jiang02.out
  8. 命令执行成功
  9. ============= jiang03 : hadoop-daemon.sh start journalnode ============
  10. starting journalnode, logging to /export/servers/hadoop-2.8.5/logs/hadoop-root-journalnode-jiang03.out
  11. 命令执行成功
  12. [root@jiang01 servers]#
启动journalnode

 

 

 先选取一个namenode（jiang01）节点进行格式化

  1. [root@jiang01 servers]# hadoop namenode -format
View Code

 

 

 

格式化zkfc，只能在nameonde节点进行

主节点上面启动 dfs文件系统：

  1. [root@jiang01 dfs]# start-dfs.sh

 

 

 jiang002启动yarm

  1. [root@jiang02 mapreduce]# start-yarn.sh
  2. starting yarn daemons
  3. starting resourcemanager, logging to /export/servers/hadoop-2.8.5/logs/yarn-root-resourcemanager-jiang02.out
  4. jiang03: starting nodemanager, logging to /export/servers/hadoop-2.8.5/logs/yarn-root-nodemanager-jiang03.out
  5. jiang01: starting nodemanager, logging to /export/servers/hadoop-2.8.5/logs/yarn-root-nodemanager-jiang01.out
  6. jiang02: starting nodemanager, logging to /export/servers/hadoop-2.8.5/logs/yarn-root-nodemanager-jiang02.out
  7. [root@jiang02 mapreduce]#
View Code

jiang03启动：resourcemanager

  1. [root@jiang03 hadoopDatas]#  yarn-daemon.sh start resourcemanager
  2. starting resourcemanager, logging to /export/servers/hadoop-2.8.5/logs/yarn-root-resourcemanager-jiang03.out
View Code

hadoop wordcount程序启动：

1  cd /export/servers/hadoop-2.8.5/share/hadoop/mapreduce/

2 生成数据文件：

  1. touch word.txt
  2. echo "hello world" >> word.txt
  3. echo "hello hadoop" >> word.txt
  4. echo "hello hive" >> word.txt

3 创建hadoop 文件目录

  1. hdfs dfs -mkdir -p /work/data/input

4 向hadoop上传数据文件

  1. hdfs dfs -put ./word.txt /work/data/input

5 计算例子

  1. hadoop jar hadoop-mapreduce-examples-2.8.5.jar wordcount /work/data/input /work/data/output

6 查看结果：

  1. [root@jiang01 mapreduce]# hadoop jar hadoop-mapreduce-examples-2.8.5.jar wordcount /work/data/input /work/data/output
  2. 19/10/09 11:44:48 INFO input.FileInputFormat: Total input files to process : 1
  3. 19/10/09 11:44:48 INFO mapreduce.JobSubmitter: number of splits:1
  4. 19/10/09 11:44:48 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1570635804389_0001
  5. 19/10/09 11:44:48 INFO impl.YarnClientImpl: Submitted application application_1570635804389_0001
  6. 19/10/09 11:44:48 INFO mapreduce.Job: The url to track the job: http://jiang02:8088/proxy/application_1570635804389_0001/
  7. 19/10/09 11:44:48 INFO mapreduce.Job: Running job: job_1570635804389_0001
  8. 19/10/09 11:45:00 INFO mapreduce.Job: Job job_1570635804389_0001 running in uber mode : false
  9. 19/10/09 11:45:00 INFO mapreduce.Job:  map 0% reduce 0%
  10. 19/10/09 11:45:11 INFO mapreduce.Job:  map 100% reduce 0%
  11. 19/10/09 11:45:20 INFO mapreduce.Job:  map 100% reduce 100%
  12. 19/10/09 11:45:20 INFO mapreduce.Job: Job job_1570635804389_0001 completed successfully
  13. 19/10/09 11:45:21 INFO mapreduce.Job: Counters: 49
  14.         File System Counters
  15.                 FILE: Number of bytes read=54
  16.                 FILE: Number of bytes written=321397
  17.                 FILE: Number of read operations=0
  18.                 FILE: Number of large read operations=0
  19.                 FILE: Number of write operations=0
  20.                 HDFS: Number of bytes read=139
  21.                 HDFS: Number of bytes written=32
  22.                 HDFS: Number of read operations=6
  23.                 HDFS: Number of large read operations=0
  24.                 HDFS: Number of write operations=2
  25.         Job Counters 
  26.                 Launched map tasks=1
  27.                 Launched reduce tasks=1
  28.                 Data-local map tasks=1
  29.                 Total time spent by all maps in occupied slots (ms)=8790
  30.                 Total time spent by all reduces in occupied slots (ms)=6229
  31.                 Total time spent by all map tasks (ms)=8790
  32.                 Total time spent by all reduce tasks (ms)=6229
  33.                 Total vcore-milliseconds taken by all map tasks=8790
  34.                 Total vcore-milliseconds taken by all reduce tasks=6229
  35.                 Total megabyte-milliseconds taken by all map tasks=9000960
  36.                 Total megabyte-milliseconds taken by all reduce tasks=6378496
  37.         Map-Reduce Framework
  38.                 Map input records=3
  39.                 Map output records=6
  40.                 Map output bytes=60
  41.                 Map output materialized bytes=54
  42.                 Input split bytes=103
  43.                 Combine input records=6
  44.                 Combine output records=4
  45.                 Reduce input groups=4
  46.                 Reduce shuffle bytes=54
  47.                 Reduce input records=4
  48.                 Reduce output records=4
  49.                 Spilled Records=8
  50.                 Shuffled Maps =1
  51.                 Failed Shuffles=0
  52.                 Merged Map outputs=1
  53.                 GC time elapsed (ms)=199
  54.                 CPU time spent (ms)=1320
  55.                 Physical memory (bytes) snapshot=325742592
  56.                 Virtual memory (bytes) snapshot=4161085440
  57.                 Total committed heap usage (bytes)=198316032
  58.         Shuffle Errors
  59.                 BAD_ID=0
  60.                 CONNECTION=0
  61.                 IO_ERROR=0
  62.                 WRONG_LENGTH=0
  63.                 WRONG_MAP=0
  64.                 WRONG_REDUCE=0
  65.         File Input Format Counters 
  66.                 Bytes Read=36
  67.         File Output Format Counters 
  68.                 Bytes Written=32
View Code

 

原文链接：http://www.cnblogs.com/ideajiang/p/11645202.html

 友情链接：直通硅谷 　点职佳 　北美留学生论坛

Hadoop热门文章

  • Hive中的去重 ：distinct,group by与ROW_Number()窗口函数
  • HBASE的安装过程及运行HBASE程序的需要配置的内容
  • hadoop--Unable to load native-hadoop library for your platform解决方法
  • Hive sampling 语法之TABLESAMPLE用法理解
  • idea项目运行时报错提示Could not locate executable null\bin\winutils.exe in the Hadoop binaries
  • hadoop 无法访问50070端口问题解决
  • Hadoop运行时遇到java.io.FileNotFoundException错误的解决方法
  • KUDU数据导入尝试一：TextFile数据导入Hive,Hive数据导入KUDU
  • Hive安装
  • Hadoop 学习笔记 生态

    • Hadoop推荐文章

  • 基于Hive的大数据分析系统
  • HBase 在统一内容平台业务的优化实践
  • hive窗口分析函数使用详解系列一
  • 两台实体机器4个虚拟机节点的Hadoop集群搭建（Ubuntu版）
  • 在阿里云和腾讯云的轻量应用服务器上搭建Hadoop集群
  • Haproxy搭建 Web 群集实现负载均衡
  • HBase Compaction 原理与线上调优实践
  • Hive执行计划之hive依赖及权限查询和常见使用场景
  • 工良出品：包教会，Hadoop、Hive 搭建部署简易教程
  • Centos7系统编译Hadoop3.3.4

    • 本站QQ群：前端 618073944 | Java 606181507 | Python 626812652 | C/C++ 612253063 | 微信 634508462 | 苹果 692586424 | C#/.net 182808419 | PHP 305140648 | 运维 608723728

    W3xue 的所有内容仅供测试，对任何法律问题及风险不承担任何责任。通过使用本站内容随之而来的风险与本站无关。
    关于我们  |  意见建议  |  捐助我们  |  报错有奖  |  广告合作、友情链接（目前9元/月）请联系QQ：27243702 沸活量
    皖ICP备17017327号-2　皖公网安备34020702000426号