经验首页 前端设计 程序设计 Java相关 移动开发 数据库/运维 软件/图像 大数据/云计算 其他经验
当前位置：技术经验 » 大数据/云/AI » Hadoop » 查看文章
Centos7系统编译Hadoop3.3.4
来源：cnblogs　　作者：huan1993　　时间：2023/2/20 15:18:11　　对本文有异议

1、背景

最近在学习hadoop，此篇文章简单记录一下通过源码来编译hadoop。为什么要重新编译hadoop源码，是因为为了匹配不同操作系统的本地库环境。

2、编译源码

2.1 下载并解压源码

源码路径

  1. [root@hadoop01 ~]# mkdir /opt/hadoop
  2. [root@hadoop01 ~]# cd /opt/hadoop/
  3. [root@hadoop01 hadoop]# wget https://archive.apache.org/dist/hadoop/common/hadoop-3.3.4/hadoop-3.3.4-src.tar.gz
  4. [root@hadoop01 hadoop]# tar -zxvf hadoop-3.3.4-src.tar.gz
  5. [root@hadoop01 hadoop]# rm -rvf hadoop-3.3.4-src.tar.gz

2.2 查看编译hadoop必要的环境

  1. [root@hadoop01 hadoop]# pwd
  2. /opt/hadoop
  3. [root@hadoop01 hadoop]# cd hadoop-3.3.4-src/
  4. [root@hadoop01 hadoop-3.3.4-src]# cat BUILDING.txt
  5. Build instructions for Hadoop
  7. ----------------------------------------------------------------------------------
  8. Requirements:
  10. * Unix System
  11. * JDK 1.8
  12. * Maven 3.3 or later
  13. * Protocol Buffers 3.7.1 (if compiling native code)
  14. * CMake 3.1 or newer (if compiling native code)
  15. * Zlib devel (if compiling native code)
  16. * Cyrus SASL devel (if compiling native code)
  17. * One of the compilers that support thread_local storage: GCC 4.8.1 or later, Visual Studio,
  18.   Clang (community version), Clang (version for iOS 9 and later) (if compiling native code)
  19. * openssl devel (if compiling native hadoop-pipes and to get the best HDFS encryption performance)
  20. * Linux FUSE (Filesystem in Userspace) version 2.6 or above (if compiling fuse_dfs)
  21. * Doxygen ( if compiling libhdfspp and generating the documents )
  22. * Internet connection for first build (to fetch all Maven and Hadoop dependencies)
  23. * python (for releasedocs)
  24. * bats (for shell code testing)
  25. * Node.js / bower / Ember-cli (for YARN UI v2 building)
  27. ----------------------------------------------------------------------------------

可以看到需要JDK1.8及以上、maven3.3及以上等等

2.3 安装JDK

需要注意JDK的版本,参考这个文档。https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+Java+Versions

2.4 安装maven

  1. [root@hadoop01 hadoop]# wget https://dlcdn.apache.org/maven/maven-3/3.9.0/binaries/apache-maven-3.9.0-bin.tar.gz
  2. [root@hadoop01 hadoop]# tar -zxvf apache-maven-3.9.0-bin.tar.gz -C /usr/local
  3. # 编辑环境变量
  4. [root@hadoop01 hadoop]# vim /etc/profile
  5. # 配置maven
  6. export M2_HOME=/usr/local/apache-maven-3.9.0
  7. export PATH=${M2_HOME}/bin:$PATH
  8. [root@hadoop01 hadoop]# source /etc/profile
  9. # 查看maven的版本
  10. [root@hadoop01 apache-maven-3.9.0]# mvn -version
  11. Apache Maven 3.9.0 (9b58d2bad23a66be161c4664ef21ce219c2c8584)
  12. Maven home: /usr/local/apache-maven-3.9.0
  13. Java version: 1.8.0_333, vendor: Oracle Corporation, runtime: /usr/local/jdk8/jre
  14. Default locale: zh_CN, platform encoding: UTF-8
  15. OS name: "linux", version: "5.11.12-300.el7.aarch64", arch: "aarch64", family: "unix"
  16. # 配置aliyun镜像加速访问
  17. [root@hadoop01 hadoop]# vim /usr/local/apache-maven-3.9.0/conf/settings.xml
  18. <mirrors>
  19.      <mirror>
  20.            <id>alimaven</id>
  21.            <name>aliyun maven</name>
  22.            <url>http://maven.aliyun.com/nexus/content/groups/public/</url>
  23.            <mirrorOf>central</mirrorOf>
  24.       </mirror>
  25. </mirrors>

2.5 安装编译相关的依赖

  1. [root@hadoop01 hadoop]# yum install gcc gcc-c++ make autoconf automake libtool curl lzo-devel zlib-devel openssl openssl-devel ncurses-devel snappy snappy-devel bzip2 bzip2-devel lzo lzo-devel lzop libXtst zlib doxygen cyrus-sasl* saslwrapper-devel* -y

2.6 安装cmake

hadoop要求* CMake 3.1 or newer (if compiling native code) cmake的版本在3.1及其以上。cmake不是必须的。
cmake下载

  1. # 卸载已有的cmake
  2. [root@hadoop01 hadoop]# yum erase cmake
  3. # 下载cmake(此处需要根据自己的操作系统进行下载)
  4. [root@hadoop01 hadoop]# wget https://github.com/Kitware/CMake/releases/download/v3.25.2/cmake-3.25.2.tar.gz
  5. [root@hadoop01 hadoop]# tar -zxvf cmake-3.25.2.tar.gz
  6. # 编译和安装cmake
  7. [root@hadoop01 cmake-3.25.2-linux-aarch64]# cd cmake-3.25.2/ && ./configure && make && make install
  8. # 查看cmake的版本
  9. [root@hadoop01 cmake-3.25.2]# cmake -version
  10. cmake version 3.25.2
  12. CMake suite maintained and supported by Kitware (kitware.com/cmake).

2.7 安装Snappy

* Snappy compression (only used for hadoop-mapreduce-client-nativetask)

  1. # 卸载已安装的snappy
  2. [root@hadoop01 hadoop]# rm -rf /usr/local/lib/libsnappy* && rm -rf /lib64/libsnappy*
  3. [root@hadoop01 hadoop]# wget https://src.fedoraproject.org/repo/pkgs/snappy/snappy-1.1.3.tar.gz/7358c82f133dc77798e4c2062a749b73/snappy-1.1.3.tar.gz
  4. [root@hadoop01 snappy]# tar -zxvf snappy-1.1.3.tar.gz
  5. [root@hadoop01 snappy]# cd snappy-1.1.3/ && ./configure && make && make install
  6. [root@hadoop01 build]#

snappy安装目录

2.8 安装ProtocolBuffer

* Protocol Buffers 3.7.1 (if compiling native code) 安装3.7.1的版本

  1. [root@hadoop01 hadoop]# wget https://github.com/protocolbuffers/protobuf/releases/download/v3.7.1/protobuf-java-3.7.1.tar.gz
  2. [root@hadoop01 hadoop]# tar -zxvf protobuf-java-3.7.1.tar.gz
  3. # 编译和安装
  4. [root@hadoop01 hadoop]# cd protobuf-3.7.1/ && ./autogen.sh && ./configure && make && make install
  5. # 验证是否安装成功
  6. [root@hadoop01 protobuf-3.7.1]# protoc --version
  7. libprotoc 3.7.1

2.9 编译hadoop

  1. [root@hadoop01 hadoop-3.3.4-src]# pwd
  2. /opt/hadoop/hadoop-3.3.4-src
  3. [root@hadoop01 hadoop-3.3.4-src]# export MAVEN_OPTS="-Xms3072m -Xmx3072m" && mvn clean package -Pdist,native -DskipTests -Dtar -Dbundle.snappy -Dsnappy.lib=/usr/local/lib -e

此处的mvn 命令也可以从BUILDING.txt文件中获取。

出现这个图说明编译成功了

2.10 编译后的安装包路径

hadoop-3.3.4-src/hadoop-dist/target/hadoop-3.3.4.tar.gz

2.11 检测native

  1. [root@hadoop01 hadoop]# tar -zxvf hadoop-3.3.4.tar.gz
  2. [root@hadoop01 bin]# cd hadoop-3.3.4/bin
  3. [root@hadoop01 bin]# ./hadoop checknative -a
  4. 2023-02-18 16:58:39,698 INFO bzip2.Bzip2Factory: Successfully loaded & initialized native-bzip2 library system-native
  5. 2023-02-18 16:58:39,700 INFO zlib.ZlibFactory: Successfully loaded & initialized native-zlib library
  6. 2023-02-18 16:58:39,700 WARN erasurecode.ErasureCodeNative: ISA-L support is not available in your platform... using builtin-java codec where applicable
  7. 2023-02-18 16:58:39,760 INFO nativeio.NativeIO: The native code was built without PMDK support.
  8. Native library checking:
  9. hadoop:  true /opt/hadoop/hadoop-3.3.4/lib/native/libhadoop.so.1.0.0
  10. zlib:    true /lib64/libz.so.1
  11. zstd  :  false
  12. bzip2:   true /lib64/libbz2.so.1
  13. openssl: true /lib64/libcrypto.so
  14. ISA-L:   false libhadoop was built without ISA-L support
  15. PMDK:    false The native code was built without PMDK support.
  16. 2023-02-18 16:58:39,764 INFO util.ExitUtil: Exiting with status 1: ExitException

可以看到上方还有很多false的，不过这不影响hadoop的使用。 如果要解决的话，可以安装这些依赖，然后重新编译hadoop。

3、参考文章

1、https://www.vvave.net/archives/how-to-build-hadoop-334-native-libraries-full-kit-on-amd64.html
2、https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+Java+Versions

原文链接：https://www.cnblogs.com/huan1993/p/17136933.html

 友情链接：直通硅谷 　点职佳 　北美留学生论坛

Hadoop热门文章

  • Hive中的去重 ：distinct,group by与ROW_Number()窗口函数
  • HBASE的安装过程及运行HBASE程序的需要配置的内容
  • hadoop--Unable to load native-hadoop library for your platform解决方法
  • Hive sampling 语法之TABLESAMPLE用法理解
  • idea项目运行时报错提示Could not locate executable null\bin\winutils.exe in the Hadoop binaries
  • hadoop 无法访问50070端口问题解决
  • Hadoop运行时遇到java.io.FileNotFoundException错误的解决方法
  • KUDU数据导入尝试一：TextFile数据导入Hive,Hive数据导入KUDU
  • Hive安装
  • Hadoop 学习笔记 生态

    • Hadoop推荐文章

  • 基于Hive的大数据分析系统
  • HBase 在统一内容平台业务的优化实践
  • hive窗口分析函数使用详解系列一
  • 两台实体机器4个虚拟机节点的Hadoop集群搭建（Ubuntu版）
  • 在阿里云和腾讯云的轻量应用服务器上搭建Hadoop集群
  • Haproxy搭建 Web 群集实现负载均衡
  • HBase Compaction 原理与线上调优实践
  • Hive执行计划之hive依赖及权限查询和常见使用场景
  • 工良出品：包教会，Hadoop、Hive 搭建部署简易教程
  • Centos7系统编译Hadoop3.3.4

    • 本站QQ群：前端 618073944 | Java 606181507 | Python 626812652 | C/C++ 612253063 | 微信 634508462 | 苹果 692586424 | C#/.net 182808419 | PHP 305140648 | 运维 608723728

    W3xue 的所有内容仅供测试，对任何法律问题及风险不承担任何责任。通过使用本站内容随之而来的风险与本站无关。
    关于我们  |  意见建议  |  捐助我们  |  报错有奖  |  广告合作、友情链接（目前9元/月）请联系QQ：27243702 沸活量
    皖ICP备17017327号-2　皖公网安备34020702000426号