Windows 10编译Hadoop 3.3.4


1 环境准备

1.1 Hadoop

Apache Hadoop 3.3.4:https://archive.apache.org/dist/hadoop/common/hadoop-3.3.4/hadoop-3.3.4-src.tar.gz

1.2 GIT

git version 2.37.0.windows.1:https://github.com/git-for-windows/git/releases/download/v2.47.1.windows.1/PortableGit-2.47.1-64-bit.7z.exe
解压后设置环境变量%GIT_HOME%,以及把%GIT_HOME%\bin、%GIT_HOME%\usr\bin、%GIT_HOME%\mingw64\bin、%GIT_HOME%\mingw64\libexec\git-core添加到%PATH%中。

1.3 Maven

Apache Maven 3.8.4:https://archive.apache.org/dist/maven/maven-3/3.8.4/binaries/apache-maven-3.8.4-bin.zip
解压后设置环境变量%MAVEN_HOME%,以及把%MAVEN_HOME%\bin添加到%PATH%中。

1.4 JDK

Java SE Development Kit 8u421:https://www.oracle.com/java/technologies/javase/javase8u211-later-archive-downloads.html
解压后设置环境变量%JAVA_HOME%,以及把%JAVA_HOME%\bin添加到%PATH%中。

1.5 CMAKE

cmake 3.31.1:https://github.com/Kitware/CMake/releases/download/v3.31.1/cmake-3.31.1-windows-x86_64.zip
解压后设置环境变量%CMAKE_HOME%,以及把%CMAKE_HOME%\bin添加到%PATH%中。

1.6 ZLIB

zlib 1.3.1:https://www.zlib.net/current/zlib.tar.gz
解压后设置环境变量%ZLIB_HOME%。

1.7 Protoc

Protocol Buffers v3.7.1:https://github.com/protocolbuffers/protobuf/releases/download/v3.7.1/protoc-3.7.1-win64.zip
解压后设置环境变量%PROTOCOL_HOME%,以及把%PROTOCOL_HOME%\bin添加到%PATH%中。

1.8 Yarn

yarn 1.22.5:https://github.com/yarnpkg/yarn/releases/download/v1.22.5/yarn-v1.22.5.tar.gz
将文件名修改为yarn-1.22.5.tar.gz,然后放到目录Repository\com\github\eirslett\yarn\1.22.5中去,其中Repository就是Maven定义的本地仓库根目录。

1.9 Node

Node 14.0.0:https://nodejs.org/dist/v14.0.0/win-x64/node.exe
将文件名修改为node-14.0.0-win-x64.exe,然后放到目录Repository\com\github\eirslett\node\14.0.0中支,其中Repository就是Maven定义的本地仓库根目录。

1.10 Visual Studio 2019

需要安装Visual Studio 2019社区版,并且工作负荷界面勾选使用C++的桌面开发。
安装完成以后,新增%MSBUILD_HOME%路径(默认为C:\Program Files (x86)\MSBuild\14.0),并将%MSBUILD_HOME%\Bin添加到%PATH%中。
新增环境变量%Platform%,设置值为x64。

2 修改源码

2.1 修改pom.xml

vim pom.xml
<shell-executable>bash</shell-executable>
修改为
<shell-executable>sh</shell-executable>

2.2 修改failonerror

vim hadoop-hdfs-project\hadoop-hdfs-native-client\pom.xml
148行、154行
failonerror="true"
修改为
failonerror="false"

2.3 建立目录

mkdir -p hadoop-hdfs-project\hadoop-hdfs-native-client\target\native\bin\RelWithDebInfo

2.4 修改yarn配置文件

vim hadoop-yarn-project\hadoop-yarn\hadoop-yarn-applications\hadoop-yarn-applications-catalog\hadoop-yarn-applications-catalog-webapp\.yarnrc
--registry "https://registry.npmmirror.com/"
--modules-folder target/generated-sources/vendor

2.5 修改node版本

vim hadoop-yarn-project\hadoop-yarn\hadoop-yarn-applications\hadoop-yarn-applications-catalog\hadoop-yarn-applications-catalog-webapp\pom.xml
修改270行
<nodeVersion>${nodejs.version}</nodeVersion>
修改为
<nodeVersion>14.0.0</nodeVersion>

3 编译

使用管理员权限运行x64 Native Tools Command Prompt for VS 2019,进入到源代码目录,执行以下编译命令:
mvn package -e -X -Pdist,native-win -DskipTests -Dtar -Dmaven.javadoc.skip=true -Dmaven.wagon.http.ssl.insecure=true -Dmaven.wagon.http.ssl.allowall=true
编译目标文件:hadoop-dist\target\hadoop-3.3.4.tar.gz

,

发表回复

您的电子邮箱地址不会被公开。 必填项已用 * 标注