Detailed explanation of Hadoop 2.7.2 compiling 64 bit source code

Time:2020-7-29

1、 Environmental preparation

1. CentOS configuration

The best way is to use the newly cloned virtual machine, set the virtual machine memory to a bit larger (4G I set), configure the network, host name, turn off the firewall, and close SELinux
Note: use root role compilation to reduce the folder permissions

2. Jar package preparation (Hadoop source code, jdk8, maven, ant, protobuf)

(1)hadoop-2.7.2-src.tar.gz
(2)jdk-8u144-linux-x64.tar.gz
(3)apache-ant-1.9.9- bin.tar.gz (build tool, for packing)
(4)apache-maven-3.0.5-bin.tar.gz
(5)protobuf-2.5.0. tar.gz (serialized framework)

3. Jar package installation (Note: all operations must be completed under the root user)

JDK

1. Decompression

tar -zxf jdk-8u141-linux-x64.tar.gz -C /export/compile

2. Configure environment variables


vim /etc/profile
export JAVA_HOME=/export/compile/jdk1.8.0_141
export PATH=$PATH:$JAVA_HOME/bin

3. Update the profile file

source /etc/profile

4. Verify that the installation is successful

Verification command: JPS the JPS process indicates that the installation and configuration is successful

2.Maven

1. Decompress & Rename


tar -zxf apache-maven-3.0.5-bin.tar.gz -C /export/compile
mv /export/compile/apache-maven-3.0.5-bin.tar.gz /export/compile/maven

2. Modify the configuration file

vim /export/compile/maven/conf/settings.xml

Add alicloud image address:


<mirrors>
 <mirror>
 <id>nexus-aliyun</id>
 <mirrorOf>central</mirrorOf>
 <name>Nexus aliyun</name>
 <url>http://maven.aliyun.com/nexus/content/groups/public</url>
 </mirror>
</mirrors>

3. Configure environment variables


vim /etc/profile
export MAVEN_HOME=/export/compile/maven
export PATH=$PATH:$MAVEN_HOME/bin

4. Update the profile file

source /etc/profile

5. Verify that the installation is successful

Validation command:mvn -version   The presence of version information indicates that the installation was successful

3.ant

1. Decompress & Rename


tar -zxf apache-ant-1.9.9-bin.tar.gz -C /export/compile
mv apache-ant-1.9.9 ant

2. Configure environment variables


vim /etc/profile
export ANT_HOME=/export/compile/ant
export PATH=$PATH:$ANT_HOME/bin

3. Update the profile file

source /etc/profile

4. Verify that the installation is successful

Verification command: ant – version version version information shows that the installation is successful

4.glibc-headers


yum -y install glibc-headers
yum -y install gcc-c++

5. Make and cmake


yum -y install make
yum -y install cmake

6.protobuf

1. Unzip & Rename & enter the root directory of protobuf


tar -zxf protobuf-2.5.0.tar.gz -C /export/compile
mv protobuf-2.5.0/ protobuf
cd /export/compile/protobuf

2. Execute the following commands in turn

1 ./configure
2 make
3 make check
4 make install
5 ldconfig

3. Configure environment variables


vim /etc/profile
export LD_LIBRARY_PATH=/export/compile/protobuf
export PATH=$PATH:$LD_LIBRARY_PATH

4. Update the profile file

source /etc/profile

5. Verify that the installation is successful

Validation command:protoc --version  The presence of version information indicates that the installation was successful

7. Install OpenSSL library

yum -y install openssl-devel

8. Install the ncurses devel library

yum -y install ncurses-devel

2、 Compile source code

1. Decompression

tar -zxf hadoop-2.7.2-src.tar.gz -C /export/compile

2. Enter Hadoop source code main directory

/export/compile/hadoop-2.7.2-src

3. Execute the compile command through Maven

mvn package -Pdist,native -DskipTests -Dtar

4. Then there is a long wait… (the waiting time is about 30 minutes, and the final success is all success, as shown in the figure below)

5. Successful 64 bit Hadoop package/export/compile/hadoop-2.7.2-src/hadoop-dist/targetUnder the table of contents

3、 Common problems and solutions in the process of compiling source code

(1) JVM memory overflow during Maven install

Reason and treatment: Maven can be adjusted in the environment configuration file and Maven execution file_ The heap size of opt.
(for details, please refer to Maven compilation JVM tuning issues, such as: http://outofmemory.cn/code-snippet/12652/maven-outofmemoryerror-method )

(2) Maven reported an error during compilation

Causes and solutions: possible network congestion may lead to incomplete download of dependent library, and multiple command execution (one pass is more difficult)

mvn package -Pdist,nativeN -DskipTests -Dtar

(3) Report ant, protobuf and other errors

Reasons and solutions: the plug-in download is not complete or the plug-in version problem. There are many special cases in the initial link.

summary

This article about Hadoop 2.7.2 64 bit source code compilation article introduced here, more related Hadoop source code content please search the previous article or continue to browse the following related articles, I hope you can support developeppaer more in the future!