Hadoop 3.1.1 fully distributed installation guide under CentOS 6.8 (recommended)

Time:2021-3-6

Above:

This document is based on the three virtual machines Ping each other, firewall closed, hosts file modification, SSH password free login, host name modification and so on.

1、 Incoming file

1. Create installation directory

mkdir /usr/local/soft

2. Open xftp, find the corresponding directory and pass in the required installation package

Check the installation package: CD / usr / local / soft

2、 Install Java

1. Check if JDK: Java – version is installed

2. Not installed, decompress Java installation package: tar – zxvf jdk-8u181-linux-x64 tar.gz

(everyone’s installation package may be different, for your own reference)

3. Rename JDK and check the current location: MV jdk1.8.0_ 181 java

4. Configure JDK environment: VIM / etc / profile. D/ jdk.sh


export JAVA_HOME=/usr/local/soft/java
export PATH=$PATH:$JAVA_HOME/bin
export CLASSPATH=.:$JAVA_HOME/lib/tools.jar:$JAVA_HOME/lib/rt.jar

5. Update environment variables and check: source / etc / profile

3、 Install Hadoop

1. Unzip the Hadoop installation package: tar – zxvf hadoop-3.1.1 tar.gz

2. View and rename: MV hadoop-3.1.1 Hadoop

3. Configure Hadoop configuration file

3.1 modify core- site.xml Configuration file: VIM Hadoop / etc / Hadoop / core- site.xml


<property>

<name>fs.defaultFS</name>

<value>hdfs://master:9000</value>

</property>

<property>

<name>hadoop.tmp.dir</name>

<value>file:/usr/local/soft/hadoop/tmp</value>

<description>Abase for other temporary directories.</description>

</property>

<property>

 <name>fs.trash.interval</name>

 <value>1440</value>

</property>

3.2 modify HDFS- site.xml Configuration file: VIM Hadoop / etc / Hadoop / HDFS- site.xml


<property>
<name>dfs.namenode.secondary.http-address</name>
<value>node1:50090</value>
</property>
<property>
<name>dfs.replication</name>
<value>3</value>
</property>
<property>
<name>dfs.namenode.name.dir</name>
<value>file:/usr/local/soft/hadoop/tmp/dfs/name</value>
</property>
<property>
<name>dfs.datanode.data.dir</name>
<value>file:/usr/local/soft/hadoop/tmp/dfs/data</value>
</property>

3.3 modify the workers configuration file: VIM Hadoop / etc / Hadoop / workers

3.4 modify Hadoop- env.sh File: VIM Hadoop / etc / Hadoop / Hadoop- env.sh

export JAVA_HOME=/usr/local/soft/java

3.5 modify yarn- site.xml File: VIM Hadoop / etc / Hadoop / yarn- site.xml


<property>
<name>yarn.resourcemanager.hostname</name>
<value>master</value>
</property>
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>

3.6 update configuration file: source Hadoop / etc / Hadoop / Hadoop- env.sh

3.7 modify start- dfs.sh Configuration file: Im Hadoop / SBIN / start- dfs.sh


export HDFS_NAMENODE_SECURE_USER=root
export HDFS_DATANODE_SECURE_USER=root
export HDFS_SECONDARYNAMENODE_USER=root
export HDFS_NAMENODE_USER=root
export HDFS_DATANODE_USER=root
export HDFS_SECONDARYNAMENODE_USER=root
export YARN_RESOURCEMANAGER_USER=root
export YARN_NODEMANAGER_USER=root

3.8 modify stop- dfs.sh Configuration file: VIM Hadoop / SBIN / stop- dfs.sh


export HDFS_NAMENODE_SECURE_USER=root
export HDFS_DATANODE_SECURE_USER=root
export HDFS_SECONDARYNAMENODE_USER=root
export HDFS_NAMENODE_USER=root
export HDFS_DATANODE_USER=root
export HDFS_SECONDARYNAMENODE_USER=root
export YARN_RESOURCEMANAGER_USER=root
export YARN_NODEMANAGER_USER=root

3.9 modify start- yarn.sh Configuration file: VIM Hadoop / SBIN / start- yarn.sh


export YARN_RESOURCEMANAGER_USER=root
export HADOOP_SECURE_DN_USER=root
export YARN_NODEMANAGER_USER=root

3.10 modify stop- yarn.sh Configuration file: VIM Hadoop / SBIN / stop- yarn.sh


export YARN_RESOURCEMANAGER_USER=root
export HADOOP_SECURE_DN_USER=root
export YARN_NODEMANAGER_USER=root

3.11 cancel printing warning message: VIM Hadoop / etc / Hadoop / log4j.properties

log4j.logger.org.apache.hadoop.util.NativeCodeLoader=ERROR

4、 Synchronization configuration information:

1. Synchronous node1: SCP – R soft [email protected] :/usr/local/

Synchronous node2: SCP – R soft [email protected] :/usr/local/

2. Wait for all transfers to complete and configure the profile file: VIM / etc / profile. D/ hadoop.sh


#SET HADOOP
export HADOOP_HOME=/usr/local/soft/hadoop
export HADOOP_INSTALL=$HADOOP_HOME
export HADOOP_MAPRED_HOME=$HADOOP_HOME
export HADOOP_COMMON_HOME=$HADOOP_HOME
export HADOOP_HDFS_HOME=$HADOOP_HOME
export YARN_HOME=$HADOOP_HOME
export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native
export PATH=$PATH:$HADOOP_HOME/sbin:$HADOOP_HOME/bin
export PATH=$PATH:$HADOOP_HOME/sbin:$HADOOP_HOME/bin

3. Continue transmission

For node1: SCP / etc / profile. D/ jdk.sh [email protected] :/etc/profile.d/

scp /etc/profile.d/hadoop.sh [email protected]:/etc/profile.d/

For node2: SCP / etc / profile. D/ jdk.sh [email protected] :/etc/profile.d/

scp /etc/profile.d/hadoop.sh [email protected]:/etc/profile.d/

4. Execute on all three virtual machines

source /etc/profile

source /usr/local/soft/hadoop/etc/hadoop/hadoop-env.sh

(only one display)

5. Format HDFS file system: HDFS namenode – Format (only on Master)

5、 Start cluster

cd /usr/local/soft/hadoop/sbin/

./start-all.sh

After starting, input JPS on three virtual machines respectively

The results are as follows

Google Browser Test under Windows

http://192.168.204.120 : 8088 / cluster (enter the IP address of your master)

http://192.168.204.120:9870

Hadoop test (MapReduce execution calculation test)

hadoop jar/usr/local/soft/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-3.1.1.jar wordcount /input /output

To view the running results:

The above Hadoop configuration is completed.

summary

The above is the complete distributed installation guide of Hadoop 3.1.1 under centos6.8 introduced by Xiaobian. I hope it can help you. If you have any questions, please leave me a message and Xiaobian will reply you in time. Thank you very much for your support to developer!
If you think this article is helpful to you, please reprint, please indicate the source, thank you!