Sharpening a knife doesn’t miss a firewood cutter. Here’s a question:
What is the relationship between MapReduce and yarn?
A: yarn is not the next generation MapReduce (mrv2). The next generation MapReduce is exactly the same as the first generation MapReduce (mrv1) in programming interface and data processing engine (maptask and reducetask), It can be considered that mrv2 reuses these modules of mrv1. The difference is resource management and job management system. In mrv1, both resource management and job management are implemented by jobtracker, which integrates the two functions. In mrv2, the two parts are separated, Among them, the job management is implemented by applicationmaster, while the resource management is completed by the new system yard. Because yard is universal, yard can also be used as a resource management system for other computing frameworks, not only MapReduce, but also other computing frameworks, such as spark and storm. Generally speaking, we call the computing framework running on yard “x on” Such as “MapReduce on horn”, “spark on horn”, “storm on horn”, etc.
Hadoop 2.0 consists of three subsystems, namely HDFS, yarn and MapReduce. Yarn is a new resource management system, while MapReduce is only an application running on yarn. If yarn is regarded as a cloud operating system, MapReduce can be regarded as an app running on this operating system.
What’s the relationship between MapReduce and yarn? Today, we are going to formally build the environment.
System: ubuntu-12.04 (other versions are also available)
Mode: pseudo distributed
Build user: Hadoop
Hadoop-2.2.0 download address:http://mirrors.hust.edu.cn/apache/hadoop/common/hadoop-2.2.0/
Choose the installation package you need, here we choose hadoop-2.2.0 tar.gz
Attached Hadoop image link address:http://www.apache.org/dyn/closer.cgi/hadoop/common/
Statement 1: the directory where I configure hadoop-2.2.0 is / home / Hadoop
Statement 2: the yarn directory is created under / home / Hadoop. The hadoop-2.2.0 directory and Hadoop data directory are both under the yarn directory.
Statement 3: you can change / home / Hadoop to your own directory in the following build process.
Step 1: upload hadoop-2.2.0 tar.gz And extract it to / home / Hadoop / yarn directory. At this time, extract hadoop-2.2.0 directory from yarn directory
sudo chown -R hadoop:hadoop hadoop-2.2.0
Create Hadoop data directory:
mkdir -p /home/hadoop/yarn/yarn_data/hdfs/namenode mkdir -p /home/hadoop/yarn/yarn_data/hdfs/datanode
Before setting up the configuration file, let’s give a general introduction to each folder in hadoop-2.2.0 directory, and pay attention to the differences between hadoop-2.2.0 and hadoop-1.
The outer startup script is in the SBIN directory The called scripts of the inner layer are in the bin directory Native so files are all in the Lib / native directory The configuration program files are placed in libexec The configuration files are all in the etc directory, corresponding to the conf directory of the previous version All jar packages are in the share / Hadoop directory
Step 2: configure environment variables
I didn’t make the environment global, so I didn’t configure the system environment / etc / profile in hadoop-2.2.0
If configured, execute
source /etc/profileTo make it effective.
Step 3: Core- site.xml hdfs- site.xml mapred- site.xml yarn- site.xml to configure
Next, we will configure it in the / home / Hadoop / yarn / hadoop-2.2.0/etc/hadoop directory.
core- site.xml to configure
<configuration > <property> <name>fs.default.name</name> <value>hdfs://localhost:9000</value> < description > specify the IP address and port number of namenode </property> </configuration>
<configuration> <property> <name>dfs.replication</name> <value>2</value> < description > number of backups </property> <property> <name>dfs.namenode.name.dir</name> <value>file:/home/hadoop/yarn/yarn_data/hdfs/namenode</value> </property> <property> <name>dfs.datanode.data.dir</name> <value>file:/home/hadoop/yarn/yarn_data/hdfs/datanode</value> </property> </configuration>
<configuration> <property> <name>mapreduce.framework.name</name> <value>yarn</value> </property> <property> <name>mapreduce.jobhistory.address</name> <value>localhost:10020</value> </property> <property> <name>mapreduce.jobhistory.webapp.address</name> <value>localhost:19888</value> </property> </configuration>
<configuration> <!-- Site specific YARN configuration properties --> <property> <name>yarn.resourcemanager.address</name> <value>localhost:8032</value> </property> <property> <name>yarn.resourcemanager.scheduler.address</name> <value>localhost:8030</value> </property> <property> <name>yarn.resourcemanager.resource-tracker.address</name> <value>localhost:8031</value> </property> <property> <name>yarn.resourcemanager.admin.address</name> <value>localhost:8033</value> </property> <property> <name>yarn.resourcemanager.webapp.address</name> <value>localhost:8088</value> </property> <property> <name>yarn.nodemanager.aux-services</name> <value>mapreduce_shuffle</value> </property> <property> <name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name> <value>org.apache.hadoop.mapred.ShuffleHandler</value> </property> </configuration>
Step 4: slave configuration
Because it is pseudo distributed, we only have localhost
Step 5: synchronize the configured hadoop-2.2.0 distribution to each data node
Because it’s pseudo distributed, skip this step.
Step 6: format namenode
bin/hdfs namenode –format
bin/hadoop namenode –format
Step 7: start HDFS and yarn
Or it can be executed
Start HDFS and yarn together.
In addition, you need to start the history service, otherwise you cannot open the history link in the panel.
sbin/mr-jobhistory-daemon.sh start historyserver
Next, use the JPS command to view the startup process:
4504 ResourceManager 4066 DataNode 4761 NodeManager 5068 JobHistoryServer 4357 SecondaryNameNode 3833 NameNode 5127 Jps
Step 8: Test
Create a file in HDFS: bin / Hadoop FS - MKDIR / wordcount Upload file to HDFS: bin / Hadoop FS / home / Hadoop / file2.txt / wordcount View the HDFS file directory: HDFS DFS – LS/
Yarn test: run wordcount test program,
bin/hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.2.0.jar wordcount /wordcount /output2
bin/hadoop fs -cat /output2/*
The results are as follows
hadoop 1 hello 2 java 4 jsp 1
Here, the hadoop-2.2.0 environment is finished, and the configuration files are configured according to the specific requirements. There may be improper configuration. If you see it, please correct it.