Flink on yarn: deployment and startup of Flink cluster

Time:2020-11-23
Deployment of Flink cluster

There are three deployment modes of Flink, which are local, standalone cluster and Yan cluster. Here we mainly talk about how to configure the Yan cluster.

Before configuring Flink on yarn, you must ensure that both HDFS and yarn are turned onFor Hadoop cluster deployment and start-up, container memory resource allocation should be considered in yarn mode

Installation version:flink-1.7.1-bin-hadoop28-scala_2.11.tgz

mkdir /usr/local/flink
tar zxvf flink-1.7.1-bin-hadoop28-scala_2.11.tgz -C /usr/local/flink

Modify the corresponding relationship between domain name and IP (hadoop2 and hadoop3 also need to modify the hosts file)

vi /etc/hosts
10.2.15.176 hadoop1
10.2.15.177 hadoop2
10.2.15.170 hadoop3

Configure environment variables (Hadoop 2 and Hadoop 3 also need to modify the hosts file)

vi /etc/profile
export FLINK_HOME=/usr/local/flink/flink-1.7.1
export PATH=$PATH:$FLINK_HOME/bin
export HADOOP_HOME=/usr/local/hadoop/hadoop-2.8.3
export PATH=$HADOOP_HOME/bin:$PATH
source /etc/profile

Modify Flink- conf.yaml file

vi /usr/local/flink/flink-1.7.1/conf/flink-conf.yaml
#==============================================================================
# Common
#==============================================================================
jobmanager.rpc.port: 6123
jobmanager.heap.size: 1024m
taskmanager.heap.size: 1024m
taskmanager.numberOfTaskSlots: 1
parallelism.default: 1
env.java.home: /usr/local/jdk/jdk1.8.0_251
jobmanager.heap.mb: 6192m
taskmanager.heap.mb: 8192m
#==============================================================================
# High Availability
#==============================================================================
high-availability: zookeeper
high-availability.storageDir: hdfs:///flink/ha/
high-availability.zookeeper.quorum: 10.2.15.181:2181,10.2.15.174:2181,10.2.15.172:2181
high-availability.zookeeper.path.root: /flink_on_yarn
high-availability.zookeeper.path.namespace: /cluster_yarn
#==============================================================================
# Fault tolerance and checkpointing
#==============================================================================
state.backend: filesystem
state.backend.fs.checkpointdir: hdfs:///flink/checkpoints
#==============================================================================
# Web Frontend
#==============================================================================
rest.port: 8081
#==============================================================================
# Advanced
#==============================================================================
taskmanager.memory.preallocate: false
taskmanager.network.numberOfBuffers: 64000
fs.hdfs.hadoopconf: /usr/local/hadoop/hadoop-2.8.5/etc/hadoop

Modify the master and slave files

vi conf/masters
hadoop1:8081
hadoop2:8081
vi conf/slaves
hadoop2
hadoop3
Submit job

First start zookeeper JPS arbitration

./start-zookeeper-quorum.sh

Then start the per job cluster task, which can be accessed through the./bin/flink run -m yarn-cluster -d -c mainClass /path/to/user/jarThe command starts a cluster in split mode, which is a single task single cluster

./bin/flink run -m yarn-cluster  ./examples/batch/WordCount.jar -input hdfs://hadoop1:9000/input/input_hadoop_demo_test.txt  -output hdfs://hadoop1:9000/output/wordcount-result1.txt
or 
./bin/flink run -m yarn-cluster -yn 2 -yjm 800 -ytm 800  ./examples/batch/WordCount.jar -input hdfs://hadoop1:9000/input/input_hadoop_demo_test.txt  -output hdfs://hadoop1:9000/output/wordcount-result2.txt

Enter in browserhttp://hadoop1:8088, you can view the relevant information

Recommended Today

Summary of recent use of gin

Recently, a new project is developed by using gin. Some problems are encountered in the process. To sum up, as a note, I hope it can help you. Cross domain problems Middleware: func Cors() gin.HandlerFunc { return func(c *gin.Context) { //Here you can use * or the domain name you specify c.Header(“Access-Control-Allow-Origin”, “*”) //Allow header […]