Learning Flink from 0 to 1 – how does the Flink project run?

Time:2021-9-21

Learning Flink from 0 to 1 - how does the Flink project run?

preface

I have written many Flink articles and many demos before, but when writing the articles, I directly run the main method of the main class locally. In fact, Flink supports uploading the jar package of Flink job on the UI and running smoothly. It started in the first articleLearning Flink from 0 to 1 — an introduction to building Flink 1.6.0 environment and building and running simple programs on MACIn fact, we mentioned Flink’s own UI interface in. Today, let’s see how to package our project and publish it here.

get ready

Compile package

The project code takes my previous articleLearning Flink from 0 to 1 – Flink writes data to elasticsearchWell, the code address is the GitHub warehouse address:https://github.com/zhisheng17/flink-learning/tree/master/flink-learning-connectors/flink-learning-connectors-es6, if you are interested, you can directly pack it and test the water.

We execute the following command in the folder of pom.xml of the whole project (Flink learning):

mvn clean install

Learning Flink from 0 to 1 - how does the Flink project run?

Then you will find that in the target directory of flink-learning-connectors-es6, there is flink-learning-connectors-es6-1.0-snapshot.jar.

Start es

Note that your Kafka data source and es have been started, and the data in the ES directory has been cleared to check whether there is really data stored.

Learning Flink from 0 to 1 - how does the Flink project run?

Submit jar package

Submit this file to flinkserver, as shown below:

Learning Flink from 0 to 1 - how does the Flink project run?

Click the “Upload” button in the red box below:

Learning Flink from 0 to 1 - how does the Flink project run?

As shown in the figure below, select the file just uploaded, fill in the class name, and then click the “submit” button to start the job:

Learning Flink from 0 to 1 - how does the Flink project run?

View run results

As shown in the following figure, the running tasks can be seen on the overview page:

Learning Flink from 0 to 1 - how does the Flink project run?

You can see the metric data about the task in the task manager
, log information, and stdout information.

Learning Flink from 0 to 1 - how does the Flink project run?

Check kibana, and there is already data in es:

Learning Flink from 0 to 1 - how does the Flink project run?

We can cancel the job in overview on the Flink UI interface, and then we can see the job log:

Learning Flink from 0 to 1 - how does the Flink project run?

Learning Flink from 0 to 1 - how does the Flink project run?

summary

This article describes how to compile and package our job and submit it to Flink to run on the server UI. It is also a supplement to the previous article. Of course, Flink job not only supports the operation of this mode, but also can run on k8s, mesos, etc. I will write it later.

The original address of this article is:http://www.54tianzhisheng.cn/2019/01/05/Flink-run/, reprinting without permission is prohibited.

Pay attention to me

The official account of WeChat:zhisheng

In addition, I have compiled some Flink learning materials, and I have put all the official account of WeChat. You can add my wechat:zhisheng_tian, and then reply to the keyword:FlinkYou can get it unconditionally.

Learning Flink from 0 to 1 - how does the Flink project run?

GitHub code warehouse

https://github.com/zhisheng17/flink-learning/

In the future, all the code of this project will be put in this warehouse, including some demos and blogs for learning Flink

Related articles

1、Learning Flink from 0 to 1 – Introduction to Apache Flink

2、Learning Flink from 0 to 1 — an introduction to building Flink 1.6.0 environment and building and running simple programs on MAC

3、Learn Flink from 0 to 1 – detailed explanation of Flink profile

4、Learning Flink from 0 to 1 – Introduction to data source

5、Learn Flink from 0 to 1 – how to customize the data source?

6、Learning Flink from 0 to 1 – Introduction to data sink

7、Learn Flink from 0 to 1 – how to customize data sink?

8、Learning Flink from 0 to 1 – Flink data transformation

9、“Learning Flink from 0 to 1” — introducing stream windows in Flink

10、Learning Flink from 0 to 1 — detailed explanations of several times in Flink

11、Learning Flink from 0 to 1 – Flink writes data to elasticsearch

12、Learning Flink from 0 to 1 – how does the Flink project run?

13、Learning Flink from 0 to 1 – Flink writes data to Kafka

14、Learning Flink from 0 to 1 – Flink jobmanager high availability configuration

15、Learning Flink from 0 to 1 — Introduction to Flink parallelism and slot

16、Learning Flink from 0 to 1 – Flink reads Kafka data and writes it to MySQL in batches

17、Learning Flink from 0 to 1 – Flink reads Kafka data and writes it to rabbitmq

18、Learning Flink from 0 to 1 – where is the jar package you uploaded?