Learn Flink from 0 to 1 – where is the jar package you uploaded?

Time:2020-2-24

Preface

Writing this article is actually a question asked by a small partner in the knowledge Planet:

Which directory is the jar submitted through the Flink UI dashboard stored in?

In fact, I have asked this question myself, but I haven’t thought about it deeply because I don’t have any pressure. Now it’s asked by the paying partner of the knowledge planet, so I naturally have to force myself to go deep before I can give the correct answer.

Follow me to see my search steps! How dare little jar play hide and seek with me?

View profile

The first thing to think about is that this can definitely be set in the configuration file:

Learn Flink from 0 to 1 - where is the jar package you uploaded?

Google Dafa

Although there is an upload directory, it’s not what we want! So I had to use my “Google method”.

Learn Flink from 0 to 1 - where is the jar package you uploaded?

Find one and click in to see the issue as follows:

Learn Flink from 0 to 1 - where is the jar package you uploaded?

Don’t you find that TM is what you want? All support configuration files to fill in the directory stored after the uploaded jar! Hurry to look at the source code:

Learn Flink from 0 to 1 - where is the jar package you uploaded?

Source confirmation

thisjobmanager.web.upload.dirRight? I’ll check the source code of 1.8 to confirm:

Learn Flink from 0 to 1 - where is the jar package you uploaded?

Find thisjobmanager.web.upload.dirIt’s overdueWebOptionsClassUPLOAD_DIRInstead!

Keep coming in and have a look at thisUPLOAD_DIRWhat is it?

Learn Flink from 0 to 1 - where is the jar package you uploaded?

Look at this comment to say, if this configurationweb.upload.dirIf no specific path is configured, it will be usedJOB_MANAGER_WEB_TMPDIR_KEYDirectory, let’s see if this directory is configured?

Learn Flink from 0 to 1 - where is the jar package you uploaded?

The directory uploaded by the jar file is not configured, so let’s take a look at the temporary directoryJOB_MANAGER_WEB_TMPDIR_KEYWhere is it?

Learn Flink from 0 to 1 - where is the jar package you uploaded?

It’s another expired directory, MMP. Go on and look at this directoryTMP_DIR

Learn Flink from 0 to 1 - where is the jar package you uploaded?

Let’s check whether the configuration file has thisweb.tmpdirThe value of is not:

Learn Flink from 0 to 1 - where is the jar package you uploaded?

So, it definitely usesSystem.getProperty("java.io.tmpdir")This directory,

I have checked the following configuration of the job from my local computer:

java.io.tmpdir    /var/folders/mb/3vpbvkkx13l2jmpt2kmmt0fr0000gn/T/

Learn Flink from 0 to 1 - where is the jar package you uploaded?

After observing the next job, we find that there is a job managerweb.tmpdirConfiguration:

web.tmpdir    /var/folders/mb/3vpbvkkx13l2jmpt2kmmt0fr0000gn/T/flink-web-ea909e9e-4bac-452d-8450-b4ff082298c7

Learn Flink from 0 to 1 - where is the jar package you uploaded?

Find thisweb.tmpdirIt is composed of Java. Io. TMPDIR + “flick Web -” + UUID!

When the water subsides, the rocks emerge

Enter this directory and find that the jar we uploaded is finally found:

Learn Flink from 0 to 1 - where is the jar package you uploaded?

Configuration upload jar directory confirmation

Although we have known that the jar we uploaded is stored in this temporary directory, now we need to verify that we configure the fixed location of the uploaded jar in the configuration file. First, we create a jars directory in the directory, and then add the configuration in the configuration file:

web.tmpdir: /usr/local/blink-1.5.1/jars

Learn Flink from 0 to 1 - where is the jar package you uploaded?

See after the changeweb.tmpdirThat’s true:

Learn Flink from 0 to 1 - where is the jar package you uploaded?

Three jars have been uploaded from the Flink UI. Check/usr/local/blink-1.5.1/jars/flink-web-7a98165b-1d56-44be-be8c-d0cd9166b179Our jar appears in the directory.

Learn Flink from 0 to 1 - where is the jar package you uploaded?

When we restart Flink, we found that the three jars were gone again, which can also explain the problems I had before. After the restart of Flink, all the uploaded jars were deleted! As a production environment, it’s definitely not good to play like this, so we still have to fix a directory to store all the uploaded jar packages, and it can’t be deleted. To configure a fixed directory (if Flink is restarted and not deleted), you need to configure it as follows:

web.upload.dir: /usr/local/blink-1.5.1/jars

In this way, you can ensure that your jar will no longer be deleted!

Let’s see how the source code is written

//Find upload? Dir from the configuration file
final Path uploadDir = Paths.get(
    config.getString(WebOptions.UPLOAD_DIR,    config.getString(WebOptions.TMP_DIR)),
    "flink-web-upload");

return new RestServerEndpointConfiguration(
    restAddress,restBindAddress,port,sslEngineFactory,
    uploadDir,maxContentLength,responseHeaders);

It’s from the configuration fileUPLOAD_DIRIfnullJust look forTMP_DIRDirectory as the path of jar upload!

summary

From the problem of a friend of knowledge planet, from the phenomenon to the essence to the solution, this paper explains how to find the hiding place of the jar package uploaded on the Flink UI, and puts forward how to solve the problem of deleting the jar package uploaded on the Flink.

The link of this article is: http://www.54tianzhisheng.cn/2019/03/13/flick-job-jars/

Pay attention to me

The official account of WeChat:zhisheng

In addition, I have compiled some Flink learning materials, and I have put all the official account of WeChat. You can add my wechat:zhisheng_tian, then reply to the keyword:FlinkCan be obtained unconditionally.

Learn Flink from 0 to 1 - where is the jar package you uploaded?

More private information please join the knowledge planet!

Learn Flink from 0 to 1 - where is the jar package you uploaded?

GitHub code warehouse

https://github.com/zhisheng17/flink-learning/

In the future, all the code of this project will be put in this warehouse, including some demos and blogs for learning Flink.

Related articles

1. Learn Flink from 0 to 1: introduction to Apache Flink

2. Learn Flink from 0 to 1: building Flink 1.6.0 environment and building and running simple programs on MAC

3. Learn Flink from 0 to 1: details of Flink profile

4. Learn Flink from 0 to 1: introduction to data source

5. Learn Flink from 0 to 1 – how to customize data source?

6. Learn Flink from 0 to 1: introduction to data sink

7. Learn Flink from 0 to 1 – how to customize data sink?

8. Learn Flink from 0 to 1: Flink data transformation

9. Learn Flink from 0 to 1: introducing stream windows in Flink

10. Learn Flink from 0 to 1

11. Learn Flink from 0 to 1: Flink writes data to elasticsearch

12. Learn Flink from 0 to 1 – how does the Flink project work?

13. Learn Flink from 0 to 1: Flink writes data to Kafka

14. Learn Flink from 0 to 1 – Flink jobmanager high availability configuration

15. Learn Flink from 0 to 1: introduction to Flink parallelism and slot

16. Learn Flink from 0 to 1: Flink reads Kafka data and writes it to MySQL in batches

17. Learn Flink from 0 to 1: Flink reads Kafka data and writes it to rabbitmq

18. “Learn Flink from 0 to 1” – where is the jar package you uploaded?