Writing this article is actually a question asked by a small partner in the knowledge Planet:
Which directory is the jar submitted through the Flink UI dashboard stored in?
In fact, I have asked this question myself, but I haven’t thought about it deeply because I don’t have any pressure. Now it’s asked by the paying partner of the knowledge planet, so I naturally have to force myself to go deep before I can give the correct answer.
Follow me to see my search steps! How dare little jar play hide and seek with me?
The first thing to think about is that this can definitely be set in the configuration file:
Although there is an upload directory, it’s not what we want! So I had to use my “Google method”.
Find one and click in to see the issue as follows:
Don’t you find that TM is what you want? All support configuration files to fill in the directory stored after the uploaded jar! Hurry to look at the source code:
jobmanager.web.upload.dirRight? I’ll check the source code of 1.8 to confirm:
Keep coming in and have a look at this
UPLOAD_DIRWhat is it?
Look at this comment to say, if this configuration
web.upload.dirIf no specific path is configured, it will be used
JOB_MANAGER_WEB_TMPDIR_KEYDirectory, let’s see if this directory is configured?
The directory uploaded by the jar file is not configured, so let’s take a look at the temporary directory
JOB_MANAGER_WEB_TMPDIR_KEYWhere is it?
It’s another expired directory, MMP. Go on and look at this directory
Let’s check whether the configuration file has this
web.tmpdirThe value of is not:
So, it definitely uses
I have checked the following configuration of the job from my local computer:
After observing the next job, we find that there is a job manager
web.tmpdirIt is composed of Java. Io. TMPDIR + “flick Web -” + UUID!
When the water subsides, the rocks emerge
Enter this directory and find that the jar we uploaded is finally found:
Configuration upload jar directory confirmation
Although we have known that the jar we uploaded is stored in this temporary directory, now we need to verify that we configure the fixed location of the uploaded jar in the configuration file. First, we create a jars directory in the directory, and then add the configuration in the configuration file:
See after the change
Three jars have been uploaded from the Flink UI. Check
/usr/local/blink-1.5.1/jars/flink-web-7a98165b-1d56-44be-be8c-d0cd9166b179Our jar appears in the directory.
When we restart Flink, we found that the three jars were gone again, which can also explain the problems I had before. After the restart of Flink, all the uploaded jars were deleted! As a production environment, it’s definitely not good to play like this, so we still have to fix a directory to store all the uploaded jar packages, and it can’t be deleted. To configure a fixed directory (if Flink is restarted and not deleted), you need to configure it as follows:
In this way, you can ensure that your jar will no longer be deleted!
Let’s see how the source code is written
//Find upload? Dir from the configuration file final Path uploadDir = Paths.get( config.getString(WebOptions.UPLOAD_DIR, config.getString(WebOptions.TMP_DIR)), "flink-web-upload"); return new RestServerEndpointConfiguration( restAddress,restBindAddress,port,sslEngineFactory, uploadDir,maxContentLength,responseHeaders);
It’s from the configuration file
nullJust look for
TMP_DIRDirectory as the path of jar upload!
From the problem of a friend of knowledge planet, from the phenomenon to the essence to the solution, this paper explains how to find the hiding place of the jar package uploaded on the Flink UI, and puts forward how to solve the problem of deleting the jar package uploaded on the Flink.
The link of this article is: http://www.54tianzhisheng.cn/2019/03/13/flick-job-jars/
Pay attention to me
The official account of WeChat:zhisheng
In addition, I have compiled some Flink learning materials, and I have put all the official account of WeChat. You can add my wechat:zhisheng_tian, then reply to the keyword:FlinkCan be obtained unconditionally.
More private information please join the knowledge planet!
GitHub code warehouse
In the future, all the code of this project will be put in this warehouse, including some demos and blogs for learning Flink.
1. Learn Flink from 0 to 1: introduction to Apache Flink
2. Learn Flink from 0 to 1: building Flink 1.6.0 environment and building and running simple programs on MAC
3. Learn Flink from 0 to 1: details of Flink profile
4. Learn Flink from 0 to 1: introduction to data source
5. Learn Flink from 0 to 1 – how to customize data source?
6. Learn Flink from 0 to 1: introduction to data sink
7. Learn Flink from 0 to 1 – how to customize data sink?
8. Learn Flink from 0 to 1: Flink data transformation
9. Learn Flink from 0 to 1: introducing stream windows in Flink
10. Learn Flink from 0 to 1
11. Learn Flink from 0 to 1: Flink writes data to elasticsearch
12. Learn Flink from 0 to 1 – how does the Flink project work?
13. Learn Flink from 0 to 1: Flink writes data to Kafka
14. Learn Flink from 0 to 1 – Flink jobmanager high availability configuration
15. Learn Flink from 0 to 1: introduction to Flink parallelism and slot
16. Learn Flink from 0 to 1: Flink reads Kafka data and writes it to MySQL in batches
17. Learn Flink from 0 to 1: Flink reads Kafka data and writes it to rabbitmq
18. “Learn Flink from 0 to 1” – where is the jar package you uploaded?