How to migrate Flink tasks to real-time computing

Time:2021-9-13

Introduction: This article is shared by Alibaba technical expert Jing lining (Yantian). It mainly introduces how to migrate Flink tasks to real-time computing Flink.

Usually, users mainly use Flink run offline, which will cause some problems. For example, the same configuration changes due to the version; Unable to quickly switch between different versions; Cannot recover from previous job.

How to migrate Flink tasks to real-time computing

So how to migrate from offline to online? This article will introduce you through the following four parts.

How to migrate Flink tasks to real-time computing

1、 How to migrate: from Flink run to streaming computing platform

As shown in the following figure, how to migrate offline commands to online commands. First, open the VVP UI and add the basic configuration, mainly including job name, jar URI, some main parameters and parallelism. You can also click Advanced Configuration to configure more information.

How to migrate Flink tasks to real-time computing

For example, the behavior configuration in advanced configuration is divided into three parts: upgrade strategy, initial state and recovery strategy. Generally, stateless is selected for the upgrade strategy, running is selected for the initial state, and latest state is selected for the recovery strategy.

How to migrate Flink tasks to real-time computing

Stateless is selected as the upgrade strategy, which means that for the job that is already running, after editing and saving the configuration, the original job will be stopped directly, and then an updated job will be restarted; Select stateable. If you edit and save the configuration for the running job, the job will first do a savepoint, then load a new configuration and start a new job with the savepoint.
Latestsavepoint is selected as the recovery strategy, which means that if the job is restarted when it is suspended, it will be started with the latest savepoint.
The above describes the more important part of Flink configuration. The following picture shows another part of the configuration that can set checkpoint interval, etc., followed by resource configuration. In the log configuration part, you can select the log storage location to facilitate troubleshooting if there are problems in the operation in the future.

How to migrate Flink tasks to real-time computing

After the Flink job is configured and started to run, if there are exceptions, you can view the operation status and problems through the running events. You can also open the job manager of Flink UI to view the log.

How to migrate Flink tasks to real-time computing

2、 How can users add other dependent jars in the case of multiple jars

Some users have custom dependencies and cannot solve the problem through fat jar. For example, user a has his own main jar and some other jars. The application scenarios of these jars are different. First, you need to upload the jar package on the resource management page. After uploading, you can use the file on the page.
After uploading, go to the advanced configuration, find “additional dependent jar”, pull down and select the jar package just uploaded.

How to migrate Flink tasks to real-time computing

3、 How do users define jobs through profiles

Upload files in resource management, then select Add dependency in advanced configuration and select the required dependency. If the main class needs to read a file in the startup function, first add the file through dependency, and then read the file according to the prompt.

The above two methods are the way to place user jars. Jars and other files can be stored here.

4、 How to reuse the original state to accelerate job recovery

If a problem is found when running a job, it is usually stopped, fixed and restarted. The user needs to specify the OSS required by the checkpoint when creating it. If checkpoint is enabled, subsequent jobs can be recovered directly from checkpoint.
First, confirm that lateststate or latestsavepoint is selected as the recovery policy in the advanced configuration. On this premise, you can pause the job (never stop the job). The pause will trigger the savepoint. When you click start again, it will resume from this savepoint.

How to migrate Flink tasks to real-time computing

Open the Flink UI and click job snapshot. In the job snapshot, you can view the Flink checkpoint indicator, which will display the number of restores and the address of the last restore, so as to determine that the job is restored from the latest state.

How to migrate Flink tasks to real-time computing

When a job is running, how do you want to copy the current job input? Click checkpoint on the job control page, find the historical snapshot, and click the “copy job from this snapshot” on the far right to complete the operation of copying the current job, including the jar package configuration of the running job. After clicking start, you can see from the job snapshot that the new job is copied and started from the latest savepoint of the previous job.

Author:Jing lining (Yantian)
Original link
This article is the original content of Alibaba cloud and cannot be reproduced without permission