Hard core! Eight pictures to understand the end-to-end processing semantics of Flink exactly once

Time:2021-5-4

Flink

There are three locations in Flink that need end-to-end precise processing at one time

Hard core! Eight pictures to understand the end-to-end processing semantics of Flink exactly once

  • Source side: when data enters Flink from the previous stage, it needs to ensure that the message is consumed accurately at one time.
  • Internal side of FlinkWe have learned that by using checkpoint mechanism, the state can be saved to disk, which can be recovered in case of failure to ensure internal state consistency. If you don’t know, you can read my previous article:

    Detailed analysis of checkpoint mechanism, the cornerstone of Flink reliability

  • Sink end: when the processed data is sent to the next stage, it is necessary to ensure that the data can be accurately sent to the next stage.

Before Flink version 1.4, precise one-time processing was limited to the Flink application, that is, all operators were completely saved and managed by the Flink state to achieve precise one-time processing. But after Flink processes the data, most of the results need to be sent to external systems, such as sink to Kafka. In this process, Flink does not guarantee accurate processing at one time.

In Flink version 1.4, a milestone function has been formally introduced: the two-phase submit sink function. The sink function is extracted and encapsulatedA two-phase commit protocol is proposedFrom then on, Flink matches specific source and sink (e.g. Kafka version 0.11)Realize precise semantic processing at one time(English abbreviation: EOS, namely exactly only semantics).

End to end precise once processing semantics (EOS)

The following applies to Flink 1.4 and later

For the source side: the precise processing of the source side is relatively simple. After all, the data falls into Flink, so Flink only needs to save the offset of consumption data. For example, when consuming the data in Kafka, Flink uses Kafka consumer as the source to save the offset. If the subsequent task fails, the offset can be reset by the connector during recovery, Re consume data to ensure consistency.

For sinkSink is the most complexBecause the data is landed on other systems, once the data leaves Flink, Flink will not be able to monitor the data. Therefore, the precise one-time processing semantics must also be applied to the external systems where Flink writes the data. Therefore, these external systems must provide a means to allow these write operations to be submitted or rolled back, At the same time, we should also ensure that it can be used in coordination with Flink checkpoint (Kafka 0.11 version has realized precise processing semantics at one time).

Taking the combination of Flink and Kafka as an example, Flink reads data from Kafka, and the processed data is written into Kafka.

The first reason why we take Kafka as an example is that most of the data reading and writing in Flink system are carried out with Kafka system. The second reason is thatThe most important reason is that Kafka version 0.11 has officially released support for transactions, which is a necessary condition for Flink applications interacting with Kafka to achieve end-to-end precise semantics

Of course, Flink supports this kind of precise one-time processing semantics, which is not limited to the combination with Kafka. Any source / sink can be used as long as they provide the necessary coordination mechanism.

The combination of Flink and Kafka

Hard core! Eight pictures to understand the end-to-end processing semantics of Flink exactly once

As shown in the figure above, Flink contains the following components:

  1. A source to read data from Kafka (namely kafkaconsumer)
  2. A time windowed party operation (window)
  3. A sink to write the result to Kafka (Kafka producer)

To support precise one-time processing semantics (EOS), sink must write data to Kafka in the form of transactionWhen a transaction is committed, all writes between two checkpoints are committed as one transaction. This ensures that these writes can be rolled back in the event of a failure or crash.

Yes, of course,In a distributed application with multiple simultaneous sink execution, it is not enough to perform a single commit or rollback, because all components must reach a consensus on these commit or rollback, so as to ensure a consistent result. Flink uses a two-phase commit protocol and a pre commit phase to solve this problem

Two phase commit protocol (2pc)

Two phase commit (2pc) is a very common way to solve the problem of distributed transaction. It can ensure that all the participating processes commit or cancel the transaction in distributed transaction, that is, to realize a (atomicity) in acid

In the environment of data consistency, the meaning it represents is: either all backup data change a certain value at the same time, or do not change, in order to achieve data consistencyStrong consistency

There are two important roles in two-phase commit protocol, coordinator and participant. There is only one coordinator, which plays the role of coordination and management of distributed transactions, and there are multiple participants

As the name suggests, two-stage submission divides the submission process into two consecutive stagesVoting and commit

The two-stage protocol submission process is shown in the figure below:

Hard core! Eight pictures to understand the end-to-end processing semantics of Flink exactly once

The first stage: voting stage

  1. The coordinator sends a vote to all participants_ Request message.
  2. When participants receive vote_ Request message, send vote to Coordinator_ In response, the commit message tells the coordinator that he is ready to submit. If the participant is not ready or encounters other failures, a vote is returned_ Abort message, which tells the coordinator that the transaction cannot be committed at this time.

The second stage: submission stage

  1. The coordinator collects voting messages from each participant. IfIf all participants agree that the transaction can be committed, the coordinator decides the final commit of the transactionIn this case, the coordinator sends a global to all participants_ Commit message, informing participants to submit locally; If any of the participantsAny return message is vote_ Abort, the coordinator cancels the transactionTo broadcast a global message to all participants_ The abort message tells all participants to cancel the transaction.
  2. Each participant who submitted the voting information waits for the coordinator to return a message, if the participant receives a global_ Commit message, then the participant submits the local transaction, otherwise if global is received_ Abort message, the participant cancels the local transaction.

Application of two phase commit protocol in Flink

Flink’s idea of two stage submission

We start the Flink program, consume the Kafka data, and finally sink the data to Kafka to analyze the accurate processing of Flink.

  1. When checkpoint is started, jobmanager will inject checkpoint Battier into the data stream, and checkpoint barrier will be passed between operators, as shown below:

Hard core! Eight pictures to understand the end-to-end processing semantics of Flink exactly once

  1. Source sideFlink Kafka source is responsible for saving Kafka consumption offsetWhen chkpoint succeeds, Flink is responsible for submitting these writes, otherwise it will terminate and cancel them. When chkpoint finishes saving the displacement, it will pass the checkpoint barrier to the next operator, and then each operator will take a snapshot of the current state,Save to state backend(State Backend)。

    For the source task, the current offset is saved as the state. The next time you restore from checkpoint, the source task can re submit the offset and re consume the data from the last saved locationAs shown in the figure below:

Hard core! Eight pictures to understand the end-to-end processing semantics of Flink exactly once

  1. Slink end: starting from the source side, each internal transform task will save its status to checkpoint when it encounters checkpoint barrier. When the data is processed to the sink, the sink task first writes the data to the external Kafka. These data belong to the pre committed transaction (which cannot be consumed yet),At this time, in the pre commit phase, data sink must not only save the state to the state back end, but also advance its external transactionsAs shown in the figure below:

Hard core! Eight pictures to understand the end-to-end processing semantics of Flink exactly once

  1. When the snapshot of all operator tasks is completed(all snapshots created are considered part of checkpoint),In other words, when the checkpoint is completed, the job manager will send a notification to all tasks to confirm that the checkpoint is completed. At this time, the pre commit phase is considered to be completed. Just officially arrivedThe second phase of two-phase commit protocol: Commit phase. In this stage, the jobmanager will initiate the callback logic of checkpoint completed for each operator in the application.

    The data source and window operations in this example have no external state, so at this stage, the two operators do not need to perform any logic, butData sink has an external state. At this time, we must commit external transactionsWhen the sink task receives the confirmation notice, it will formally submit the previous transaction. The unconfirmed data in Kafka will be changed to “confirmed”, and the data can be consumed, as shown in the figure below:

Hard core! Eight pictures to understand the end-to-end processing semantics of Flink exactly once

Note: in Flink, the job manager coordinates each task manager to store checkpoints. Checkpoints are saved in stateback end. By default, stateback end is memory level. It can also be changed to file level for persistence.

Finally, a chart summarizes Flink’s EOS

Hard core! Eight pictures to understand the end-to-end processing semantics of Flink exactly once

It is suggested to save this picture, summarize it comprehensively and concisely, and never advise the interviewer again!