Detailed analysis of time and window in Flink

Time:2021-5-7

preface

Flink is a streaming, real-time computing engine

The above sentence has two concepts, one is streaming, the other is real-time.

Streaming: the data flow in continuously, that is, the data has no boundary, but when we calculate it, we must do it within a boundary range, so there is a problem, how to determine the boundary? There are only two ways,According to the time period or the amount of data to determineAccording to the time period, a boundary is divided according to how long it takes. According to the amount of data, a boundary is divided according to how many pieces of data come. This is how the boundary is divided in Flink. This article will explain it in detail.

real time: that is, after the data is sent, the relevant calculation is carried out immediately, and then the result is output. There are two kinds of calculation

  • One is to calculate only the data within the boundaryFor example, if you count the number of news browsed by each user in the last five minutes, you can take all the data in the last five minutes, and then group them according to each user to count the total number of news.
  • The other is to calculate the relationship between internal data and external dataFor example, if you want to count the regions of the users who browse the news in the last five minutes, you need to associate the information of the users who browse the news in the last five minutes with the region dimension table in hive, and then do the relevant calculation.

The content of Flink in this article is a detailed analysis around the above concepts!

Time and window

Time

In Flink, if the boundary is divided by time period, then time is an extremely important field.

There are three types of time in Flink, as shown in the figure below:

Detailed analysis of time and window in Flink

  • Event Time: is the time the event was created. It is usually described by the time stamp in the event. For example, in the collected log data, each log will record its own generation time. Flink accesses the event time stamp through the time stamp allocator.
  • Ingestion Time: is the time when the data enters Flink.
  • Processing Time: is the local system time of each time-based operator. It is machine related. The default time attribute is processing time.

For example, the time when a log enters Flink is 2021-01-22 10:00:00.123, and the system time when it arrives in window is 2021-01-22 10:00:01.234. The contents of the log are as follows:
2021-01-06 18:37:15.624 INFO Fail over to rm2

For business, which time is the most meaningful time to count the number of fault logs in one minute—— Event time, because we need to make statistics according to the generation time of the log.

Window

Window, that is, window. The boundary we mentioned earlier is the window here.

Official explanation:Streaming computing is a data processing engine designed to deal with infinite data sets. Infinite data sets refer to a growing essentially infinite data set, and window is a means of cutting infinite data into finite blocks for processing

thereforeWindow is the core of infinite data stream processing. Window splits an infinite stream into “buckets” of finite size, on which we can perform computation

Window type

At the beginning of this article, there are two ways to divide windows

  1. Time driven window based on time, such as once every 1 minute or once every 10 minutes.
  2. According to the data to intercept (data driven window), such as every 5 data statistics or every 50 data statistics.

Detailed analysis of time and window in Flink

Timewindow can be divided into three categories according to the principle of window implementationScrolling window, sliding window and session window

  1. Scrolling windows

Slice the data according to the fixed window length.

characteristic:Time aligned, window length fixed, no overlap

The scrolling window allocator allocates each element to a window with a specified size. The scrolling window has a fixed size and does not overlap.

For example, if you specify a 5-minute scrolling window, the creation of the window is as follows:

Detailed analysis of time and window in Flink

Applicable scenarios: suitable for Bi Statistics (aggregation calculation of each time period).

  1. Sliding windows

Sliding window is a more generalized form of fixed window, which is composed of fixed window length and sliding interval.

characteristic:Time alignment, fixed window length, overlap

The sliding window allocator allocates elements to a fixed length window. Similar to the rolling window, the size of the window is configured by the window size parameter, and another window sliding parameter controls the starting frequency of the sliding window. Therefore, if the sliding parameter of the sliding window is smaller than the window size, the windows can be overlapped. In this case, the elements will be assigned to multiple windows.

For example, if you have a 10 minute window and a 5-minute slide, the 5-minute window in each window contains the data generated in the last 10 minutes, as shown in the figure below:

Detailed analysis of time and window in Flink

Applicable scenario: Statistics in the latest time period (calculate the failure rate of an interface in the latest 5min to decide whether to alarm).

  1. Session windows

It is composed of a series of events and a specified length of timeout gap, which is similar to web application session, that is, a new window will be generated when no new data is received for a period of time.

characteristic:No time alignment

Session window allocator groups elements through session activities. Compared with rolling window and sliding window, session window has no overlapping and fixed start time and end time,When it no longer receives elements in a fixed period of time, that is, the inactive interval is generated, the window will close. A session window is configured by a session interval, which defines the length of the inactive period. When the inactive period is generated, the current session will be closed and subsequent elements will be assigned to the new session window.

Detailed analysis of time and window in Flink

Window API

TimeWindow

Timewindow is a window composed of all the data within the specified time range, which can calculate all the data in a window at a time (that is, the data within a boundary is calculated at the beginning of this paper).

We takeNumber of cars passing at traffic lightsFor example:

There will be cars passing at the intersection of traffic lights. How many cars will pass in total is impossible to calculate. Because of the continuous flow of traffic, the calculation has no boundary.

So we count the number of cars passing the traffic lights every 15 seconds, such as two in the first 15 seconds, three in the second 15 seconds, and one in the third 15 seconds

  • Tumbling time window (non overlapping data)

We use the NC command in Linux to simulate the sender of the data

1. Open the send port with the port number of 9999
nc -lk 9999

2. Send content (key represents different intersections, value represents vehicles passing each time)
Send one line at a time. The time interval of sending represents the time interval of the car
9,3
9,2
9,7
4,9
2,6
1,5
2,3
5,7
5,4

Flink collects the data and calculates:

object Window {
  def main(args: Array[String]): Unit = {
    //TODO time-window
    //1. Create a running environment
    val env = StreamExecutionEnvironment.getExecutionEnvironment

    //2. Define the data flow source
    val text = env.socketTextStream("localhost", 9999)

    //3. Convert data format, text > carwc
    case class CarWc(sensorId: Int, carCnt: Int)
    val ds1: DataStream[CarWc] = text.map {
      line => {
        val tokens = line.split(",")
        CarWc(tokens(0).trim.toInt, tokens(1).trim.toInt)
      }
    }

    //4. Perform statistical operation, one tumbling window for each sensorid, and the size of the window is 5 seconds
    //That is to say, count the number of cars passing the traffic lights every five seconds in the past five seconds.
    val ds2: DataStream[CarWc] = ds1
      .keyBy("sensorId")
      .timeWindow(Time.seconds(5))
      .sum("carCnt")

    //5. Display the statistical results
    ds2.print()

    //6. Trigger flow calculation
    env.execute(this.getClass.getName)

  }
}

The data we send does not specify a time field, so Flink uses the default processing time, which is the time when the Flink system processes the data.

  • Sliding time window (with overlapping data)
//1. Create a running environment
val env = StreamExecutionEnvironment.getExecutionEnvironment

//2. Define the data flow source
val text = env.socketTextStream("localhost", 9999)

//3. Convert data format, text > carwc
case class CarWc(sensorId: Int, carCnt: Int)

val ds1: DataStream[CarWc] = text.map {
  line => {
    val tokens = line.split(",")
    CarWc(tokens(0).trim.toInt, tokens(1).trim.toInt)
  }
}
//4. Perform statistical operation, one sliding window for each sensorid, with window time of 10 seconds and sliding time of 5 seconds
//That is to say, count the number of cars passing the traffic lights every 5 seconds in the past 10 seconds.
val ds2: DataStream[CarWc] = ds1
  .keyBy("sensorId")
  .timeWindow(Time.seconds(10), Time.seconds(5))
  .sum("carCnt")

//5. Display the statistical results
ds2.print()

//6. Trigger flow calculation
env.execute(this.getClass.getName)

CountWindow

Countwindow triggers execution according to the number of the same key elements in the window. During execution, only the result corresponding to the key whose number of elements reaches the size of the window is calculated.

Note: the window of countwindow_ Size refers to the number of elements with the same key, not the total number of all elements entered

  • Tumbling count window (non overlapping data)
//1. Create a running environment
val env = StreamExecutionEnvironment.getExecutionEnvironment

//2. Define the data flow source
val text = env.socketTextStream("localhost", 9999)

//3. Convert data format, text > carwc
case class CarWc(sensorId: Int, carCnt: Int)

val ds1: DataStream[CarWc] = text.map {
  (f) => {
    val tokens = f.split(",")
    CarWc(tokens(0).trim.toInt, tokens(1).trim.toInt)
  }
}
//4. Perform statistical operation, one tumbling window for each sensorid, and the size of the window is 5
//According to the key, the corresponding key appears five times as a result
val ds2: DataStream[CarWc] = ds1
  .keyBy("sensorId")
  .countWindow(5)
  .sum("carCnt")

//5. Display the statistical results
ds2.print()

//6. Trigger flow calculation
env.execute(this.getClass.getName)

  • Sliding count window (with overlapping data)

It is also the operation of window length and sliding window: window length is 5, sliding length is 3

//1. Create a running environment
val env = StreamExecutionEnvironment.getExecutionEnvironment

//2. Define the data flow source
val text = env.socketTextStream("localhost", 9999)

//3. Convert data format, text > carwc
case class CarWc(sensorId: Int, carCnt: Int)

val ds1: DataStream[CarWc] = text.map {
  (f) => {
    val tokens = f.split(",")
    CarWc(tokens(0).trim.toInt, tokens(1).trim.toInt)
  }
}
//4. Perform statistical operation, one sliding window for each sensorid, with 3 pieces of data in the window size and 3 pieces of data in the window sliding
//That is to say, each intersection counts the number of cars passing each intersection in the last five messages when receiving three messages about it
val ds2: DataStream[CarWc] = ds1
  .keyBy("sensorId")
  .countWindow(5, 3)
  .sum("carCnt")

//5. Display the statistical results
ds2.print()

//6. Trigger flow calculation
env.execute(this.getClass.getName)

  • Window summary
  1. Flink supports two ways of window partition (time and count)

    • If the window is divided according to time, it is a time window
    • If the window is divided according to the data, it is a count window
  2. Flink supports two important properties of windows (size and interval)

    • If size = interval, a tumbling window will be formed
    • If size > interval, a sliding window will be formed
    • If size < interval, the window will lose data. For example, every 5 seconds, statistics of the past 3 seconds through the intersection of the car data, will miss 2 seconds of data.
  3. Through the combination, we can get four basic windows

    • Time tumbling window time window without overlapping data, setting method example: timewindow (time. Seconds (5))
    • Time sliding window time window with overlapping data, setting method example: timewindow (time. Seconds (5), time. Seconds (3))
    • Count tumbling window the number window of non overlapping data. Example of setting method: countwindow (5)
    • Count sliding window the number window with overlapping data. Example of setting method: countwindow (5,3)

Window Reduce

Windowedstream → datastream: assign a reduce function to window and return an aggregate result.

import org.apache.flink.streaming.api.scala.StreamExecutionEnvironment
import org.apache.flink.api.scala._
import org.apache.flink.streaming.api.windowing.time.Time

object StreamWindowReduce {
  def main(args: Array[String]): Unit = {
    //Get execution environment
    val env = StreamExecutionEnvironment.getExecutionEnvironment

    //Create socketsource
    val stream = env.socketTextStream("node01", 9999)

    //The stream is processed and aggregated by key
    val streamKeyBy = stream.map(item => (item, 1)).keyBy(0)

    //Introduce time window
    val streamWindow = streamKeyBy.timeWindow(Time.seconds(5))

    //Performing aggregation operations
    val streamReduce = streamWindow.reduce(
      (item1, item2) => (item1._1, item1._2 + item2._2)
    )

    //Write aggregate data to a file
    streamReduce.print()

    //Execution procedure
    env.execute("TumblingWindow")
  }
}

Window Apply

The apply method can carry out some custom processing, which is implemented by anonymous inner class method. Used when there are some complex calculations.

usage

  1. Implement a windowfunction class
  2. Specify the generics of this class as [input data type, output data type, grouping field type used in keyby, window type]

Example: using apply method to realize word statistics

Steps:

  1. Get the running environment of stream processing
  2. Build socket stream data source, and specify IP address and port number
  3. Convert the received data into word tuples
  4. Using keyby for streaming (grouping)
  5. Use timewinodw to specify the length of the window (calculated every 3 seconds)
  6. Implement an anonymous inner class of windowfunction

    • Implementation of aggregate computing in apply method
    • Collect data using collector.collect

The core code is as follows:

//1. Get the running environment of stream processing
    val env = StreamExecutionEnvironment.getExecutionEnvironment

    //2. Build socket stream data source, and specify IP address and port number
    val textDataStream = env.socketTextStream("node01", 9999).flatMap(_.split(" "))

    //3. Convert the received data into word tuple
    val wordDataStream = textDataStream.map(_->1)

    //4. Use keyby to divide (Group)
    val groupedDataStream: KeyedStream[(String, Int), String] = wordDataStream.keyBy(_._1)

    //5. Use timewinodw to specify the length of the window (calculated every 3 seconds)
    val windowDataStream: WindowedStream[(String, Int), String, TimeWindow] = groupedDataStream.timeWindow(Time.seconds(3))

    //6. Implement an anonymous inner class of windowfunction
    val reduceDatStream: DataStream[(String, Int)] = windowDataStream.apply(new RichWindowFunction[(String, Int), (String, Int), String, TimeWindow] {
      //Data aggregation in apply method
      override def apply(key: String, window: TimeWindow, input: Iterable[(String, Int)], out: Collector[(String, Int)]): Unit = {
        println("hello world")
        val tuple = input.reduce((t1, t2) => {
          (t1._1, t1._2 + t2._2)
        })
        //Collect the data to be returned and send it back
        out.collect(tuple)
      }
    })
    reduceDatStream.print()
    env.execute()

Window Fold

Windowedstream → datastream: a function that assigns a fold function to the window and returns the result after a fold.

import org.apache.flink.streaming.api.scala.StreamExecutionEnvironment
import org.apache.flink.api.scala._
import org.apache.flink.streaming.api.windowing.time.Time

object StreamWindowFold {
  def main(args: Array[String]): Unit = {
    //Get execution environment
    val env = StreamExecutionEnvironment.getExecutionEnvironment

    //Create socketsource
    val stream = env.socketTextStream("node01", 9999,'\n',3)

    //The stream is processed and aggregated by key
    val streamKeyBy = stream.map(item => (item, 1)).keyBy(0)

    //Bring in scrolling window
    val streamWindow = streamKeyBy.timeWindow(Time.seconds(5))

    //Perform the fold operation
    val streamFold = streamWindow.fold(100){
      (begin, item) =>
        begin + item._2
    }

    //Write aggregate data to a file
    streamFold.print()

    //Execution procedure
    env.execute("TumblingWindow")
  }
}

Aggregation on Window

Windowedstream → datastream: aggregate all elements in a window. The difference between min and minby is that Min returns the minimum value, while minby returns the element containing the minimum value field (the same principle applies to Max and maxby).

import org.apache.flink.streaming.api.scala.StreamExecutionEnvironment
import org.apache.flink.streaming.api.windowing.time.Time
import org.apache.flink.api.scala._

object StreamWindowAggregation {
  def main(args: Array[String]): Unit = {
    //Get execution environment
    val env = StreamExecutionEnvironment.getExecutionEnvironment

    //Create socketsource
    val stream = env.socketTextStream("node01", 9999)

    //The stream is processed and aggregated by key
    val streamKeyBy = stream.map(item => (item.split(" ")(0), item.split(" ")(1))).keyBy(0)

    //Bring in scrolling window
    val streamWindow = streamKeyBy.timeWindow(Time.seconds(5))

    //Performing aggregation operations
    val streamMax = streamWindow.max(1)

    //Write aggregate data to a file
    streamMax.print()

    //Execution procedure
    env.execute("TumblingWindow")
  }
}

Eventtime and window

Introduction of eventtime

  1. It is inconsistent with the time in the real world, which is divided into event time, extraction time and processing time in Flink.
  2. If the time window is defined based on the eventtime, the eventtimewindow will be formed, and the message itself should carry the eventtime
  3. If the time window is defined based on the ingengttime, the ingengtimewindow will be formed, and the SYSTEMTIME of the source will prevail.
  4. If the time window is defined based on the processing time benchmark, the processing time window will be formed, and the system time of the operator will prevail.

In the flow processing of Flink, the vast majority of businesses will use eventtime. Generally, only when eventtime cannot be used, will we be forced to use processingtime or digestiontime.

If you want to use eventtime, you need to introduce the time attribute of eventtime in the following way:

val env = StreamExecutionEnvironment.getExecutionEnvironment

//Add time characteristics to each stream created by env from the call time
env.setStreamTimeCharacteristic(TimeCharacteristic.EventTime)

Watermark

introduce

We know that there is a process and time in flow processing from event generation to flow through source and then to operator. Although in most cases, the data flow to operator is in the chronological order of event generation, it can not be ruled out that the network, back pressure and other reasons lead to disorder, the so-called disorder, It means that the sequence of events received by Flink is not in strict accordance with the event time sequence of events, so when Flink was originally designed, it took into account the network delay, network disorder and other issues, so an abstract concept: watermark was proposed;

Detailed analysis of time and window in Flink

As shown in the figure above, there is a problem. Once there is disorder, if we only determine the operation of the window according to the eventtime, we can’t know whether all the data are in place, but we can’t wait indefinitely. At this time, there must be a mechanism to ensure that after a specific time, the window must be triggered to calculate. This special mechanism is the watermark.

Watermark is used to deal with out of order events, and the correct handling of out of order events is usually realized by watermark mechanism combined with window

The watermark in the data stream is used to indicate that the data whose timestamp is less than watermark has arrived. Therefore, the execution of window is also triggered by watermark.

Watermark can be understood as a delay trigger mechanism. We can set the delay time t of watermark. Each time, the system will check the maximum maxeventtime of the arrived data, and then confirm that all data whose eventtime is less than maxeventtime – t have arrived. If the stop time of a window is equal to maxeventtime – t, then the window will be triggered

The watermark of ordered flow is shown in the following figure: (watermark is set to 0)

Detailed analysis of time and window in Flink

The watermark of unordered flow is shown in the following figure: (watermark is set to 2)

Detailed analysis of time and window in Flink

When Flink receives each piece of data, it will generate a watermark, which is equal to the maxeventtime delay length of all the current arrival data. In other words, the watermark is carried by the data. Once the watermark carried by the data is later than the stop time of the window that is not triggered, the execution of the corresponding window will be triggered. Since watermark is carried by data, if new data cannot be obtained during the running process, the window that has not been triggered will never be triggered

In the figure above, we set the maximum delay arrival time as 2S, so the corresponding watermark for the event with time stamp of 7S is 5S, and the corresponding watermark for the event with time stamp of 12s is 10s. If our window 1 is 1s ~ 5S and window 2 is 6S ~ 10s, then the watermark for the event with time stamp of 7S just triggers window 1, When the event with a time stamp of 12s arrives, the watermark just triggers window 2.

Flink’s treatment of late data

Watermark and window mechanism solve the disorder problem of streaming data. For the data with wrong sequence due to delay, it can process the business according to the event time. Flink also has its own solution for the delayed data. The main method is to give a delay time, and it can still accept the delayed data within the time range.

Set the time allowed to delay byallowedLateness(lateness: Time)set up

The delay data is saved bysideOutputLateData(outputTag: OutputTag[T])preservation

Get delay data throughDataStream.getSideOutput(tag: OutputTag[X])obtain

The specific usage is as follows:

allowedLateness(lateness: Time)

def allowedLateness(lateness: Time): WindowedStream[T, K, W] = {
  javaStream.allowedLateness(lateness)
  this
}

This method passes in a time value to set the time that data is allowed to be late, which is different from the concept of time in watermark. Let’s review:

Watermark = event time of data – allowed out of order time value

With the arrival of new data, the watermark value will be updated to the latest data event time allowed out of order time value. However, if a piece of historical data comes at this time, the watermark value will not be updated. In general, watermark is to receive as much out of order data as possible.

The time value here is mainly for waiting for the late data. If the data belonging to this window arrives within a certain time range, it will still be calculated. The calculation method will be explained in detail later

Note: this method is only for Windows based on event time. If it is based on processing time and a non-zero time value is specified, an exception will be thrown.

sideOutputLateData(outputTag: OutputTag[T])

def sideOutputLateData(outputTag: OutputTag[T]): WindowedStream[T, K, W] = {
  javaStream.sideOutputLateData(outputTag)
  this
}

This method is to save the late data to the given outputtag parameter, and outputtag is an object used to mark the late data.

DataStream.getSideOutput(tag: OutputTag[X])

The method is called by the datastream returned by the operation such as window, and the object marked with the delayed data is passed in to obtain the delayed data.

Understanding of delayed data

Delayed data refers to:

After the current window [assuming that the window range is 10-15] has been calculated, another data belonging to the window [assuming that the event time is 13] will still trigger the window operation. This kind of data is called delay data.

So the question is, how to calculate the delay time?

Assuming that the window range is 10-15 and the delay time is 2S, the window operation can be triggered as long as the watermark is < 15 + 2 and belongs to the window. However, if a piece of data comes and makes watermark > = 15 + 2, 10-15, the window operation can no longer be triggered, even if the event time of the new data belongs to this window time.

Flink associated hive partition table

Flink 1.12 supports the function of the latest hive partition as a temporal table. It can directly associate the latest hive partition of the hive partition table through SQL, and it will automatically monitor the latest hive partition. When the new partition is monitored, it will automatically replace the full amount of dimension table data. In this way, users do not need to write a datastream programKafka stream real-time Association of the latest hive partition to achieve data widening

Specific usage:

Register hivecatalog in SQL client:

vim conf/sql-client-defaults.yaml 
catalogs: 
  - name: hive_catalog 
    type: hive 
    Hive conf dir: / disk0 / soft / hive conf / # the directory needs hive-site.xml file

Create Kafka table

CREATE TABLE hive_catalog.flink_db.kfk_fact_bill_master_12 (  
    master Row<reportDate String, groupID int, shopID int, shopName String, action int, orderStatus int, orderKey String, actionTime bigint, areaName String, paidAmount double, foodAmount double, startTime String, person double, orderSubType int, checkoutTime String>,  
Proctime as proctime () -- proctime is used to associate with hive temporal table  
) WITH (  
 'connector' = 'kafka',  
 'topic' = 'topic_name',  
 'format' = 'json',  
 'properties.bootstrap.servers' = 'host:9092',  
 'properties.group.id' = 'flinkTestGroup',  
 'scan.startup.mode' = 'timestamp',  
 'scan.startup.timestamp-millis' = '1607844694000'  
);

Link fact table with hive’s latest partition data

dim_ extend_ shop_ Info is an existing table in hive, so we use table hint to dynamically open dimension table parameters.

CREATE VIEW IF NOT EXISTS hive_catalog.flink_db.view_fact_bill_master as  
SELECT * FROM  
 (select t1.*, t2.group_id, t2.shop_id, t2.group_name, t2.shop_name, t2.brand_id,   
     ROW_NUMBER() OVER (PARTITION BY groupID, shopID, orderKey ORDER BY actionTime desc) rn  
    from hive_catalog.flink_db.kfk_fact_bill_master_12 t1  
       JOIN hive_catalog.flink_db.dim_extend_shop_info   
  /*+ OPTIONS('streaming-source.enable'='true',  
     'streaming-source.partition.include' = 'latest',  
     'streaming-source.monitor-interval' = '1 h',
     'streaming-source.partition-order' = 'partition-name') */
    FOR SYSTEM_ Time as of T1. Prompt as T2 -- temporal table  
    ON t1.groupID = t2.group_id and t1.shopID = t2.shop_id  
    where groupID in (202042)) t  where t.rn = 1

Parameter explanation:

  • streaming-source.enableTurn on streaming to read hive data.
  • streaming-source.partition.includeThere are two values:

    1. Latest attribute: read only the latest partition data.
    2. All: read full partition data. The default value is all, which means read all partitions. Latest can only be used in temporary join, which is used to read the latest partition as dimension table. It cannot read the latest partition data directly.
  • streaming-source.monitor-intervalThe time to monitor the generation of new partitions should not be too short, and the shortest time is 1 hour, because the current implementation is that every task will query the Metastore, and high-frequency queries may cause too much pressure on the Metastore. It should be noted that 1.12.1 has released this restriction, but it is still recommended not to allocate too short an interval according to the actual business.
  • streaming-source.partition-orderThere are three main partition strategies, among which the most recommended one ispartition-name

    1. Partition name uses the default partition name order to load the latest partitions
    2. Create time creates a time sequence using a partition file
    3. Partition time uses partition time order

Search the official account: five minutes to learn big data, get big data learning secrets, and dig into big data technology.