Flink uses sideoutput to replace split to realize shunting

Time:2019-12-14

Previous data analysis project (version 1.4.2) forKafkaRead raw data stream, callingsplitInterface to achieve the shunt
New project decided to useFlink 1.7.2UsesplitWhen the interface is shunted, it is found that the interface is marked asdepracted(may be removed later)
Search related documents and find new versionsFlinkOut of band data is recommended for shunting

Pre establishmentOutputTagExample (LogEntityFromkafkaRead log instance class)

private static final OutputTag APP_LOG_TAG = new OutputTag<>("appLog", TypeInformation.of(LogEntity.class));
private static final OutputTag ANALYZE_METRIC_TAG = new OutputTag<>("analyzeMetricLog", TypeInformation.of(LogEntity.class));

YeskafkaRaw data read, viaprocessMark the interface accordingly

private static SingleOutputStreamOperator sideOutStream(DataStream rawLogStream) {
        return rawLogStream
                .process(new ProcessFunction() {
                    @Override
                    public void processElement(LogEntity entity, Context ctx, Collector out) throws Exception {
                        //Mark different objects according to the log level
                        if (entity.getLevel().equals(ANALYZE_LOG_LEVEL)) {
                            ctx.output(ANALYZE_METRIC_TAG, entity);
                        } else {
                            ctx.output(APP_LOG_TAG, entity);
                        }
                    }
                })
                .name("RawLogEntitySplitStream");
    }

    //Call function to mark the object in the original data flow
    SingleOutputStreamOperator sideOutLogStream = sideOutStream(rawLogStream);
    //According to the tags, different data flows are obtained for further analysis
    DataStream appLogStream = sideOutLogStream.getSideOutput(APP_LOG_TAG);
    DataStream rawAnalyzeMetricLogStream = sideOutLogStream.getSideOutput(ANALYZE_METRIC_TAG);

Through the above steps, data flow segmentation is realized

PS:
If you think my article is helpful to you, you can scan the code to get the red packet or scan the code support (any amount, a penny is love), thank you!

Alipay red envelope Alipay WeChat
Flink uses sideoutput to replace split to realize shunting Flink uses sideoutput to replace split to realize shunting Flink uses sideoutput to replace split to realize shunting

Recommended Today

JS function

1. Ordinary function Grammar: Function function name (){ Statement block } 2. Functions with parameters Grammar: Function function name (parameter list){ Statement block } 3. Function with return value Grammar: Function function name (parameter list){ Statement block; Return value; } Allow a variable to accept the return value after calling the function Var variable name […]