Previous data analysis project (version 1.4.2) forKafkaRead raw data stream, callingsplitInterface to achieve the shunt
New project decided to useFlink 1.7.2UsesplitWhen the interface is shunted, it is found that the interface is marked asdepracted(may be removed later)
Search related documents and find new versionsFlinkOut of band data is recommended for shunting
Pre establishmentOutputTagExample (LogEntityFromkafkaRead log instance class)
private static final OutputTag APP_LOG_TAG = new OutputTag<>("appLog", TypeInformation.of(LogEntity.class));
private static final OutputTag ANALYZE_METRIC_TAG = new OutputTag<>("analyzeMetricLog", TypeInformation.of(LogEntity.class));YeskafkaRaw data read, viaprocessMark the interface accordingly
private static SingleOutputStreamOperator sideOutStream(DataStream rawLogStream) {
return rawLogStream
.process(new ProcessFunction() {
@Override
public void processElement(LogEntity entity, Context ctx, Collector out) throws Exception {
//Mark different objects according to the log level
if (entity.getLevel().equals(ANALYZE_LOG_LEVEL)) {
ctx.output(ANALYZE_METRIC_TAG, entity);
} else {
ctx.output(APP_LOG_TAG, entity);
}
}
})
.name("RawLogEntitySplitStream");
}
//Call function to mark the object in the original data flow
SingleOutputStreamOperator sideOutLogStream = sideOutStream(rawLogStream);
//According to the tags, different data flows are obtained for further analysis
DataStream appLogStream = sideOutLogStream.getSideOutput(APP_LOG_TAG);
DataStream rawAnalyzeMetricLogStream = sideOutLogStream.getSideOutput(ANALYZE_METRIC_TAG);Through the above steps, data flow segmentation is realized
PS:
If you think my article is helpful to you, you can scan the code to get the red packet or scan the code support (any amount, a penny is love), thank you!
| Alipay red envelope | Alipay | |
|---|---|---|
 |
 |
 |